November 2020 Summaries

Creating an MVP for a data integration project posed the challenge of balancing a quick release with a product compelling enough to attract users or investors. The team initially considered using Singer as a core component to quickly bootstrap the MVP, but ultimately aimed to make Singer peripheral in the long-term. They explored various strategies, including wrapping a UI around Singer, creating their own configuration abstraction, Dockerizing integrations, and transitioning to the Airbyte protocol due to limitations with Singer. Despite the challenges and the tempting hindsight of starting with Singer at the periphery, the early integration allowed for valuable learning experiences and insights into Singer's problem-solving approaches, such as its use of a breadcrumb system for metadata mapping. This informed their decision-making and helped avoid potential mistakes, affirming their choice to iterate quickly while learning from existing solutions.

Singer.io is an open-source CLI tool for data integration, but its use case does not align with programmatically sending data from any supported source to any destination. The lack of configuration declarations and absence of a way to determine which features are compatible with each integration make it difficult to leverage Singer in this manner. Additionally, the presence of experimental features within the protocol raises uncertainty about compatibility with other systems. While Singer is not suitable for these specific requirements, its core protocol remains valuable for data integration tasks.

The article discusses how data storage and management have evolved over the last decade, moving from ETL to ELT, with companies like Fivetran pushing this trend. It suggests that the future might see a transition towards EL(T), where E and L are decoupled from T. The problems with ETL include inflexibility, lack of visibility, and limited autonomy for analysts. ELT is considered better due to cloud-based computation and storage, supporting agile decision-making for analysts, and promoting data literacy across the company. However, EL(T) is seen as the future because it supports both analytics and operational use cases, scales faster, and leverages the whole ecosystem by decoupling E and L from T.