October 2021 Summaries

The blog post reflects on the evolution of the data engineering role over the past five years, highlighting key trends in the modern data ecosystem that are reshaping the field. It notes that the shift to cloud-based, managed data infrastructure services is reducing the need for traditional data infrastructure engineers, while the rise of data integration services like Fivetran and Airbyte is simplifying data extraction tasks, making custom scripts obsolete. The post also discusses the growing importance of reverse ETL, which facilitates the integration of data from warehouses back into operational systems. It highlights the trend towards ELT over ETL, facilitated by tools like dbt, and the industry’s reliance on templated SQL and YAML for managing transformations. The emergence of the analytics engineer role is seen as complementing rather than replacing data engineers, who are increasingly focused on horizontal initiatives like data modeling, coding standards, and metadata management. There's an emphasis on the democratization of the analytics process, making data skills more accessible and fostering the rise of roles like data ops and data observability. Finally, the post touches on the erosion of the semantic layer in BI, the push for decentralized data governance, and the trend of every product becoming a data product, signaling a shift towards embedded analytics solutions.

Apache ECharts has become the preferred charting tool for the Apache Superset project due to its high performance, powerful API, and strong governance model, which aligns well with Superset's needs. Over the past year, the Superset community has gradually transitioned from using NVD3 to ECharts for creating visualizations, focusing particularly on time-series visualizations such as line charts, bar charts, scatter plots, and area charts. Initially, a single unified time-series chart was used, but it was later split into distinct visualization types to improve user experience and adhere to Superset's design patterns. Each chart type has specific use cases, ranging from line charts that track continuous changes over time to scatter plots that highlight individual data points. ECharts also supports advanced visualizations like smooth and stepped line charts, which provide different perspectives on data trends, and mixed time-series charts that combine multiple data series for comprehensive analysis. These developments are particularly relevant for users of Superset versions 1.3 and later, as well as Preset Cloud users, offering them enhanced capabilities for data visualization and interpretation.

Maxime Beauchemin, a prominent figure in data engineering known for creating Apache Airflow and Apache Superset, reflects on the evolving role of data engineers amidst advancing technologies and decentralization of data teams. As the CEO of Preset, he highlights how the shift to cloud-based solutions has transformed data engineering from managing infrastructure to focusing on data performance and reliability. Decentralized team structures now distribute data governance responsibilities, leading to challenges in consensus but also fostering innovation. The rise of roles like analytics engineers signifies a shift toward more specialized tasks, alleviating some burdens from data engineers. However, operational creep persists, requiring engineers to manage costs and quality. Modern tools like DataOps and Data Observability are aiding in automating repetitive tasks and ensuring data integrity, promoting a focus on treating data as a product. This evolution presents opportunities for engineers to build resilient and scalable data systems, emphasizing the importance of data reliability and accessibility throughout its lifecycle.

Preset has been developing an internal community tracker using open source data tools to better understand the Superset community, intending to make this work open source. This effort involves ingesting data from platforms like GitHub using Airbyte and transforming it into organized tables in a data warehouse with dbt, a tool for data transformation and validation using SQL. The process includes unifying data from various repositories, exploding JSON payloads into structured columns, and creating entity-focused tables for analytics. dbt models and macros are employed to streamline these transformations, maintain data integrity, and ensure scalability. The team aims to de-duplicate records efficiently and enrich the data with additional metadata before making it available for analytics and visualization in tools like Apache Superset. Future plans include sharing the resulting dbt models and further developing data views in Superset.