April 2018 Summaries

Apache NiFi and Streamsets are two open-source ETL tools that offer visual dataflow programming, allowing users to create programs without writing code. Both tools have their pros and cons, with Apache NiFi having a clean architecture but a spartan user interface, while Streamsets has a more attractive UI but can be less user-friendly due to its validation requirements. Data Provenance is a feature in Apache NiFi that records the history of data flows, which can be useful for debugging purposes. Both tools have similar functionality and are suitable for record-based data and streaming. Ultimately, the choice between Apache NiFi and Streamsets depends on individual preferences and needs, with live monitoring being a key feature in Streamsets that sets it apart from Apache NiFi.

While developing an open-source analytics framework Cube.js, we've encountered various data warehouses with different requirements and needs. When choosing a modern data warehouse, it's essential to consider factors such as the volume of data, dedicated human resources for support and maintenance, scalability, pricing models, and on-premises vs cloud infrastructure. The ideal solution depends on the dataset size, available resources, and specific requirements, with non-relational databases suitable for large datasets, relational databases offering great query optimizers for smaller datasets, and self-hosted options like Hadoop requiring significant setup and maintenance expertise. Pricing models vary across solutions, with Redshift, BigQuery, and Snowflake offering on-demand pricing, while Amazon S3-based solutions provide scalable and flexible pricing options. Ultimately, the choice of data warehouse depends on balancing data volume, scalability, and cost considerations to meet specific business needs.

The concept of ETL, Extract, Transform, Load, has been a traditional method for managing analytics pipelines for decades but is changing with the advent of modern cloud-based data warehouses such as BigQuery or Redshift, which are shifting towards ELT - when transformations are run directly in the data warehouse. The traditional ETL process is complicated and outdated, requiring significant time and resources to implement and maintain, particularly during transformation rules changes. Modern data warehouses have optimized for analytical operations, offer cheap storage, and are cloud-based, making it possible to perform transformations in the background or at query time, providing flexibility and agility for development of a transformation layer.