February 2020 Summaries

Fivetran is transitioning to a consumption-based pricing model for all new customers and existing customer renewals. The new model charges based on the number of monthly active rows (MAR) across all connectors in an account, with prices similar to their previous connector-based model. This approach simplifies pricing, reflects value received by customers, aligns with Fivetran's actual costs, and makes them more comparable to other data integration providers. The consumption-based pricing model is designed to be objective, transparent, and easy to understand for users.

Fivetran has been named one of the "Best Places to Work in Colorado" by tech recruiting and media firm Built In. The company was recognized for its comprehensive benefits package, which includes health insurance with vision and dental benefits, generous parental leave, flexible work schedules, unlimited vacation policy, job training and conferences, dedicated diversity and inclusion staff, community involvement program, and more. This honor follows Fivetran's recognition as a top global employer by Great Place to Work and its five-star Glassdoor rating at all locations. The company is currently hiring for various positions across teams and regions.

Fivetran and Databricks have partnered to address two major challenges in data architecture: ensuring data quality and governance, and achieving completeness of enterprise data. The collaboration aims to simplify loading data into Delta Lake, an open-source technology for building a reliable and fast Lakehouse. A Lakehouse combines the best elements of data lakes and warehouses by decoupling storage from compute, supporting diverse data types and workloads, and offering ACID transaction support. The partnership aims to solve issues with data quality and completeness in data lakes, enabling organizations to focus on extracting value from their data in a centralized location while saving development time and money.

Jesse Anderson highlights the common misconception that having a large quantity of data, such as 3 billion rows or 100 GB, automatically necessitates the use of Big Data technologies, emphasizing that this is not always the case and that small data technologies might be more suitable in such scenarios. He argues that small data solutions, like cloud data warehouses, often offer advantages such as less complexity, lower costs, and faster query speeds, making them a better fit for many use cases that do not truly require Big Data capabilities. Anderson also identifies scenarios where Big Data is genuinely needed, such as dealing with hundreds of billions of rows or petabytes of data, and stresses the importance of having a well-trained team and realistic project expectations. He warns against the tendency to overestimate Big Data as a cure-all solution, noting that companies should tailor their approach based on specific use cases rather than applying a one-size-fits-all methodology.

The Essential Guide to Data Integration provides practical steps for getting started with automated data integration. To choose the right course, organizations must assess their needs, decide whether to migrate or start fresh, evaluate cloud data warehouse and business intelligence tools, and evaluate data integration tools. They should also calculate the total cost of ownership and establish success criteria before setting up a proof of concept. The guide covers topics such as how data integration fuels analytics, the evolution from ETL to ELT to automated data integration, the benefits of automated data integration, and tips on how to evaluate data integration providers.

This excerpt from The Essential Guide to Data Integration discusses technical considerations when choosing a data integration tool. It highlights the importance of data connector quality, including open-source vs. proprietary connectors and standardized schemas and normalization. Additionally, it emphasizes the need for tools that support multiple sources and destinations, as well as automation features like API control, handling data type changes, continuous sync scheduling, automatic schema migrations, and general performance metrics. The text also explores the differences between ETL and ELT approaches to transformations within or before the data warehouse, emphasizing the benefits of non-destructive transformations in an elastic cloud-based environment. Lastly, it touches on security and regulatory compliance considerations for data integration providers.

The Essential Guide to Data Integration discusses business considerations when choosing a data integration tool, emphasizing that the choice depends on an organization's size, maturity, and specific data pipeline requirements. Common pricing models for ETL tools include flat subscription fees, volume-based pricing, per-seat pricing, and freemium trials. Factors to consider when selecting a tool include ease of use versus configurability, compatibility with existing skill sets, vendor lock-in, and future needs. The guide also provides insights into technical considerations for choosing a data integration tool.

The Essential Guide to Data Integration discusses why building your own data pipeline may not be the best approach due to high costs, time consumption, and potential negative impacts on morale. It is estimated that 80% of a data scientist's time is spent constructing data pipelines, which can take five weeks per connector and require ongoing maintenance work. Building custom connectors or manual reporting can lead to frustration, exhaustion, downtime, and misguided decisions. Additionally, not all APIs are easily integrated, and complexity increases as the number of data sources grows. The guide suggests that outsourcing pipeline engineering can be more cost-effective and efficient, allowing for standardization and easier integration with other tools. It also provides tips on how to win over engineers and convince executives about the benefits of purchasing a data pipeline tool.

At Tableau Conference 2019, Fivetran CEO George Fraser discussed how successful analytics programs incorporate automated data integration and cloud data warehouse technology. He argued that combining these technologies with a BI tool in a modern data stack creates an efficient analytics program fueled by every data source used by the business. The presentation also covered resolving common data engineering and analytics challenges, such as overcoming limitations of traditional ETL tools, using single cloud data warehouses, when to use a data lake, schema foundations, SQL's role in transformation and modeling, and accelerating analytics with prebuilt templates and dashboards.

Many tech companies are focusing on fostering a culture of diversity and inclusion (D&I) as it is not only beneficial for their employees but also helps attract top talent. Companies like Facebook, Looker, and Fivetran have implemented various initiatives to promote D&I in their organizations. These strategies include creating diverse interview panels, launching employee resource groups, and forming "Guardians of the Culture" teams. By making D&I an integral part of their corporate strategy, companies can create a more inclusive work environment and attract top talent.

In a recent interview with Jeff Meyerson, host of the Software Engineering Daily podcast, Fivetran CEO George Fraser discussed the evolution of data warehousing and the role of data engineering in modern businesses. He explained that cloud-based column-store databases have made data warehousing more accessible to companies of all sizes, allowing them to store and analyze large amounts of data from various sources. The interview also touched on Fivetran's automated data integration process, which eliminates the need for engineers to write scripts to query APIs like Salesforce and directly replicates data into a single data warehouse. Fraser shared his belief that data engineering will continue to evolve towards functional simplicity, with businesses focusing more on using data to drive their operations rather than building complex data infrastructure from scratch.