January 2023 Summaries

New Lite connectors are a new tier of connector with an accelerated development cycle built for specific use cases and based on the available APIs of SaaS applications. Fivetran's Lite connectors are generally available, offering connections to various SaaS applications such as Active Campaign, Assembled, Buildkite, Chargebee, Crossbeam, Drift, Freightview, Gong, Ironclad, Lattice, Okta, Open Data CR, and others. Additionally, a private preview of Lite connectors is also available for other applications including Ada Chat Bot, Aircall, Alchemer, Appfigures, Azure Boards, Calendly, Campaign Monitor, Chameleon, Channel Advisor, ChargeDesk, ChartMogul, Chorus, CIMI, Cin7, Collibra, Dixa, Docebo, Duoplane, E-conomic, Employment Hero, Everhour, Freshdesk Contact Center, Freshsuccess, G2, GoCardless, Google Tasks, Guru, Harvest, HiBob, Impact, Lessonly, LiveChat, Lob, Mention, Monday.com, Mux, Namely, Open Exchange Rates, Orbit, Pingdom, Pipe17, PostHog, Productboard, Ramp, Reply, SCORM, Sharetribe, ShipStation, ShortSkuVault, Squarespace, Survicate, Teamwork, Tempo, Trello, Vonage, Web Scraper, WordPress, Wrike, Yotpo, Zingtree.

The challenges of building or buying a reliable and well-functioning data pipeline include reverse engineering the underlying data model, handling schema drift, automating the process, enabling users to build new data assets, dealing with incremental updates, ensuring idempotence, parallelizing operations, pipelining and buffering, scaling and extensibility, and managing security and PID. These challenges are complex and require careful consideration of technical problems and potential solutions.

Loading data from Salesforce to Redshift can be challenging without the right technology and expertise. However, using an automated cloud ETL tool like Fivetran provides an easier alternative to manual loading methods. Fivetran offers various benefits, including easy setup, automated real-time syncing, multiple connectors, and less resource management. In contrast, manually loading data from Salesforce to Redshift can be time-consuming, labor-intensive, and prone to errors due to the complexity of ETL scripts and the need for constant monitoring of upgrades and potential breakdowns. By using Fivetran, businesses can save time and resources while ensuring accurate and reliable data integration.

Fivetran has introduced an async provider for Airflow, developed in collaboration with Astronomer, allowing data engineers to run Fivetran data syncs more efficiently in Airflow 2.2+. The new provider utilizes deferrable operators and sensors, enabling asynchronous execution of tasks without consuming worker slots, thus improving the overall efficiency of Airflow deployments. This allows for I/O bound operations like data syncing to be performed without blocking other tasks or DAGs. The Fivetran async provider can be added to any Airflow 2.2+ environment with pip install airflow-provider-fivetran-async, and provides examples on how to use it in DAGs.

Data pipelines and Extract, Transform, Load (ETL) are often used interchangeably, but they have distinct differences. ETL is a process that extracts data from sources, transforms it into a usable format, and loads it into a destination repository, whereas data pipelines are continuous structures that gather data from multiple sources, process it in real-time, and deliver it to various destinations such as analytics tools or data visualizations. While ETL focuses on batch processing, data pipelines can support real-time data analysis using technologies like Fivetran, which provides a range of extract, transform, load tools and connectors to create custom pipelines tailored to specific needs and applications.

Fivetran CEO George Fraser recently embarked on a global "world tour" to connect with Fivetran employees across Ireland, Serbia, and India, reinforcing company culture and growth. He spent quality time in each location, meeting new people and energizing teams by sharing exciting initiatives for 2023. The purpose of the trip was to understand unique regional challenges and opportunities, as well as recognize significant team contributions. George's visits allowed him to connect with leaders and staff on a personal level, learning almost as much in one day as he used to spend a week visiting different offices. He reinforced his commitment to making each location successful, highlighting the importance of talent in driving business growth. The CEO emphasized that Fivetran delivers a product customers love and is growing quickly, while also expressing his desire to realize the company's full potential.

Fivetran supports women in a male-dominated field by considering merit over other factors, creating an inclusive work culture that motivates employees to give 100% effort, providing opportunities for career growth and development, and actively promoting diversity, equity, inclusion, and belonging. The company's Employee Resource Groups (ERGs) offer a platform for women to share their experiences, learn from each other, and grow professionally. Fivetran values its female employees equally with male employees, providing a nurturing environment that encourages women to establish themselves as professionals and achieve their career goals.

Evidence-based business is about guiding decisions with high-quality evidence, starting with leaders who transform their organizations into being evidence-based. The hierarchy of evidence includes randomized controlled trials, natural experiments, observational evidence, and mechanistic reasoning. Mechanistic reasoning is often used in business to generate hypotheses, while observational evidence is common but weak. Controlling for confounders and using natural experiments can help improve the quality of observational data. Randomized controlled trials are the strongest form of evidence, but they can also produce false negatives or false positives. Leaders need to overcome organizational resistance when implementing high-quality evidence, and it's okay to reverse decisions based on new evidence. Successful businesses will see team members presenting data about evidence quality, decision-makers acknowledging uncertainty, and regular changes in mind based on high-quality evidence.

Data hygiene is crucial for achieving accurate and easily usable insights from data, as it involves ensuring that data is correct, well-structured, and efficiently queryable. It requires collaboration between the analytics team, stakeholders, and data teams to bridge the gap between raw data generation and insight creation. Data hygiene encompasses three key principles: quality, architecture, and efficiency. Quality refers to accurate representation of reality, while architecture involves clear relationships between objects and data sources. Efficiency is achieved through efficient querying with well-defined foreign keys and unique identifiers. To implement effective data hygiene, stakeholders must document properties used in business logic, while the analytics team focuses on testing, setting up alerts, and writing efficient queries. Ultimately, prioritizing data hygiene is essential for business success as it directly impacts the performance of automations and decision-making processes.

Backcountry, a premium online retailer of outdoor gear and apparel, shares key lessons on using a modern data stack to overcome data silos, complexities with legacy systems and improve its customer experience. The company's data engineering team has built and leveraged a modern data stack in three ways: automating data ingestion and pipeline maintenance, building a simple and integrated modern data stack, and developing a data strategy that supports AI and ML. By doing so, Backcountry has reduced pipeline issues by 30%, decreased the time spent on pipeline maintenance, improved accuracy of data being provisioned, and enhanced their AI and ML capabilities.

Leading fintech companies AJ Bell, OrderPay, and Tide are leveraging self-service analytics to gain better insights into their businesses. These companies have automated their data processes to make data more accessible and support decision-making capabilities. The adoption of a modern data stack has helped them overcome challenges posed by legacy technology, such as growing datasets and manual data pipelines. Automation is reaping rewards for these companies, allowing them to focus on strategic initiatives and improve overall business performance.

The article discusses the differences between traditional and modern data warehouses, highlighting that modern ones leverage cloud infrastructure for improved scalability, flexibility, and return on investment. It explains how a modern data warehouse differs from its traditional counterpart in terms of location, purpose, scope, data sources, and architecture. The benefits of using a modern data warehouse include lower upfront costs, less maintenance, faster speeds, more flexibility, easier scaling, and ease of migration. The article also provides steps to follow when migrating to a modern data warehouse, emphasizing the importance of defining data goals, identifying business needs, knowing core data processes, assessing accessibility, selecting the right architecture, using data integration tools, and staying flexible.

Autodesk, a company responsible for delivering critical insights across all platforms in their suite, faced challenges with extraction, storage and transformations. When evaluating whether to build or buy data pipelines, the team considered four key factors: cost-effectiveness, no-code or low-code route for data ingestion, time spent on solving business value problems, and alignment with existing toolchain and cloud data ecosystem. By choosing to buy tools like Fivetran and Snowflake, Autodesk has seen significant savings in time and maintenance, allowing them to focus on more strategic projects such as machine learning infrastructure and consolidating BI dashboards.

Change data capture (CDC) tools identify and capture changes in source systems, enabling faster data movement between databases for improved business intelligence capabilities. The best CDC tools include Fivetran, Qlik Replicate, Hevo Data, Talend, Oracle GoldenGate, Striim, StreamSets, and IBM Infosphere. These tools offer various features such as log-based change data capture, flexible deployment options, real-time streaming analytics, and built-in data validation to help businesses eliminate data silos, sync data across multiple systems, and reduce the burden on operational databases.