August 2022 Summaries

In the current economic climate, businesses are looking to do more with fewer resources. Data teams can leverage new tools and technologies to achieve this goal by automating wherever possible, utilizing prebuilt, scalable resources, leveraging data security initiatives for self-serve analytics, and re-evaluating their data strategy. Automation reduces manual effort and time involved in data projects while also decreasing costs. Prebuilt, scalable resources allow teams to focus on higher-value analytics and deliver new data projects quickly with minimal maintenance, hosting, and management costs. Data security initiatives can expand self-serve analytics, freeing up data teams to focus on more advanced projects. Re-evaluating the data strategy can help identify bottlenecks that could become enablers by leveraging prebuilt resources and automation. Automated ELT solutions like Fivetran can help businesses save time and money while efficiently serving stakeholders across departments.

Fivetran has extended private networking support for its Business Critical plan to include Databricks, allowing sensitive data to be sent without exposure to the public internet. This feature is particularly beneficial for highly regulated industries like healthcare and financial services. Private networking provides secure communication between Fivetran's cloud environment and a user's Databricks environment, reducing dependency on IT teams and supporting regulatory compliance requirements. The setup process varies depending on whether the Databricks instance is hosted on AWS or Azure.

Christine Pearson, a former Pediatric Intensive Care Nurse, shares her experience transitioning into tech sales after a year of career change. Despite having no prior experience, she successfully landed a Business Development Representative role at Fivetran through determination and self-belief. By leveraging her existing skills from nursing and networking on LinkedIn, Pearson demonstrated her potential to the hiring manager. Upon joining Fivetran, she received support from colleagues and participated in an impressive onboarding program. Within a year, Pearson was promoted twice and is now in a management position. She encourages others considering a career change to be their own best advocate and believe in themselves.

This article discusses how to measure the success of individual contributors in a data team. It outlines three main roles - Data Engineer, Analytics Engineer, and Data Analyst - and provides metrics for evaluating their performance. For Data Engineers, these include downtime, testing coverage, lead time, and failure rate. Analytics Engineers can be evaluated based on failure rate, testing coverage, lead time, adoption rate, and trustworthiness. Data Analysts' success is measured by lead time, adoption rate, trustworthiness, departmental metrics, and analytical rigor. The article emphasizes the importance of business model alignment, substantive expertise, and adaptability for all roles.

Fivetran has introduced a High Volume Agent (HVA) connector for SQL Server to address the growing need for faster, high-volume database replication. The HVA connectors use an agent-based approach to read directly from source system logs, minimizing replication latency and supporting large volumes of data. These connectors incorporate features such as incremental updates and schema-drift monitoring. Common high-volume database replication use cases include supply chain management, event data tracking, and customer transaction analysis. Fivetran's HVA connectors support multiple connection methods, change data capture methodology, automated schema drift handling, column blocking, masking, hashing, history mode, and all Fivetran-supported destinations. The benefits of using these connectors include ease of use, rapid integration between high-volume data sources and destinations, access to up-to-date, accurate, and granular data in near real-time, and reduced total cost of ownership.

The article discusses the success measurement of data teams by categorizing their output into informational and operational value. Informational value refers to insights derived from data, while operational value is about having data in the right place and state. Both types of value are crucial for modern data teams. Additionally, maintaining and extending value requires data pipelines that are inspectable, maintainable, and extensible. The article will further delve into these concepts by discussing individual contributors, managers, and product owners in subsequent posts.

Syncing data between Google BigQuery and Slack can enhance workflows by providing timely updates and notifications. There are three main methods to achieve this: manually, through APIs, and using Fivetran Activations. The manual method is labor-intensive but suitable for occasional use. Using APIs, particularly with tools like the Python BigQuery client library and the Python Slack SDK, allows for automation but requires setup and additional considerations for production use, such as error handling and tracking synced data. Fivetran Activations offers a no-code solution that simplifies the process, automatically syncing data and notifying users of updates, making it ideal for regular and ongoing data integration without the manual effort or complex setup. Each method offers distinct advantages depending on the frequency of data syncing and the level of automation desired.

Amazon Redshift Serverless was released in July 2022 as a serverless-based approach to computing, eliminating the need for users to provision and manage compute infrastructure. It allows data analysts, engineers, scientists, and professionals to develop, build, and run data products and applications without designing, provisioning, or managing data warehouse clusters. Redshift Serverless provides a quick and easy setup process with $300 in free trial credits. Users can change their RPU base capacity from 32 to 512 in increments of 8, with wait times ranging from under four minutes to over seven minutes. The Query Editor v2 allows users to run queries on the provided sample datasets. Redshift Serverless also provides snapshots for point-in-time backups and recovery points every 30 minutes. AWS offers a range of out-of-the-box dashboards for query, database, and resource monitoring. Users can set alarms for workgroups, namespaces, or snapshot storage. Unlike provisioned clusters, Redshift Serverless only charges users when queries are run, with data storage costs still applying. The service is available across various AWS regions in North America, EMEA, and APAC. Overall, Redshift Serverless offers a simple, intuitive user experience while maintaining key features and functionality.

The Fivetran REST API allows users to programmatically manage users, groups, and connectors for scaling data workflows and improving security posture. It offers efficient, consistent, and powerful ways to perform bulk actions, communicate with Fivetran from other applications, and automate human-led processes using codified logic. The API is used by organizations for creating and managing pipelines at scale, improving overall security posture, extending Fivetran through integrations with developer tools, and syncing data from customers to software products.

Fivetran, a company founded by George Fraser and Taylor Brown, has become a $5.6 billion enterprise after its acquisition of competitor HVR for $700 million in August 2021. The deal was made possible through a last-minute fundraising effort that raised $565 million from five investment firms within 72 hours. Fivetran's co-founders, Fraser and Brown, each own about a tenth of the company, putting their net worth at around $500 million each. The company now forecasts $189 million in revenue for this fiscal year, up from $90 million last year. Fivetran's market lead in data pipelines is considered "unassailable" by investor Martin Casado due to its ease of use and complexity behind the scenes.

The modern data stack is evolving with the integration of technologies like Apache Iceberg, a table format for analytic tables that brings together the best capabilities of both data lakes and warehouses. Iceberg was created by Netflix and has been adopted by companies such as Apple, LinkedIn, Stripe, Airbnb, Pinterest, and Expedia. It enables direct data access needed by various use cases without compromising SQL behavior in data warehouses. The adoption of Iceberg will provide more options for the modern data stack, allowing users to choose their preferred processing pattern or query layer while integrating all their data sources. This change is expected to lead to a separation of query and storage functions within the data warehouse ecosystem.

A high-volume data replication solution can help organizations gain real-time access to their SAP ERP data, enabling them to integrate it with other unstructured data and maximize its value. This approach allows for scalable access to SAP ERP data, simplified integration, optimization of business operations, reduced costs, improved data quality, and enhanced security and privacy compliance. Log-based change data capture (CDC) is the gold standard for high-volume data replication, offering continuous integration, real-time synchronization, and minimal impact on transactional systems. By automating high-volume data movement with near-zero latency to various cloud-based platforms, companies can free up their data teams to focus on improving business decisions and gain a competitive advantage in the rapidly changing business environment.

The concept of data maturity is crucial for businesses to evaluate their utilization of data effectively. Data teams play a significant role in extracting value from organizational data and laying the groundwork for advanced analytics projects. A modern, cloud-based data infrastructure can serve as a bridge from the beginning of your data journey to the final stages of data maturity. The stages of data maturity involve moving from descriptive analytics to diagnostic, predictive, and prescriptive analytics. Data-driven businesses are more likely to grow revenue and have a competitive edge. A modern data stack, consisting of automated data integration, cloud data warehouse, data transformation tool, and business intelligence tool, supports data maturity and advanced analytics by providing seamless access to all data sources and promoting universal data sharing.