July 2022 Summaries

Real-time data warehousing is crucial for businesses aiming to deliver superior digital experiences, retain customers and innovate rapidly. It involves the continuous process of loading data into a warehouse and making it instantly available to different business functions. This approach offers significant benefits such as faster decision-making, enhanced data democratization, personalized customer experiences, improved business agility, and unlocking new business use cases. Real-time data warehousing is becoming increasingly relevant and essential for companies that want to leverage data to remain competitive in the fast-paced digital era.

Fivetran CEO George Fraser, Databricks CEO Ali Ghodsi and dbt Labs CEO Tristan Handy discuss the future of technologies powering data analytics and engineering. They emphasize making analytics more accessible to non-technical users, improving latency issues, and enhancing collaboration between business users and analytics engineers through a semantics layer. The evolution of the modern data stack (MDS) is crucial for democratizing data access and enabling innovation. Open source tools play a significant role in driving MDS development, while addressing latency issues remains an important frontier.

Fivetran is dedicated to creating the best experience for women in their company by striving to close the gender gap in tech. They offer support upon arrival, ensuring new hires have access to all available resources and opportunities for growth. Many female sales professionals at Fivetran are top performers, setting high standards for driving results and implementing innovative strategies. The promotion path within the sales organization is clear and performance-based, with more than 20 females being promoted in the SDR/BDR organization globally last year. Fivetran is actively hiring across all departments and regions, encouraging women to join their team and contribute to a diverse and inclusive work environment.

In today's world, cloud computing is not new; however, the sheer quantity and diversity of available data sources make it strikingly different from previous eras. The current glut in data requires a new set of tools to manage and analyze this unprecedented amount of information effectively. A modern data stack (MDS) is a set of cloud-native technologies for managing enormous and heterogeneous data sets, which can help CDOs demonstrate results quickly. Key components of an MDS include a fully managed ELT data pipeline, a data warehouse or lakehouse, data transformation tools, and data visualization platforms. By using an MDS, companies can uncover insights buried in large datasets and share them across different business units to drive growth and innovation.

The article discusses the costs of building versus buying data pipelines, highlighting that it's often more cost-effective and efficient to buy a solution rather than build one. It outlines setup costs, maintenance costs, and opportunity costs associated with both options. Building a data pipeline can be expensive due to the need for specialized skills, ongoing maintenance requirements, and potential delays in business intelligence delivery. On the other hand, buying a fully managed data pipeline solution can save time, reduce upfront costs, and allow data teams to focus on higher-value activities such as building internal apps, advising key decision-makers, and training others in data literacy. The article suggests that companies should consider their best use of engineering team's time when making a build versus buy decision.

Fivetran has introduced a new method for database replication called Teleport Sync, which allows users to quickly replicate changes using only a read-only user. This technology was acquired from Teleport Data in June 2021 and integrates with Fivetran's existing log-based replication methods. The main advantage of Teleport Sync is that it can be used when traditional log-based methods are not feasible, such as due to hardware constraints or inability to access primary instances or third-party hosted databases. Currently, the technology supports MariaDB, MySQL, Oracle, PostgreSQL, and SQL Server, with plans for further expansion.

The article discusses the role of Sales Development Representative (SDR) as a gateway into various careers within the tech industry, even for those without coding skills. The author shares their personal experience of starting as an SDR at Fivetran and progressing to different roles such as account manager, senior customer success manager, business development manager, and manager of Enterprise Account Strategist team. They emphasize that the SDR role intersects with all aspects of a company, allowing individuals to explore various areas like marketing, product department, or operations. The article also highlights Fivetran's 30-60-90 day training program for sales hires and the importance of empathy and cooperation in the learning experience. For those interested in tech but lacking technical background, the author recommends considering an SDR role as it provides equal opportunities from any background.

99% of companies face challenges in integrating their SAP ERP data, according to a recent global survey by Dimensional Research and Fivetran. The lack of real-time access to this valuable business data can lead to slowed operations, poor decision-making, and lost revenue. Key challenges include limited access to real-time data, difficulty in seamlessly integrating SAP ERP data, and the inability to operationalize it for maximum value. Log-based change data capture (CDC) technology is considered a superior replication method that can handle large volumes of data at speed with minimal impact on transactional sources. Fivetran offers a high-volume data replication solution using log-based CDC, which helps automate SAP ERP data movement and minimize the effort required to build and manage data pipelines.

The Fivetran Airflow provider has released a new version that eliminates issues with hanging sensors. In previous versions, some connectors' syncs would finish before the corresponding sensor could start monitoring them, causing potential problems. This issue was resolved by utilizing XCOM, an Airflow mechanism that allows tasks to pass key-value pairs between each other. The latest version of the Fivetran Airflow provider now stores a timestamp of the connector's previously completed sync in XCOM, which can be used by a sensor through calling xcom_pull. This ensures that the previously completed timestamp will be correct and a sensor will return at the appropriate time, even if it does not start until after the sync that it is supposed to monitor has already completed.

The text discusses the importance of bi-directional data movement and low latency in the integration of on-premises and cloud systems for real-time analysis, highlighting the projected increase in cloud spending to over $1.3 trillion by 2025 due to the adoption of cloud technologies for analytics. It emphasizes the cloud's scalability and cost-effectiveness, allowing organizations to avoid significant upfront investments and facilitating experimentation with different configurations. The migration of operational systems to the cloud requires extensive testing to ensure performance, availability, and security, and may necessitate fallback options. Best practices for cloud integration involve using change data capture (CDC) solutions, particularly log-based CDC, to handle real-time data changes with minimal overhead. The text also mentions the role of companies like Fivetran in providing CDC solutions, enabling continuous data integration and near real-time updates to support competitive differentiation and cost-saving opportunities.

The article explores three methods for transferring data from Snowflake to Google Sheets, highlighting the pros and cons of each approach. The most efficient method involves using Fivetran Activations, which facilitates automatic syncing through a straightforward process requiring minimal configuration. Alternatively, a more complex option involves using a Python script to connect the two platforms, necessitating additional steps such as setting up a service account and enabling APIs. The most labor-intensive method involves manually exporting data from Snowflake as CSV files and importing them into Google Sheets, which can be practical for one-time transfers but is cumbersome for frequent use. Each method caters to different user needs and technical capabilities, allowing users to choose based on their frequency of data transfer and technical proficiency.

The article explores three methods to sync data from Snowflake to Slack, enhancing workflow efficiency. It outlines the use of Fivetran Activations, which automates frequent data syncing without complexity, making it ideal for recurring tasks. The manual method is recommended for occasional data transfers or when processes vary, involving exporting data via Snowflake's web UI and uploading it to Slack as a file. The API method, while more complex, offers a customizable solution using Snowflake and Slack APIs with Python libraries to automate syncing, although additional features like scheduling and error handling are necessary for production environments. Each method provides unique advantages, catering to different needs and preferences for data integration between Snowflake and Slack.