April 2024 Summaries

Fivetran's automated data platform enables the centralization of all data, modernization of data infrastructure, achievement of greater data self-service and democratization, and building differentiating data solutions in the Snowflake Data Cloud with Streamlit. The Fivetran platform allows users to quickly set up and configure connectors for various data sources, including Workday HCM, SAP, SQL Server, Oracle, Salesforce, GA4, Kafka, S3, and more than 500 other sources. Once connected, Fivetran automates the process of moving data into Snowflake, providing a standardized and predictable experience across all data sources. Streamlit is used to build custom data applications that can seamlessly query and display data from Snowflake, with features such as interactive filtering, dynamic SQL queries, and secure access control. The combination of Fivetran and Streamlit enables users to quickly and easily build high-quality, usable, and trusted data into Snowflake, which can immediately be used to build any Streamlit application they can imagine.

Fivetran helps large companies access, manipulate, and find insights from their data when and where they need it by extracting diverse areas of SAP data into a cloud data warehouse. This enables data engineers to quickly and reliably replicate data from SAP systems into the cloud-based repository for analytics and business insights without extensive development. The platform integrates with over 500 sources, including SAP and other common ERP systems, and offers low-impact change data capture solutions that can efficiently replicate data from various SAP configurations. By centralizing SAP data with Fivetran, enterprises can derive extensive value from their data, empower themselves to embrace data freedom, and improve their bottom line by unlocking powerful insights related to optimizing manufacturing processes, tracking financial performance, and holistic customer views.

The benefits of building and deploying an enterprise data platform include data democratization, productivity improvements, and the creation of a cross-functional center of excellence. By moving from legacy systems to modern cloud-native solutions, companies can increase agility and improve decision-making processes. A key aspect of this transformation is empowering data democratization by forming a central data platform and establishing a culture of independence and autonomy among data teams. This approach enables organizations to reduce duplicative effort and prioritize initiatives across business units, ultimately driving business growth and success.

While many companies are investing heavily in artificial intelligence, a significant gap exists between non-technical and technical executives' confidence levels. This disparity can lead to overconfidence and oversimplification of AI implementation, overlooking the importance of high-quality data. Companies that prioritize robust data governance and management practices tend to navigate these challenges more effectively, with 96% of surveyed companies crediting their strategic investments in data management as a key factor in overcoming obstacles. The average loss due to poor data quality is $406 million, highlighting the crucial role of strong data foundations in achieving successful AI outcomes.

While many businesses are not yet ready to invest in AI-driven innovation, organizations that have successfully implemented strong data readiness and governance frameworks can reap significant benefits. To bridge the dissonance gap, data consultancy services like Hakkoda can provide specialized tooling and expertise to support quality, governance, and automation initiatives. By modernizing their data stack with industry-leading tools like Fivetran, companies can establish a robust foundation for innovation, including AI integration, and achieve GenAI goals. With Hakkoda's guidance, organizations can overcome common data readiness blockers such as quality, governance, and automation challenges, and quickly achieve AI readiness.

In the era of AI and advanced analytics, managing and accessing data efficiently has become crucial for businesses. Centralized data management struggles with the increasing scale and complexity of enterprise-level data demands. To address these challenges, organizations are adopting a data mesh architecture that decentralizes data ownership to domain-specific teams while maintaining a cohesive governance structure. This approach alleviates pressure on central data teams and empowers domain experts to manage and use data more effectively under the oversight of a centralized governance framework. A successful data mesh strategy involves establishing a data center of excellence (DCoE) that enhances data observability, controllability, and scalability. Effective metadata management is key to maintaining visibility and governance in a data mesh architecture.

Data lakes have evolved into powerful tools for data storage and analysis, offering flexibility, scalability, and cost-effectiveness. However, making data easily accessible and usable by downstream consumers remains a challenge. Automation can play a crucial role in managing data movement to a data lake, addressing issues related to compliance, usability, and security. Key strategies for overcoming these challenges include focusing on pipelines, deploying data catalogs, and automating data processing tasks. By embracing automation, organizations can maximize the potential of their data lakes, enhance efficiency, and leverage data-driven insights to drive business success.

Fivetran enables users to move relational databases, including Microsoft SQL Server, to Google Cloud BigQuery for analysis with tools like BigQuery ML. To set up this automated fraud detection system, Fivetran is used to connect to a SQL Server database and transfer the credit card transactions dataset to BigQuery for use with BigQuery ML. The process involves selecting specific datasets and columns from the original SQL Server data, configuring incremental changes and schema drift, and setting up a new dataset in BigQuery. Once the data is moved, users can create and run machine learning models using BigQuery ML to detect fraudulent transactions. This solution provides centralized data, modernized infrastructure, and scalable ML applications for combating credit card fraud without requiring setup or maintenance.

Riley Buss, Senior Manager of IT Engineering at CHS Inc., manages multiple teams of data engineers and data scientists to build efficient supply chains across global markets. He oversees 50 business applications and 14 different enterprise resource planning systems, managing terabytes of real-time ingestion and difficult bandwidth challenges. With the help of a modern data stack consisting of Fivetran, Snowflake, and dbt, he can ingest gigabytes of data in real time from farms and assets across America, ensuring high-quality data for business users. Buss emphasizes the importance of collaboration and servant leadership, empowering team members to have a voice in strategic planning and partnering with business users to build technical solutions that meet their needs.

The volume, velocity, and variety of data have grown exponentially due to "big data" and the widespread adoption of cloud-based tools and technologies. Data lakes, centralized repositories that store structured and unstructured data at scale with minimal processing, have become increasingly important for managing this data. They offer scalable, flexible, and affordable large-scale storage, which is essential for analytics. With the rise of machine learning and artificial intelligence workloads, data lakes are preferred as they can handle large volumes of semi-structured and unstructured data. Fivetran now supports data lakes as a destination, with support for structured data lake formats like Delta Lake and Iceberg. These formats enable data lakes to be governed, offering capabilities normally associated with data warehouses such as ACID compliance, schema enforcement, cataloging, governance, security, and SQL-based querying and editing. Fivetran automates data integration and movement, ensuring the reliability and integrity of data syncs, handling schema drift or evolution, and guaranteeing pipeline and network performance through optimization, parallelization, and pipelining.

Fivetran has been named the Technology Partner of the Year for Data Ingestion by Google Cloud for the third consecutive year, highlighting its commitment to innovation and collaboration within the ecosystem. The award signifies the continued value Fivetran delivers to Google Cloud users through its best-in-class data integration platform, which helps organizations centralize data from various sources into Google BigQuery. Companies such as Backcountry, Vida Health, and GroupM have seen significant benefits from adopting Fivetran and Google Cloud. The recognition fuels Fivetran's passion for pushing the boundaries of data integration and fostering a collaborative environment with Google Cloud to help businesses harness the power of their data.

Over the past year, Fivetran has significantly expanded its services to support more data sources, destinations, and deployment models. The company now supports over 500 connectors, including SaaS applications, events, files, databases, and ERPs. It achieved a 99.97% uptime and introduced one-minute sync frequencies for fast and reliable data movement at scale. Fivetran also supports all modern destinations across databases, warehouses, and event streaming platforms, as well as three deployment models: cloud, on-premises, and hybrid. Additionally, the company offers a 14-day free trial to test its services.

The Fivetran REST API offers a powerful solution to automate and optimize data pipelines, making them more efficient and adaptable than ever before. It enables users to centralize data from multiple departments, each with its own database, by automating the deployment, configuration, and management of connectors. The API supports various use cases such as pipeline management, data transformation, data governance, resource management, and webhook magic. It also provides a flexible way to create destinations, groups, and connectors, allowing users to fine-tune their data operations with ease. Additionally, it offers a scalable solution for synchronizing and transforming data, with options for integrated, partially integrated, or independent scheduling of transformations. The API is designed to be user-friendly, with essential resources such as API documentation, code samples, and authentication essentials. By leveraging the Fivetran REST API, users can streamline their data pipeline management and ensure it scales with their organization's evolving needs.

Fivetran accelerates building GenAI applications for customer service by providing a fully automated and managed data movement platform that supports delivering high-quality, usable, trusted data to any data workload in BigQuery. Fivetran's automated data platform centralizes data in a secure way while modernizing data infrastructure, achieving greater data self-service and building differentiating data solutions like GenAI apps. The author sets up multiple connectors to sources housing customer service data using Fivetran, moves the data into Google BigQuery, and then quickly prototypes simple GenAI search and chat apps in Vertex AI. Vertex AI enables building generative AI apps quickly with a range of models to choose from and automatically trains, tests, and tunes predictive models within a single platform. The author builds two simple GenAI apps using Vertex AI and BigQuery, including a search app and a chat app, and showcases their performance and capabilities. Fivetran provides seamless integration with dbt, including more than 20 Quickstart data models for connectors like Jira and Zendesk, allowing for no-code transformations and analytics-ready tables. The author concludes that building search and chat Gen AI apps with Google BigQuery and Vertex AI is simple and fast, thanks to Fivetran's automated data pipeline providing reliability, scalability, predictability, security, and context for GenAI apps.