September 2021 Summaries

Fivetran, a data integration service, offers multi-cloud solutions that provide flexibility, control, and cost reduction for organizations. By allowing customers to deploy Fivetran on different cloud platforms such as Google Cloud, AWS, and Microsoft Azure, users can avoid vendor lock-in and adhere to internal policies or regulatory requirements. Multi-cloud also enables data ingress and egress optimization by minimizing costs associated with moving data between clouds. With multi-cloud capabilities available in Fivetran's Enterprise and Business Critical plans, customers can run Fivetran components and process their data where needed while benefiting from industry-leading security and regulatory compliance features.

The Modern Data Stack Conference celebrated data professionals who are transforming businesses and communities with data-driven decisions. The conference featured winners of the inaugural Modern Data Stack Awards across seven categories, highlighting mentors building institutional knowledge, leaders designing innovative architectures, and individuals putting data tools in the hands of underserved communities. Among the award recipients were teams like Callahan and Netlify, as well as individuals such as Chris Wagner, Daniel Guzmán Burgos, Emilie Schario, Kelly Kohlleffel, and NextAfter.

Fivetran, a data integration service, has announced its deployment on Microsoft Azure, allowing joint customers to leverage automated data pipelines while remaining within the Azure cloud and meeting relevant security and compliance requirements. This collaboration aims to provide an enhanced experience for customers like Paul Hewitt and Hyper Hippo Productions, enabling them to minimize egress and ingress costs, address compliance and security concerns, and benefit from IT requirements and policy adherence, lower costs, improved security and compliance certifications, and flexibility for multi-cloud adoption. Fivetran on Azure also offers early access to its Business Critical offering with features such as Azure Private Link, customer-managed encryption keys, and PCI DSS Level 1 validation.

The article explores three methods for exporting data from Google BigQuery to a CSV file, catering to different user needs and technical proficiencies. The first method involves using the BigQuery console's web UI, which is straightforward for one-time exports. The second method utilizes the bq command-line tool, offering efficiency for users who are comfortable with command-line operations and need frequent exports. The third method employs a client library, allowing for integration into automated workflows with various programming languages, though it requires more initial setup. The article also highlights limitations such as the 1 GB export limit per file and the challenge of exporting nested data, suggesting workarounds like using wildcards and the UNNEST function. Additionally, it introduces Fivetran Activations as a simpler alternative for moving data from BigQuery to other destinations, bypassing the need for manual CSV exports.

The Modern Data Stack Conference 2021 features over 40 sessions across four virtual stages, with keynotes by industry leaders. Key topics covered include insights as a service, reverse ETL, interactive data applications, and self-service analytics. Can't-miss sessions include discussions on the future of analytics, making businesses data-literate, leveraging analytics to survive a pandemic, machine learning acceleration, predictable revenue from ecommerce and product analytics, moral psychology in persuasion, self-service analytics, insights as a service, and more. The conference aims to provide valuable insights for professionals working in the data space.

A data catalog is an inventory of all an organization's data assets, bundled with tools to maintain the catalog. It not only enumerates the data but also describes it, including database and table names, column names, column descriptions, and access rights. Unlike a database schema that applies to data from a single source, a data catalog describes all the data in an organization across all data sources and repositories. A data catalog can perform automated discovery of data sets, build data dictionaries, maintain relationships between data sets, track data lineage, and provide a visual interface for analysts with search capabilities. It makes analytics easier for non-experts by helping them see what data is available, where it comes from, and what other data it's related to.

Fivetran hosted their annual Partner Summit, where they recognized and celebrated the contributions of their SI and tech alliance partners. The company launched its inaugural Partner Awards to honor those who have gone above and beyond in their support for Fivetran's ecosystem. Tech Alliances Awards winners included Snowflake, Databricks, Google Cloud, and Looker, while SI Partner Awards winners were Slalom, Brooklyn Data, Data Culture, Montreal Analytics, Biztory, CIS Consulting, Infinite Lambda, Servian, Goldstone Technologies, and Classmethod. These partners play a crucial role in Fivetran's success by accelerating time to value for customers and sharing the common goal of evangelizing the modern data stack and creating the best customer experience possible.

Fivetran plans to acquire HVR, a leader in enterprise database replication, and has raised $565 million in Series D funding led by Andreesen Horowitz. This acquisition aims to enhance the modern data stack for customers, employees, and partners. HVR's technology focuses on replicating mission-critical databases for large enterprises, while Fivetran specializes in managing complex SaaS tools used by businesses. The combination of both companies will enable easier access to data for various company sizes and types of data sources, aligning with Fivetran's mission to make data access as simple and reliable as electricity.

The Modern Data Stack Conference, taking place on September 22-23, is a virtual event for data professionals of all levels. With over 40 sessions lined up, the conference will cover best practices, innovations, and technological developments in the industry. Notable presentations include "Accelerate Product Time-to-Value with Machine Learning-based Models" by Bid Ops and Dataiku, "Building Fantastic Data Products and How to Support Them" by 4 Mile Analytics, "Build Interactive Data Apps With Embedded Search-Driven Analytics" by ThoughtSpot and KlearNow, and "Insights as a Service: Creating Data Products from your Customers' Data" by Untitled. Registration is open for those interested in attending the conference.

Fivetran, a data pipeline platform, has achieved PCI DSS Level 1 Validation for its Business Critical plan. This is crucial for ecommerce customers who can now connect Fivetran to their PCI-validated data sources and destinations, enabling the transmission of cardholder data to a PCI-validated destination. The validation process involved documenting and assessing 300 controls encompassed in 12 requirements on an annual basis by a qualified security assessor (QSA). Fivetran's mission is to make business data as accessible as electricity, supporting all compliance and regulatory requirements that its customers follow.

Fivetran is a data integration platform that prioritizes security and adheres to industry leading standards such as ISO 27001, PCI DSS, EU 94/95 privacy rules, GDPR, and notably, SOC 2 Type 2. This commitment ensures customers' data privacy and allows them to focus on core business activities without worrying about security. Fivetran's Powered By Fivetran (PBF) solution enables end users to authenticate their credentials directly in Fivetran, reducing the time for building analytics products while maintaining high levels of security. The platform ensures data encryption at rest and during transit, as well as secure HTTPS connections for handling authentication processes. Overall, Fivetran's commitment to security compliance standards makes it a reliable choice for organizations looking to migrate to cloud-based technology.

The Modern Data Stack Conference, taking place on September 22-23, is a virtual event for data professionals of all levels. With over 40 sessions lined up, the conference will feature insights from accomplished executives and industry leaders discussing best practices, innovations, and technological developments in the field. Key presentations include "The Future of Security for the Modern Data Stack," "How DonorsChoose Uses Data to Transform Their Community," "Why Analytic Projects Fail and What to Do When it Happens," and "Breaking Your Data Team Out of the Service Trap." The conference aims to provide valuable learning experiences and networking opportunities for attendees.

Amazon Redshift offers several methods for exporting data to CSV files, each suited to different user needs and technical contexts. The UNLOAD command allows users to export data directly to an S3 bucket through SQL queries, with options to customize the output, such as adding headers or using different delimiters. Alternatively, users can leverage AWS SDKs, such as boto3 for Python, to programmatically interact with Redshift, execute queries, and manage the data export process. The AWS command-line interface (CLI) offers another means to run SQL queries and download results directly, while SQL clients provide a more traditional method for frequent interactions with Redshift. Each approach varies in complexity and setup requirements, offering flexibility depending on whether the task is a one-off or part of a regular workflow. For those seeking an even simpler solution, third-party services like Fivetran Activations can streamline data movement without the repetitive manual effort of exporting to CSV.

Fivetran has released its first SDK for leveraging the Fivetran API in Go, which is used to create a HashiCorp Terraform-verified provider for Fivetran. This integration allows users to manage and provision Fivetran connectors, destinations, and users within Terraform, an open-source project for writing and running infrastructure as code. The new Terraform provider can be used with commands such as init, refresh, plan, apply, and destroy to manage Fivetran accounts and services. Fivetran has also entered the HashiCorp Technology Partner Program and the Terraform Integration Program, making it the only automated data integration provider in the Terraform Registry.

Fivetran has released its first software development kit (SDK) for Go, a popular open-source programming language created by Google. The new SDK includes the Fivetran REST API and enables developers to access Fivetran services in a Go environment. The Fivetran Go SDK allows for user, group, connector, and destination management, making it useful for managing modern data stacks and facilitating change within enterprises as new data teams are onboarded. The SDK is currently in alpha and can be installed using the command "go get github.com/fivetran/go-fivetran".

The scientific method, a proven approach for successful improvement, can be combined with Business Intelligence (BI) to enhance businesses' prospects. By following the six steps of the scientific method - observation, background research, conjecture, predictions, experimentation, and analysis - businesses can make data-driven decisions that lead to better outcomes. BI tools such as Tableau, Microsoft Power BI, or Qlik, along with a data warehouse and ETL data pipeline, are crucial components in using the scientific method for finding insights within business data.

Database schemas are crucial in organizing and structuring data into tables with columns, keys, and interrelations. They play an essential role in data integration and analytics by ensuring that data is easily understood and can be transformed into dashboards and reports. A well-designed schema simplifies the work of analysts by removing cleaning and preprocessing tasks from their workflow, eliminating the need to reverse-engineer underlying data models, and providing a clear starting point for analytics projects. Database normalization helps overcome redundancy and lack of clarity in raw data by disaggregating or eliminating redundant, duplicative, or derived data values. To design an effective schema, one must understand the data source's underlying data model, create an entity-relationship diagram (ERD), apply a consistent naming convention, draft and populate tables from the ERD, and produce software to extract, preprocess, and load data into the destination. Automated data integration solutions can help leverage expertise in schema design without having to reinvent the wheel.

The Modern Data Stack Conference 2021 is a two-day virtual event on September 22-23, aimed at data professionals of all levels. It features sessions with insights from accomplished executives, investors, and professionals in the industry. Key topics include promoting data literacy, being data-driven when data isn't readily available, real-world analytics experiences, and accelerating analytics with a modern approach. With over 40 events scheduled, attendees can learn about best practices, innovations, and technological developments in the field of data.

Fivetran, a managed data integration service, has introduced a new security feature called customer-managed keys for its Business Critical users running on AWS. This feature allows organizations to control the master key used by Fivetran to encrypt credentials and temporary data. By managing their own encryption keys, customers can disable access to the key and stop Fivetran from syncing data at any time, enhancing security and control over sensitive information. The setup process is simple and involves configuring a key and role in AWS KMS and entering the key details into the Fivetran dashboard. This feature is available with Fivetran Business Critical, which offers industry-leading cloud data integration solutions for security, data privacy, and regulatory compliance.

The Modern Data Stack Conference offers a comprehensive lineup of 40+ sessions across four virtual stages, featuring prominent industry leaders such as Azure Data VP Rohan Kumar, Databricks CEO Ali Ghodsi, and Fivetran CEO George Fraser. Topics covered include insights as a service, reverse ETL, interactive data applications, self-service analytics, and more. Attendees can easily create their personalized conference agenda by registering for the event, accessing the schedule on the Modern Data Stack Conference site, clicking the calendar icon next to desired sessions, and adding them to their preferred calendar app.

The open-source tool dbt has revolutionized data modeling by enabling software engineering principles for data analysts, analytics engineers, and data engineers. Its reusability of code allows the creation of "best-practice transforms" for widely adopted data sources, leading to a library of standardized schemas and reusable code for all data teams. The three major dbt components that facilitate this reusability are packages, macros, and dependencies. Packages enable rapid extensibility by containing macros and models, allowing users to leverage work done by other data professionals within their projects. Macros leverage Jinja to create templates, enabling loops, modularization, and repetition of business logic. Dependencies are created through the ref function, which allows one model to depend on another being referenced, creating a dependent acyclic graph (DAG) of the entire data model. By leveraging these components, dbt can rapidly deploy modern data stacks for clients in days instead of months, ensuring that every company obtains the necessary data modeling to empower decision-making.

The Modern Data Stack Conference 2021, taking place on September 22-23, is a community event for data professionals of all levels. With over 40 sessions lined up, the conference will cover best practices, innovations, and technological developments in the field. Notable speakers include Jason Pohl from Databricks discussing the evolution of data warehouses into lakehouses, Korri Jones from Chick-fil-A sharing insights on prioritizing analytics projects during a pandemic, Tejas Manohar explaining how to build a Customer Data Platform (CDP) out of your data warehouse, and a fireside chat on product analytics with Mixpanel, Brooklyn Data, and Census. The full agenda is available for those interested in attending the conference.

A good SQL expert possesses the ability to manipulate data, particularly dates, to support various business operations and reporting needs. Different business functions, like finance and customer success, have varying requirements for date manipulation, which is crucial for comparing performance across time periods. The text explores five essential SQL DATE functions—DATE_TRUNC(), DATE_DIFF(), DATE_ADD()/DATE_SUB(), EXTRACT(), and CURRENT_DATE()—demonstrating their practical applications using a fictional company, Sandy Shores, which rents beach chairs. These functions help the company analyze rental periods, project returns, and group data for insightful business reporting. Additionally, the text suggests resources such as Leetcode and w3resource to further enhance SQL skills, encouraging readers to apply their knowledge and improve their proficiency in handling date-related data queries.

Fivetran Business Critical has integrated AWS PrivateLink to secure data traffic within a single cloud, avoiding exposure to the public internet. This feature is particularly beneficial for enterprises in highly regulated industries like healthcare and finance, as it ensures secure data movement between sources and destinations without violating security compliance requirements. With AWS PrivateLink support, Fivetran customers can connect their sources and destinations within AWS while maintaining ease of use, reliability, and agility. The feature is available on Fivetran Business Critical and offers several benefits such as eliminating the need for ongoing maintenance, enabling regulatory compliance, and reducing overall data stack infrastructure complexity.

Online Transaction Processing (OLTP) systems handle daily business operations such as sales, registrations, and updates while focusing on individual and unique elements of data. These systems are designed for efficient processing to ensure timely updates and immediate synchronization. OLAP systems, on the other hand, focus on aggregating data to provide analysis and use distributed computing to scale across large datasets. While OLTP is application-centric with row-oriented databases optimized for transactional performance, OLAP is database-centric with columnar data stores in data warehouses designed for analytical processing. The choice between OLTP and OLAP systems depends on the specific needs of a business, with OLTP being suitable for frequent updates and OLAP for reporting and summarizing large datasets.

MySQL and PostgreSQL are the two most popular open source relational databases, serving as backends for numerous applications. Both have been around for a long time and offer secure RDBMSes with support for clustering and network fault tolerance. However, they differ in several aspects such as architecture, data types supported, indexes supported, performance, security, and support. PostgreSQL is an object-relational database offering more complex data types and allowing objects to inherit properties, while MySQL is purely relational with support for 16 different storage engines suitable for various use cases. In terms of data types, Postgres offers a wider variety than MySQL, making it a better choice if dealing with unique or unstructured data. Both databases have comparable security options and support community-based forums, but MySQL also offers paid support plans from Oracle. The choice between the two depends on the specific needs of an application, such as its expected growth, complexity of queries, frequency of write operations, familiarity with developers, and third-party tool availability.

Digital transformation can greatly enhance financial analytics by integrating digital technologies into all areas of business operations. This process involves automating data extraction from various sources and centralizing it in a cloud data warehouse for efficient analysis. The modern data stack, consisting of an ELT data pipeline, cloud-based warehouse or data lake, data transformation tool, and BI or data visualization platform, enables financial analytics teams to access accurate, up-to-date, and standardized data. Automating data integration can improve revenue forecasting, enhance employee productivity, and facilitate high-value projects like dynamic process simulation and predictive performance analytics. Blend, a financial technology startup, has successfully automated its data integration process using Fivetran and Hightouch, resulting in significant efficiency gains and improved business-wide data analysis.

In a recent fireside chat titled "The Multi-Cloud Modern Data Stack," four industry leaders discussed the modern data stack (MDS) and its implementation. They agreed that multi-cloud is the best option for businesses, as it allows them to leverage different cloud services for various functions within the stack, leading to increased efficiencies. The panelists also highlighted the importance of interoperability, open source, and future-proofing one's organization by avoiding single technology lock-in. They emphasized the need for a common governance framework that manages data storage and avoids duplication while ensuring compliance. Additionally, they touched upon the role of emerging technologies like AI and ML in shaping the modern data stack. The panelists believe that multi-cloud will become a must-have feature for software applications, with exciting times ahead for the MDS landscape.