August 2021 Summaries

Fivetran introduces its latest product tier, Business Critical, designed to help enterprises with highly sensitive data meet strict security and privacy requirements. The new offering provides the highest level of protection for sensitive data, enabling companies to create a secure and compliant modern data stack. Fivetran Business Critical supports various industry-specific data security and privacy regulations such as GDPR and compliance measures like HIPAA and PCI DSS Level 1. Key features include private networking with AWS PrivateLink, enterprise-level security for cloud migration, customer-managed keys, and PCI DSS Level 1 validation. This product aims to address the growing need for secure data integration in modern data stacks while ensuring compliance requirements are met.

The Modern Data Stack Conference 2021, taking place on September 22-23, offers a comprehensive program featuring over 40 sessions across four virtual stages. With speakers from leading businesses such as AWS, Netflix, and Snowflake, the conference aims to bring together data analysts, engineers, and leaders to discuss modern analytics' impact, challenges, and opportunities. The event will cover topics like the future of the modern data stack, best practices for building and maintaining a data stack, and how to create a data stack tailored to specific company types and growth stages. Registration is free and open now.

Fivetran has been recognized for the second consecutive year in the 2021 Gartner Magic Quadrant for Data Integration Tools, following its appearance in Gartner's "Market Share Analysis: Data Integration Software, Worldwide, 2020." During this period, Fivetran nearly doubled its revenue while the data integration market grew by 6.8% year over year. The company has added over 1,500 customers to its roster and is now considered a "vendor of choice" for enterprises looking to modernize their data infrastructure. Fivetran's focus on making data access simple and reliable has resonated with thousands of customers adopting its cloud-native managed service for data integration. The company manages over 50,000 connectors and more than one million schema changes every month, with guaranteed 99.9% uptime. Fivetran's commitment to continual innovation and product improvement has led to increased adoption of its fully managed data integration approach, allowing companies to focus on putting their data to work and increasing the speed of data-driven decisions.

Good database schema design is crucial for efficient data analytics and faster creation of reports and dashboards. A well-designed schema organizes data into tables with appropriate attributes, shows interrelationships between tables and columns, and imposes constraints such as data types. Normalization is a process that helps eliminate duplicate, redundant, and derived data values in a database. Entity-relationship diagrams (ERDs) are visual representations of schemas or entity-relationship models, representing the interrelations between tables. By understanding the underlying data model and normalizing the schema, analysts can make sense of the data and build effective dashboards and reports.

Achieving HIPAA compliance while maintaining an accessible data warehouse is crucial for US organizations handling sensitive personal health information. To balance accessibility and protection, follow certain practices to handle and exchange protected health information (PHI). Complying with HIPAA is a shared responsibility between your organization and the data warehouse provider you use. Key safeguards include administrative procedures, physical safeguards, and technical factors such as access control and encryption. Implementing these measures ensures the confidentiality, integrity, and availability of PHI in a data warehouse. Additionally, using data integration software like Fivetran can help maintain HIPAA compliance by identifying and managing PHI within the data pipeline.

The Modern Data Stack Conference will take place virtually on September 22-23, featuring speakers from Fivetran, Databricks, and more. With over 40 sessions, the event aims to explore how data can drive ideas and businesses forward. This year's conference expands from one to four virtual stages, covering topics for data engineers, analysts, team leaders, and product managers. Attendees will hear keynotes on the future of analytics, moral psychology in data persuasion, and Fivetran's upcoming product roadmap. Registration is open now.

The article discusses the pitfalls and best practices for maximizing the impact of data transformations. It advises against using ETL workflows, developing in a black box, relying on GUIs, and utilizing BI tools for transformation. Instead, it recommends using ELT workflows, promoting collaboration through tools like dbt Core and dbt Cloud, leveraging SQL for its flexibility and accessibility, and preparing data with specific transformation tools before visualizing it in a BI tool. Additionally, the article suggests accelerating analytics by adopting prebuilt SQL solutions, improving documentation across the organization to enhance data literacy, and joining communities of experts for additional support and best practices sharing.

Fivetran, a data integration company, has implemented a unique program called "Fivetran Research Rotation" where engineers are allowed to spend three months pondering one interesting technical problem in close collaboration with the CTO. This program is open to any Fivetran engineer worldwide and aims to enhance experimental research while providing an incredible creative and intellectual outlet for engineers. The rotation allows selected engineers to focus on one problem for several months, encouraging deep thinking and innovation. It also fosters a rich cross-pollination of ideas between the researchers and their home teams, ultimately making everyone better engineers and researchers.

Fivetran offers pre-built data models packages to accelerate analytics with near instant reporting. These packages are open-source dbt projects containing SQL logic designed to work out of the box with data ingested through Fivetran. The company develops and maintains data models for both single-schema and multi-schema sources, reducing time spent on transformations by data engineers and analysts. Users can leverage these packages to create clear, comprehensible tables for their reports, visualizations, and dashboards.

The article discusses the differences between data analytics and business intelligence (BI), as well as their applications in organizations. Data analytics is defined as analyzing raw data to make conclusions, while BI is a subset of data analytics focused on business problems and use cases. Data analytics includes complex pursuits like predictive modeling and machine learning, which are beyond the scope of BI. The article also highlights different levels of data analytics: descriptive (focusing on past events), predictive (forecasting potential outcomes), and prescriptive (advising actions based on analytics models). To make use of these tools, organizations need a multilevel data analytics software stack, including reporting tools like Google Data Studio, Microsoft Power BI, Tableau, and Looker. The article concludes by emphasizing the importance of having a fully managed data pipeline to populate a central analytics repository for better business outcomes.

Businesses face challenges such as reduced data team impact, extended time to insights, outdated data, and unreliable data, which affect their data transformation processes. SQL-coded transformations and graphical user interface (GUI) tools offer different approaches to address these issues. GUI tools, with their visual drag-and-drop components, are easy to access and offer wider visibility but can result in vendor lock-in and have limited capabilities. SQL scripting, being a universal language, provides flexibility and transferability but can be complex and slow to develop. Modern tools like dbt Core have emerged to modernize SQL by introducing advanced features such as testing, version control, and data lineage, helping to overcome traditional SQL limitations. Fivetran supports SQL-based transformations using dbt Core to provide automated, reliable solutions, offering prebuilt SQL logic to reduce entry barriers and accelerate insights.

Fivetran, a billion-dollar company with over 600 employees globally, is renowned for its collaborative culture and healthy work-life balance. The company's mission to make data access as simple and reliable as electricity has attracted engineers from around the world. Fivetran provides a supportive environment that encourages engineers to challenge themselves while working on interesting technical challenges. The company offers numerous data source connectors, empowering businesses with valuable insights. Additionally, Fivetran is recognized for its excellent workplace culture, earning several best employer awards in various regions.

As digitalization accelerates, fraud has reached unprecedented levels, with losses amounting to $42 billion in 2020, impacting nearly half of all companies. This situation highlights the necessity and efficacy of fraud prediction models, which leverage technological advancements like big data and modern algorithms. These models can be categorized mainly into profile-specific and transaction-specific types, focusing on user-level and transaction-level fraud, respectively. Additionally, fraud detection employs rules-based models, which rely on pre-set rules, and machine learning models, which automatically identify fraudulent patterns from large datasets. Key features for these models include transaction timing, location, cost-to-average spend ratio, and email information. Evaluating these models presents unique challenges due to the imbalance between fraudulent and non-fraudulent data, making accuracy an inadequate metric. Instead, precision and recall are recommended to measure the effectiveness of fraud detection. These metrics help balance the need to correctly identify fraudulent activities while minimizing false positives.

A recent survey by Fivetran reveals that data analysts are overworked, managing multiple projects across different departments. The majority of these professionals spend more than half their time on tasks other than analyzing data and uncovering insights. Data cleanup is the top-ranked project they would like to take on if given more time, indicating a lack of confidence in current processes. Other challenges include inadequate information to fulfill requests, poorly defined requirements, and too many requests. Potential solutions include data workshops, standardized request processes, and self-service options for non-analyst employees. Automation tools can also help data teams manage large volumes of data more efficiently.

Change data capture (CDC) is a mechanism that helps businesses stay competitive by keeping their data up to date. It identifies and captures changes in data, often automatically delivering detected changes from a source like a database to a target such as a cloud data warehouse. CDC enables organizations to make faster, more informed decisions based on accurate and timely data. Key benefits include improved data governance and compliance, fast and accurate data-driven decision making, enhanced change management and risk assessment, and operational efficiency gains. Fivetran offers CDC as a feature for most of its connectors to applications and all of its connectors to databases, using log-based replication or other mechanisms depending on the database type.

Fivetran has introduced AWS-specific deployment options for its customers, allowing them to run Fivetran services on Amazon Web Services (AWS) or Google Cloud based on their preferences and policies. This enables customers to meet IT, security, procurement, and legal requirements while also minimizing data movement costs between cloud providers. Additionally, Fivetran offers support for various geographies and maintains robust security capabilities and compliance certifications such as SOC 2 Type II, ISO 27001, HIPAA, and PCI DSS Level 1 validation. The shared responsibility model ensures that both AWS and Fivetran maintain industry-leading security and compliance standards.

Business intelligence (BI) is a set of technologies that convert data into actionable insights, enabling more informed strategic decision-making. It presents data as reports, visualizations, and dashboards, which may include graphs, maps, and textual summaries. BI helps managers correlate data from different sources to reveal trends. The modern data stack comprises data sources, a data warehouse, an ETL tool, and analytics software. To generate useful business intelligence, the first step is to replicate OLTP data into an OLAP platform like a data warehouse. Then, reporting software can be used to create reports, visualizations, and dashboards. While BI tools are excellent at presenting data, they cannot make decisions for users. Managers still need human expertise to consider budgets, workforce availability, organizational goals, and priorities when making strategic and tactical decisions based on the insights provided by BI tools.