June 2021 Summaries

Data pipeline performance is crucial for efficient data processing and can be improved through algorithmic optimization, parallelization, and pipelining. Algorithmic optimization involves using the best methods for computations to directly save costs. Parallelization splits simple processes into multiple queues that work simultaneously, reducing sequential request-response times. Pipelining separates data integration workflows into distinct stages with buffers between them, ensuring continuous work and improved performance. These techniques can be applied in sequence, starting with algorithmic optimization before considering parallelization or pipelining for further enhancements.

Two data experts, Bahadir Sahin of Onfido and Igor Chtivelband of Billi, discussed their journeys towards better customer engagement at the recent Modern Data Stack Conference EMEA. Both companies operate in niche B2B markets where maximizing relationships with every prospect and customer is crucial. The main obstacle to a 360-degree customer view, according to Igor Chtivelband, is siloed data. Bahadir Sahin emphasized the importance of developing a consolidated view of touchpoints as customers engage with a business. Both companies are working towards creating comprehensive customer views and improving onboarding processes. Onfido uses a "health score methodology" to keep customers satisfied and catch them before they leave, while Billi aims to identify issues early and alert account managers for increased retention.

Fivetran has acquired new technology for database replication that combines the benefits of snapshots and log-based systems while avoiding their drawbacks. This acquisition extends Fivetran's existing replication capabilities, offering a simpler and faster alternative to traditional methods. The new approach, called Teleport Sync, is less intrusive than current log-based solutions and captures all data without requiring specialized security access or highly customized scripts. It also improves throughput without needing network engineers or extra/load query analysis. Fivetran's commitment to innovation aims to make access to data as simple and reliable as electricity, positioning the company as a leader in ELT solutions.

Fivetran has experienced rapid growth over the past few years, expanding from a small Oakland startup to a global business with over 600 employees. Throughout this expansion, the company has remained committed to creating an inclusive workplace culture that allows its employees to take risks and thrive professionally without hiding their authentic selves. This commitment is reflected in Fivetran's numerous best-employer awards from multiple countries. The company's workplace culture is built around several key principles, including seeing every employee as a leader building a career, building a company culture around core values, making diversity, equity and inclusion a priority, supporting ERGs, continually improving benefits based on experience and dialogue, and listening to their people. These principles have been successfully implemented in various regions such as India, Europe, and North America, leading to positive experiences for Fivetran employees. The company is currently hiring for positions across all departments and regions.

In today's business landscape, strong analytics are crucial, with SQL proficiency being vital across various data-related roles. A specific SQL concept, window functions, is highlighted as essential for advanced analytics and deeper insights. Unlike aggregate functions that group result sets, window functions provide aggregate values while maintaining the number of rows in the output. The article explores five key window functions beneficial for business operations: ROW_NUMBER() for identifying specific records, SUM() for cumulative sums, AVG() for moving averages, LEAD()/LAG() for comparing consecutive periods, and DENSE_RANK() for ranking data. These functions enable sophisticated data analysis, facilitating tasks such as tracking customer purchases, calculating growth metrics, and ranking top customers. Understanding and utilizing these window functions can significantly enhance one’s data analysis capabilities, with further learning resources available for those interested in expanding their SQL skills.

Video analytics involves using machine learning to automatically identify patterns and events in video content, such as detecting abnormal behavior or monitoring traffic flow. This technology has various applications, including security and surveillance, transport monitoring, healthcare, and user-generated content moderation. The main technologies behind video analytics include video processing, object detection, object recognition, and tracking. However, organizations implementing video analytics face challenges related to data drift, complexity of tools, and data storage. To address these issues, continuous training and algorithm updates are necessary, managed video solutions can be adopted, and cloud-based storage services can be utilized.

At the recent Modern Data Stack Conference EMEA, three analytics leaders discussed their experiences transitioning from legacy architecture to modern data tools and practices. Key insights include prioritizing good governance, treating data as a product, getting buy-in from across the business, focusing on quick wins and fostering organization-wide data literacy. Automation and scaling capabilities were identified as significant advantages of modern data stacks over traditional on-premise solutions.

The data lakehouse is an emerging technology that combines elements of data warehouses and data lakes to offer the best of both worlds. Data warehouses are structured repositories suitable for analyzing organized data, while data lakes can hold unstructured data but lack schema enforcement. A data lakehouse supports both structured and unstructured data, offers ACID transactions, and runs on cloud platforms with high scalability. It is too early to determine if the data lakehouse will replace traditional architectures, but many vendors are moving in that direction.

In honor of Pride Month, Fivetran's employee resource group has organized various events and meetings. Travis Hatfield, the company's Global People Operations Manager, shares his perspective on what Pride means to him, celebrating unsung heroes in the community like the San Francisco Lesbian and Gay Freedom Band, and how he plans to celebrate this year amidst the pandemic. He also discusses his experience at Fivetran and his interest in events management if not working in human resources.

Companies today face the challenge of decentralized data due to the boom in cloud "as-a-service" offerings. Many companies use multiple software packages, creating a multitude of different places where users must go to get their data questions answered. Fivetran offers over 150 connectors to applications and cloud platforms commonly used by modern-day companies, helping to normalize and transform data from various sources into one shared data language stored in a data warehouse like Snowflake, Amazon Redshift, Google BigQuery, or Databricks. This enables cross-platform reporting and the ability to combine Facebook, Google, Twitter ad conversion data with Marketo campaign data set for a unified view of business operations. Salesforce Customer 360 is an all-in-one unified platform that combines various apps like Marketing Cloud, Sales Cloud, Service Cloud, and Commerce into one integrated solution, providing a 360-degree view of customers. Fivetran can also help gather data for ERP, HCM, and collaboration tools to combine all datasets for the sake of BI and reporting.

MongoDB is a popular NoSQL database used by many organizations due to its document-centric approach and ability to accommodate massive scale. It is particularly well suited for use cases such as handling big data, rapid prototyping, caching and high availability, streaming feeds, and content management and cataloging. MongoDB's distributed nature allows it to be easily scaled horizontally across multiple servers, making it a cost-effective solution for businesses with varying data demands. Fivetran has developed a connector for MongoDB to help organizations unlock the value of their stored data by replicating it to their company's data warehouse.

This article discusses the differences between SQL (relational databases) and NoSQL databases, explaining that each type is well-suited for different use cases. SQL databases are good for online transaction processing (OLTP), while NoSQL databases work well for prototyping, applications running across geographically distributed data centers, mobile applications handling big data, and real-time response rates. There are four main types of NoSQL databases: document stores, key-value stores, graph databases, and wide-column stores. The article concludes that organizations have a variety of software tools at their disposal to meet specific needs, and it's less about choosing between SQL or NoSQL but more about when to use each type depending on the use case.

The Modern Data Stack Conference EMEA discussed how cloud technology is revolutionizing data architecture and analytics, making it as simple and reliable as electricity. Automated data integration and cloud data warehouses are driving advances in analytics by allowing organizations to easily centralize data from various sources and offering low-cost storage with dynamically scalable compute. The modern data stack democratizes data access, enabling teams to work with different sources without specialized engineering expertise. Cloud-based analytics offers back-to-basics simplicity, helping organizations quickly access data for insights. The modern data stack accelerates new business models and opens up market opportunities by allowing more organizations to monetize their data and spurring advances in virtualization.

The article discusses the importance of developing a data culture in marketing departments as the use of data becomes increasingly vital for success. It highlights the challenges posed by the disappearance of third-party cookies and emphasizes the need to centralize first-party data in cloud data warehouses. The article also outlines essential building blocks for creating a data culture, including defining goals, assessing current data practices, leadership support, celebrating wins, stamping out silos, facilitating skills development, ensuring accurate marketing data, and implementing a modern data stack.

The article compares popular data integration tools, focusing on cost, security, reliability and number of supported sources. It highlights Fivetran as a reliable choice with 150 data sources and 10 destinations, including Redshift, BigQuery, Microsoft Azure Synapse, Snowflake, Microsoft SQL Server, PostgreSQL and MySQL. The article also compares Fivetran to other ETL providers such as Informatica, Matillion, Xplenty, and Stitch, emphasizing the advantages of Fivetran in terms of reliability, compliance, and pricing model. It suggests that users should test each tool with their company's live data and consider factors like documentation, training services, implementation help, and handling of data source schema changes at the destination before making a decision.

Business Critical Fivetran (BCF) is a new offering designed to provide secure, efficient, and performant data integration for Snowflake customers. It aims to address the unique security requirements of enterprises by automating their data pipeline process while ensuring data remains secure throughout its journey. The BCF plan includes support for AWS PrivateLink, customer-managed encryption keys, PCI DSS Level 1 validation, and additional security measures across all Fivetran plans such as ISO 27001 certification, SOC 2 (Type 2) certification, GDPR and HIPAA compliance, and stringent data encryption protocols. This new plan is a perfect complement to Snowflake's own Business Critical Edition, allowing enterprises with extremely sensitive data to create secure, automated data pipelines into Snowflake.

Frustration often arises when sales opportunities are missed due to underutilized data, especially when important details from other systems are not integrated into Pipedrive, a sales CRM. Custom fields can be created using the Pipedrive API to enhance sales processes by incorporating additional data like trial end dates, which are not available in default fields. By connecting to the Pipedrive API with Python, users can import necessary libraries and check existing fields to create and write data to a new custom field if needed. Writing data to custom fields is straightforward, but ensuring the process is flawless in production involves scheduling, adhering to rate limits, incorporating logging for successful and failed data points, and developing a retry mechanism for failures. While managing these technical challenges can be daunting, services like Fivetran Activations offer to handle custom connector engineering, simplifying the sync of customer data from warehouses to business tools.

Shauna Healy, a mother and Sales Development Representative at Fivetran, shares her experience transitioning from child care to tech in search of better work-life balance. After working long hours in child care, Shauna found that company culture was more important than industry when it came to supporting a healthy work-life balance. She praises Fivetran's flexibility and understanding, which has allowed her to spend more time with her family while still excelling at work. Shauna also highlights the support network available for career progression within the company.

Fivetran CEO George Fraser predicts the evolution of modern data stacks over the next five years towards becoming a hybrid analytical-operational system, with more companies serving production use cases directly from their data warehouses. He also anticipates an increase in interfaces focusing on individual data entities and their relationships rather than aggregated data. As the modern data stack develops, analytics teams will adopt practices like testing and continuous integration, similar to engineering teams. Business users with varying levels of data literacy should be accommodated by self-service systems or analyst involvement for agile decision making.

Marketing analytics offers numerous benefits such as providing an integrated view of marketing data, enabling campaign optimization through A/B testing, facilitating brand awareness analysis, improving customer segmentation, delivering relevant information for each stage of the customer journey, revealing important touchpoints with attribution modeling, and offering insight into cost of acquisition. By embracing analytics, marketers can make smarter decisions, become more proactive, and achieve maximum efficiency and impact in their marketing efforts.

ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) tools are crucial for moving data from various sources, such as SaaS applications and databases, to locations where data processing and analytics can occur, like data warehouses or data lakes. These tools require ETL connectors, which establish connections to data sources, enabling data extraction and loading. The number of supported connectors in an ETL tool can indicate its versatility, but it is essential to evaluate how well-suited the tool is to your organization's needs. Businesses commonly use numerous applications, often beyond IT-sanctioned ones, leading to the need for a comprehensive inventory of all software in use. This includes CRM, ERP, marketing automation, and other critical data sources like Salesforce, Google Ads, and analytics tools. Organizations should identify these tools' most important data sources and ensure that the chosen ETL platform supports them. Additionally, determining the right cloud data warehouse is crucial before selecting an ETL tool, as it impacts the data replication process. Once the list of data sources and destinations is established, businesses can evaluate ETL platforms through trial accounts and case studies to ensure they meet organizational needs.