December 2022 Summaries

The Operational Analytics Book Club has published its second volume, covering chapters 4-7 of the book on The Fundamentals of Data Engineering. The authors emphasize the importance of focusing on abstract concepts rather than concrete tools and technologies to create value from data. The data landscape is evolving rapidly, with new tools and technologies emerging daily, making it essential for organizations to regularly evaluate their data stack and prioritize business needs. Containers have become a powerful operational technology, allowing developers to package up dependencies for software scripts to run on any machine without worrying about the details of said machine. The build vs buy debate highlights the importance of considering whether building custom solutions will create a competitive advantage or free up resources within the team. Data storage is a cornerstone of the data engineering lifecycle, with various types such as hard drives, solid-state drives, and random access memory being discussed. Understanding how these storage solutions work can bolster credibility as a data engineer, and block and object storage are recommended for specific use cases.

To unlock the potential of your customer experience, businesses must collect and utilize customer data in a structured and actionable manner. This involves collecting both quantitative and qualitative data, storing it in a single source of truth, and making it accessible to relevant teams. By leveraging this data, companies can build 360° views of customers, deliver answers faster, reduce friction, and improve personalization at scale. A data platform like Census helps make this data actionable immediately, driving revenue growth and better overall customer interactions.

Five years ago, Customer Data Platforms (CDPs) emerged as an all-in-one solution for collecting, unifying, and activating customer data, but they failed to meet the needs of most companies due to rigid data models, long onboarding times, and redundancies with other analytics and marketing tools. Now, the modern data stack and data activation have given companies access to best-in-class solutions for each component, leading to the rise of Composable CDPs as a more ideal solution for data-forward organizations. A Customer Data Platform is an all-in-one platform built for marketing teams to collect, transform, and activate customer data, but it has limitations, such as duplicating existing data and not being able to replace traditional data warehouses. The modern data stack already provides components of a CDP, making it possible for companies to turn their existing data platform into a Composable CDP by adding the data activation layer. This approach allows companies to leverage best-in-class tooling from their existing data platform, providing a true single source of truth for customer data, flexibility, better data governance, and future-proofing by design. The future is moving towards warehouse-native solutions, with data activation and reverse ETL democratizing access, positioning the data warehouse as the system of customer record that powers business operations.

This year's conferences saw a mix of in-person and online events, with a focus on data engineering, analytics, and marketing. Snowflake Summit in Las Vegas featured talks on operational analytics, which presented an opportunity to address common issues such as dashboards failing to meet business needs. The conference also highlighted the importance of data quality, definitions, and alerting systems. In contrast, Databricks' Data + AI Summit in San Francisco focused on technical architecture and showcased innovative approaches to data storage and processing. Summer Community Days, a virtual event with IRL happy hours, brought together data professionals for keynotes, workshops, and expert sessions. Hubspot's Inbound Conference in Boston explored how stakeholders felt about their data teams, revealing common complaints such as data teams being perceived as "too slow." Meanwhile, dbt Coalesce in New Orleans offered a platform for data engineers to share knowledge and experiences. The events showcased the growing importance of data professionals and the need for collaboration and innovation in the field.

The OA Digest Newsletter shared 125 resources in 2022, covering various forms of content including blogs, videos, podcasts, and events. The top 10 most popular resources were highlighted, featuring practical knowledge for data professionals to level up their careers. Key topics included SQL cheat sheets, best practices for analytics tools, and data modeling techniques. The newsletter also featured expert workshops and tutorials on SQL patterns, tracking plans, and more, catering to a wide range of data professionals.

Faced with the need for high-quality, reliable data, Prolific's data team built a tech stack from the ground up to create complete Customer Behavioral Profiles that enable actionable data to best engage with their customers. A high-quality, data-first culture is essential as it empowers marketing teams and informs marketing strategies by providing valuable insights on customer behavior and driving growth for businesses. Collecting customer data is the first step to unlocking rich, unified, and accessible behavioral data across an organization, which can then be used to eliminate unattributed conversions, create audience personalization, and implement ML-powered marketing and ad tech workflows. The process of setting up a powerful Behavioral Data Platform (BDP) like Snowplow takes time, but the rewards are well worth it as it provides a single source of truth for business users to reference, enables self-service data, and fuels business growth by empowering marketing teams with actionable insights. By choosing the right data warehouse according to current and future needs, organizations can avoid bottlenecks in their data team and unlock the full potential of their customer data.

Clockwise, a time orchestration platform, collaborates with its data team to create more impact by activating data in its business. The company's central data team creates core data infrastructure and curates product, billing, and financial data into a data warehouse for use by other teams. The business operations team works closely with the data team to onboard new data types, such as lead enrichment vendors, and integrates this data into their CRM platforms using tools like Census. Clockwise uses Snowflake for its data warehouse, dbt for data transformations, Metabase for data visualization, and Census to send data to its four CRM tools. The company faced challenges accessing real-time product engagement and usage data, but overcame these by setting up a reverse ETL tool like Census. To drive more personalized onboarding flows, Clockwise synced product segmentation data to HubSpot, enabling targeted communications with users based on their onboarding paths. The company advises others using data for go-to-market teams to clearly define the needs of end users and communicate effectively with the team generating the data.

This part 3.1 of the series evaluates reverse ETL tools, focusing on the difference between bundled and dedicated tools. It emphasizes four key considerations for evaluating a reverse ETL vendor: data connector quality, sync robustness, observability, and security & regulatory compliance. The guide highlights the importance of choosing the right tool to drive personalization, growth, and operational analytics, while also acknowledging that there are challenges in selecting a suitable reverse ETL vendor. It provides an overview of the two lanes of reverse ETL tools: bundled and dedicated platforms. Dedicated reverse ETL tools are considered more specialized and better suited for most companies, offering high-quality connectors, robust syncs, observability, and security features. The guide encourages readers to consider data connector quality, including breadth and depth, as well as sync characteristics such as reliability, validation, incremental syncing, scheduling, triggering, and speed. It also emphasizes the importance of observability features like alerting, integration with monitoring tools, detailed logging, usage audit logs, and real-time debugging. The guide concludes by highlighting five security considerations: data ownership, progressive regulatory standards, data encryption, API connector security, and governance plans.

Mike Johnston, Director of Business Analytics at fintech company Spreddy, has driven business impact by leveraging data analytics tools like Looker and Census to trigger personalized marketing messages based on product usage. He shares his journey into working in data, from teaching math to joining a finance operations team, and how he eventually landed at Spreddy. Johnston's first project at Spreddy involved implementing a modern data tech stack and using Looker to build out business rules. He now uses Census to pull product usage data into Salesforce and Pardot, enabling personalized messages based on customer behavior. This has had a significant impact on the business, allowing for more informed prioritization and operationalization of data, ultimately fueling pipeline revenue growth and customer retention. Johnston advises others in the data space to prioritize understanding business impact and working closely with customers and operations teams to identify opportunities for data-driven improvements.

The Census Segments update is part of Audience Hub, a suite of features designed to help marketers activate 360° data in their warehouse. A one-size-fits-all approach is no longer sufficient in the digital world where personalization is expected; segmenting target markets is essential to entice the right audience with the right message. The new version includes brand new visual segmentation, allowing marketers to build dynamic segments defined across related objects and events without code or SQL. This enables data activation, which unlocks data directly from the warehouse, providing a single platform for marketers to view 360° customer data, build complex segmentation logic, and send those audiences to all their tools. The update also includes features such as operationalizing first-party data without code, segmenting and activating in a single place, and turning ideas into action in seconds, not weeks.

This announcement from Census marks an effort to improve collaboration between data teams and business teams by providing a solution to define and expose business objects in a way that enables business teams to build segments and syncs without requiring manual code or SQL. The new feature, called Entities, allows data teams to create trusted models and share them with business users, who can then use them for segmentation and other purposes. Entities provide a central place to define data models and make them accessible to the average business user, reducing the need for manual upkeep and improving data governance and quality. This solution is designed to bridge the gap between marketing and data teams, enabling them to collaborate more effectively on data projects with confidence.

This debut edition of Prolific's Census series highlights data practitioners driving impact with Operational Analytics, specifically Jim Lumsden, the company's first data hire, who drove more impact by syncing key product analytics to Hubspot using modern tools like Census, Snowplow, and Redshift. As Data Lead at Prolific, Jim shares his journey from PhD in online research methods to building the hard data stack and hiring a data team to serve product needs. He emphasizes the importance of self-serve access to data for decision-making and showcases an impactful operational analytics use case with Hubspot, which has increased visibility and closed over 10 new big logos, allowing sales teams to work blind-free and focus on growth frontiers. Jim advises other small data teams to prioritize building Customer Behavioral Profiles early on and not trying to solve every problem at once, focusing on delivering value in one specific area rather than drowning in the demands of data.

Five years ago, Customer Data Platforms (CDPs) seemed like the right answer to the problem of accessing a unified profile of the customer. However, times have changed, and CDPs have not delivered on their promises. Instead, data-forward companies are switching to an alternative solution that costs far less and delivers value much faster: activating their data warehouse. A unified view of the customer is still the key challenge for marketers, but they know building one will enable them to run better campaigns with richer data to drive more ROI. However, CDPs have not lived up to their promises, with only 1% of companies meeting their current and future needs with CDPs. The main reasons why CDPs are not living up to their promises are that they don't replace data warehouses as the single source of truth for analytics, offer limited flexibility and control, and rely on engineering and long time-to-value. Instead, using a data warehouse as a customer data platform offers a true single source of truth, faster Time-to-Value, and more control and flexibility. The future of the CDP is the data warehouse, with vendors recognizing that the current CDP is incomplete and customers need to enrich and activate data from the warehouse. Warehouse-native data activation tools like Census are democratizing access to the data warehouse, making it possible for organizations to build granular segments and sync customer data to all their marketing and advertising tools without any code.

The Operational Analytics Club's book club has been a valuable resource for the author, who initially viewed it as a means to network with other data professionals but soon discovered opportunities for deeper connections through technical workshops, mentorship programs, and a book club. The book being discussed is "Fundamentals of Data Engineering: Plan and Build Robust Data Systems" by Joe Reis and Matt Housley, which emphasizes the importance of focusing on abstract concepts rather than specific tools and technologies. The authors argue that business needs should drive decisions regarding data, and reversible decisions should be prioritized to maintain agility and mitigate risks. They also discuss the concept of loosely coupled systems, which allows for modularity and plug-and-play solutions, making it easier to replace obsolete or unsuitable tools without significant downtime. The author has gained valuable insights from the book club and is excited to continue learning with the community.

Analytics engineers focus on the data itself, working on issues like data quality, freshness, and proper arrival time, owning the data pipeline from ingestion to visualization. They possess skills in data modeling, SQL, data warehousing, modern data stack tools like dbt, analytics dashboards, and are familiar with transactional database models. Data engineers focus on the data infrastructure of the company site and systems, working on data pipelines, focusing on tasks such as Python programming, DevOps, bash scripting, Git version control, orchestration tools like Airflow, Dagster, and Prefect. They typically work on backend processes to capture customer data, building custom data pipelines, and ensuring proper deployment of applications. When deciding which role is right for your team, consider the pain points you're trying to solve, such as data quality issues, capturing data on your website, establishing a single source of truth for your data, or building a custom data pipeline. If you need help with data quality, are having trouble collecting customer data, or want to establish a unified data view, an analytics engineer might be the best fit. However, if you're struggling with capturing data on your website, building a custom data pipeline, or ensuring proper deployment of applications, a data engineer could be the better choice. Ultimately, understanding the specific needs of your organization will help you make an informed decision about which role to hire.

Building and maintaining a custom reverse ETL (reversing the flow of data from downstream operational tools back into your data warehouse) solution can be costly, both in terms of labor and resources. Assuming an average annual salary of $200k for a data engineer, the initial cost to build a bespoke reverse ETL tool for three CRM connectors could range from $24k to $8k per engineer per connector, with ongoing maintenance costs that can balloon beyond initial estimates. In contrast, using a pre-built reverse ETL tool like Census can provide significant cost savings, including unlimited connectors and destination fields, scalability, and API quota management. Moreover, investing in a reverse ETL tool can free up talent to focus on higher-value work, such as unlocking business value through models and applications, rather than being bogged down by maintenance tasks. Finally, building a custom reverse ETL solution is a complex endeavor that requires specialized expertise and consideration of factors like data governance, write APIs, and organizational use cases.

The data warehouse is emerging as a single source of truth for marketing teams to do their job better and faster with customer data, but access to it has been historically limited to data teams who speak SQL. Warehouse-native no-code tools like Census enable marketers to unlock data directly from the warehouse without needing to know SQL, allowing them to access and activate customer data across marketing channels without engineering effort. This shift is bridging the gap between marketing and data teams, enabling more efficient campaigns and faster audience experimentation. The data warehouse has all the data, but prior to data activation, it was not accessible to marketers, whereas Customer Data Platforms (CDP) were designed to be highly accessible to marketers but lack the data and are therefore ineffective for data activation. With data activation, marketers can build segments during meetings, launch campaigns in 30 minutes, and remove false dependencies between marketing and data teams, making them more agile and enabling each side to spend more time on value-adding work.

Reverse ETL is a doorway that allows companies to step into the new era of Operational Analytics, leveraging insights to take action in day-to-day operations. With the increasing number of applications used by business teams, data engineers are struggling to keep up with pipeline maintenance and integration requests, leading to wasted skills and reduced trust in data. Reverse ETL enables personalization and growth while unlocking the data team's ability to do their best work, driving happier data teams and stronger customer dialogue. The new generation of data leaders needs to evaluate and implement these tools successfully to achieve operational excellence. Companies need better personalization and engagement to be competitive, and dedicated services like reverse ETL do a better job than point-to-point platforms in doing things "half-well." Reverse ETL allows companies to adapt quickly with the flip of a switch, empowering them with data-enriched context about customer behavior in real-time. This is particularly important for marketing teams who need fast access to data insights to power personalized messages and campaigns.

You'll encounter unexpected `<EOF>` syntax errors in Snowflake when the SQL compiler encounters an obstacle while parsing your code, such as an incomplete statement or missing parameters, causing it to terminate and display this error message, which can be confusing due to its ambiguous text; these errors can occur from most SQL commands run in Snowflake and can be fixed by adding missing required syntax elements, using the correct directory path with the PUT command, and ensuring all elements are present in the command line, as error messages are helpful tools for debugging processes.

In the 10 years since Harvard Business Review declared data science the sexiest job of the century, a significant demand for skilled data professionals has emerged. However, early-career data analysts often struggle to navigate their careers due to undefined career paths and limited guidance from senior data professionals. To address this issue, Jessica Cherny, a senior data analyst at Fivetran and founder of Data Angels, shares her expertise on building a successful data career through three key tips: building a personal brand, networking, and taking charge of one's career. By showcasing their work, curating relationships with senior data professionals, and actively seeking opportunities to grow, early-career data analysts can set themselves up for success in this rapidly evolving field.