January 2023 Summaries

The pervasive metaphor of data as oil has significantly influenced the way organizations approach data management, often limiting them to the extraction and refinement of existing data sources. This approach can restrict data scientists, as they often find the necessary data to be unavailable or insufficient in quality, scope, and structure for advanced analytics and machine learning. Consequently, organizations should consider deliberately creating high-quality data tailored to their specific needs, rather than solely relying on existing data sources. Such proactive data creation, as exemplified by certain financial and insurance companies, can enhance predictive modeling and provide a competitive edge. Although the current tooling for data creation is not as advanced as extraction tools, the potential benefits of creating tailored datasets underscore the importance of investing in better data creation processes. Adopting a mindset beyond the "data is oil" metaphor could unlock greater value by empowering organizations to develop data platforms that support future applications and insights.

Data Contracts have emerged as a contentious yet promising concept in the data community, aiming to provide clarity and stability in data exchange by formalizing agreements between data producers and consumers regarding data structure, semantics, and usability. The concept has sparked debate, with proponents arguing that Data Contracts can prevent disruptions in data applications by ensuring changes in data sources are communicated and managed effectively, while critics contend that enforcing such contracts is impractical and burdensome for data providers. The discussion highlights the challenges of relying on extracted data from sources like Salesforce, which are often volatile and difficult to manage, versus the potential of deliberately created event-based data that adheres to Data Contracts. Event-based data creation is seen as a more robust approach, as it allows for intentional data emission governed by contracts, ensuring data integrity and reducing the likelihood of breaking downstream applications. While Data Contracts are criticized for their perceived complexity, they are increasingly recognized as essential for organizations aiming to build reliable, data-driven applications by encouraging the deliberate creation and governance of data.

Snowplow has announced the acquisition of Poplin Data, a Sydney-based data-specialist consulting firm, as part of its international expansion efforts, establishing the Snowplow APAC Operating Hub. This acquisition follows recent expansions in Boston and Berlin and aims to enhance Snowplow's capabilities in the APAC region, particularly in media and retail marketing solutions. Poplin Data brings expertise in algorithmic attribution and extensive experience with Snowplow's platform, which will enhance Snowplow's data modeling capabilities and accelerate the delivery of Data Product Accelerators. The collaboration is set to advance data maturity for clients by focusing on first-party data collection, privacy-compliant practices, real-time data collection, identity merging, and data completeness. This move is expected to make Snowplow's behavioral data solutions more accessible to a global audience, aligning with its vision of expanding data creation possibilities.

Open-source analytics tools offer various functionalities across categories such as behavioral data platforms, product analytics, A/B testing, customer data platforms (CDPs), data validation, analytics engineering, anomaly detection, databases, and data visualization. Snowplow, a prominent behavioral data platform, emphasizes data ownership and integration with cloud data warehouses, while Countly and PostHog provide mobile and self-hosted product analytics, respectively. Wasabi serves real-time A/B testing but lacks recent development, and Grouparoo facilitates data integration with cloud-based tools. Great Expectations and dbt enhance data validation and transformation workflows, though dbt focuses solely on transformation. Anomaly detection is covered by Hastic, which integrates with Grafana. For databases, Apache Druid and Timescale cater to real-time and time-series data needs. Visualization tools such as Superset, Metabase, and Redash offer diverse querying and visualization capabilities, though they vary in complexity and setup requirements. The adoption of open-source tools allows users to build customized data stacks, benefiting from community support and flexibility.