July 2026 Summaries

Stitch, a managed ELT extraction tool, efficiently extracts data from various sources like Salesforce and HubSpot but does not natively support ClickHouse as a destination, requiring a workaround strategy for integration. To effectively use Stitch with ClickHouse, data must first be landed in object storage such as S3 or GCS before being loaded into ClickHouse, where users are responsible for setting up table engines and sort keys. This process involves using object storage as an intermediary, allowing ClickHouse to efficiently read data formats like Parquet, JSON, or CSV while enabling users to control the MergeTree design for data upserts. Stitch is praised for its source connectors, incremental replication, and ability to deliver data into supported destinations like Snowflake, BigQuery, and Redshift, but the gap in ClickHouse support necessitates a deliberate landing strategy. Users must account for schema drift, watermark logic, and freshness expectations, especially when integrating Stitch for product analytics, where a secondary hop into ClickHouse or a platform like Tinybird might be necessary for interactive analytics.

StreamSets and ClickHouse® integration involves designing pipelines that effectively translate data into analytical tables without forcing ClickHouse® to mimic MySQL operations. StreamSets is responsible for source capture, data transformation, and managing delivery retries, while ClickHouse® handles table engine configurations and aggregations. This setup facilitates efficient data processing by mapping pipeline records to ClickHouse® tables using either JDBC or HTTP clients, depending on the environment. While StreamSets manages data capture and delivery, Tinybird or similar solutions can be used for analytical serving with HTTP endpoints. The process requires careful consideration of CDC operations, batching, and pipeline configurations to ensure real-time data processing, with a focus on maintaining data integrity and minimizing lag. Control Hub and runtime considerations are crucial for seamless pipeline operations, including managing JDBC drivers and secrets, ensuring error handling, and preparing for upgrades. Additionally, field processors are vital for data type conversions, PII handling, and ensuring data consistency before storage in ClickHouse®.

Integrating Talend with ClickHouse® requires a thoughtful approach to job architecture, focusing on batch processing rather than frequent, small transactions. Traditional Talend job designs, suitable for databases like Oracle, often fail with ClickHouse® due to their inability to efficiently handle large inserts. Successful integration involves extracting data with watermarks, performing light cleansing, and writing in bulk using JDBC batches or NDJSON/Parquet files. Heavy aggregations should occur in ClickHouse® after data landing, rather than within complex Talend mappings. When JDBC is cumbersome, HTTP bulk inserts or file staging to object storage are viable alternatives. Handling Change Data Capture (CDC) requires a version column without relying on ClickHouse® for OLTP updates. During deployment, environment-specific details such as database credentials should be managed through contexts and secret stores to ensure smooth promotion across environments. Transformations should be strategically allocated between Talend and ClickHouse®, with repetitive analytical processes possibly offloaded to external tools like Tinybird for optimized performance.

Apache NiFi serves as an adaptable data movement system, while ClickHouse® functions as a high-performance analytical database, and their integration requires careful design to prevent performance issues like queue swelling and insert overloads. Effective integration involves selecting the right topology based on the data flow and processing needs, with three primary configurations: stream edge flow, file-based flow, and API platform flow, each suited for different enterprise scenarios. The focus should be on managing back pressure, setting practical defaults, and using tools like MergeContent for batching to optimize insert efficiency and avoid overloading ClickHouse®. Provenance tracking in NiFi is critical for replaying data during outages, and careful attention to attributes like batch IDs and Kafka offsets ensures data integrity. While NiFi can enhance data processing with features like multi-destination fan-out and protocol translation, it is essential to avoid anti-patterns such as excessive concurrent tasks or schema inference on every FlowFile. The use of Tinybird as a terminal sink offers an alternative for teams requiring HTTP-based data queries, maintaining a clear boundary between NiFi's routing and Tinybird's analytics capabilities. Proper validation and testing in a controlled environment are recommended before scaling up the throughput to ensure stability and reliability of the data flow.

Datadog and ClickHouse serve distinct purposes within the observability stack, with Datadog functioning as a managed SaaS platform that offers infrastructure monitoring, log management, APM, and dashboards through a unified interface, while ClickHouse operates as a columnar SQL database designed for high-volume event data storage and querying. Datadog is ideal for teams seeking an all-in-one solution with minimal setup, providing built-in agents, integrations, and a comprehensive UI for metrics and alerts. In contrast, ClickHouse offers greater flexibility through SQL analytics, allowing teams to perform arbitrary queries on raw data, making it suitable for scenarios where custom dashboards and product features are necessary. While Datadog's pricing is based on infrastructure footprint and event volume, ClickHouse's costs are determined by its deployment model, which can offer storage efficiency and cost predictability at scale. Teams often employ a hybrid approach, using Datadog for operational monitoring and ClickHouse for long-term analytics, especially when observability data needs to be integrated into customer-facing applications. Tinybird emerges as a managed service that leverages ClickHouse for observability, offering ingestion, schema management, and API endpoints without the complexity of infrastructure management.

A ClickHouse workshop provides developers with practical experience in schema design, data ingestion, queries, and pre-aggregation, preparing them for real-world analytics projects using this column-oriented SQL database tailored for OLAP workloads. The half-day session consists of five modules, each with specific exercises that utilize standard ClickHouse SQL and can be executed on a local ClickHouse instance, ClickHouse Cloud, or Tinybird's managed environment. Participants are expected to have prior SQL experience and can optionally use sample NDJSON event files for ingestion exercises. Key skills developed include designing a MergeTree table with appropriate partition and sort keys, ingesting events via batch processes and Kafka for production, writing optimized aggregation queries with PREWHERE and sampling, building and querying materialized views, and applying TTL for data lifecycle management. The workshop also discusses transitioning from this learning environment to production settings, emphasizing the need for ingestion pipelines, monitoring, and API layers, with Tinybird offering a managed solution that integrates these components and facilitates a smoother setup and execution process.

This ClickHouse tutorial provides a comprehensive guide for users familiar with SQL and row-oriented databases like Postgres, demonstrating the setup and utilization of ClickHouse, a column-oriented SQL database optimized for OLAP workloads. The tutorial walks through creating an append-only events table using the MergeTree engine, inserting sample data, performing aggregation queries, and setting up materialized views for pre-aggregated data to enhance query efficiency. It further explains the use of TTL rules for automatic data expiration and highlights common mistakes such as improper sort keys and inefficient data inserts. Additionally, the tutorial introduces Tinybird as a managed ClickHouse platform offering enhanced developer tools and real-time API endpoints, illustrating the integration of ClickHouse SQL within Tinybird Pipes for seamless deployment. The guide emphasizes foundational practices such as schema design, batch ingestion, and the use of materialized views, serving as a solid groundwork for any ClickHouse analytics project.

Postgres and ClickHouse serve different purposes in database management, with Postgres being a row-oriented OLTP database optimized for transactions and ClickHouse being a columnar OLAP database designed for handling large-scale analytical queries. While Postgres excels in managing application transactions with its support for ACID compliance and single-row updates, ClickHouse is more efficient in executing aggregation queries over vast datasets due to its columnar storage model and vectorized execution. Many teams use both databases in tandem, with Postgres managing everyday transactions and ClickHouse handling analytics, particularly when faced with high event volumes and the need for real-time analytics. Extensions like TimescaleDB and Citus enhance Postgres's analytical capabilities, but ClickHouse remains preferred for significant event volumes and low-latency analytical queries. The integration of systems like Tinybird enables seamless transition and management of analytical workloads without the overhead of operating ClickHouse clusters directly, offering solutions such as CDC through Kafka for real-time data synchronization. In essence, the choice between Postgres and ClickHouse hinges on whether to maintain analytics within Postgres or to transition to ClickHouse while leveraging Postgres for its transactional strengths.

Tinybird is transitioning its Free and Developer plan workspaces from Classic to the new Forward platform, with a migration deadline set for September 15, 2026. The Forward platform offers enhanced features for managing real-time data projects, such as deployments, automatic schema migrations, CI/CD, and typed SDKs, focusing on long-term project development and maintenance. Users are encouraged to migrate before this deadline to take advantage of early-migration offers and have more control over the transition process. The migration involves significant changes, such as replacing Classic branches and releases, updating CI/CD scripts, and introducing new testing models. While the transition is designed to be smooth, it is permanent, and users will not be able to revert to Classic once migrated. The migration guide provides detailed steps for preparation, compatibility checks, and executing the migration, ensuring a seamless transition to the Forward platform.

The DynamoDB Connector, now available in Tinybird Forward, allows users to integrate their DynamoDB tables with Tinybird to enable complex SQL operations like aggregations, joins, and window functions, which are not natively supported by DynamoDB. This integration provides real-time analytics capabilities by mirroring DynamoDB tables in Tinybird with minimal lag, allowing users to publish results as low-latency API endpoints while DynamoDB handles transactional operations. Two groups can immediately benefit: Forward users can incorporate DynamoDB tables into their existing project structures using the same Git repository and CI/CD pipeline, while Classic users can now migrate their DynamoDB-backed workspaces to Forward, overcoming previous limitations. The setup involves using an interactive wizard to generate necessary access policies and configuration files, which can be managed as secrets to maintain security. The connector supports local development and cloud branches, with options for sampling data to reduce AWS costs during testing. Additionally, TypeScript or Python SDK users can easily integrate and migrate their projects using the provided tools. The migration process from Classic to Forward is streamlined, preserving existing configurations while enabling new functionalities.

ScyllaDB's pricing structure is based on resource provisioning rather than per-request models, offering transparency in costs related to instance type, storage, service tier, and deployment model while excluding individual read/write charges. The service is available in three tiers—Standard, Professional, and Premium—each with varying support levels and additional features like multi-region active-active capabilities and bring-your-own-encryption options. Billing options include On-Demand, Subscription, and Flex Credits, catering to different workload predictabilities, and ScyllaDB Cloud supports deployment models such as Fully Managed SaaS and Bring Your Own Account (BYOA). ScyllaDB emphasizes its suitability for high-throughput workloads due to its provisioned-resource model, but for analytics, it suggests integrating with an OLAP engine like ClickHouse via Tinybird to avoid inefficiencies in handling cross-partition workloads. The platform also offers Change Data Capture (CDC) for real-time data ingestion into analytics layers, promoting a hybrid approach that leverages both operational and analytical database strengths. The pricing page highlights ScyllaDB X Cloud for elastic scaling and notes the inclusion of daily backups in its managed services, with additional backup storage incurring extra costs.

ClickHouse revolutionizes analytics database storage by employing a columnar storage model that enhances compression and maintains stable query performance even as datasets scale to billions of rows. It utilizes MergeTree engines to organize data into partitions, typically by month, allowing for efficient background merges and automated data deletion through TTL rules. This architecture supports sub-linear storage growth relative to event volume, making it effective for high-speed querying. ClickHouse optimizes storage through various compression codecs tailored to data patterns, such as ZSTD for general-purpose use and Delta for monotonic timestamps. Its partition design facilitates efficient data lifecycle operations, such as fast partition drops and targeted TTL expiration, while tiered storage policies manage data across hot and cold volumes, balancing performance and cost. Different MergeTree variants cater to specific storage needs, from raw event logging to pre-aggregated rollups, offering flexibility in storage patterns to suit diverse use cases. Whether self-hosted or managed, ClickHouse provides options for storage management, emphasizing compression, tiering, and lifecycle automation, with platforms like Tinybird offering fully managed services with automatic merge tuning and storage provisioning, ensuring efficient storage at scale without the operational burden.

Cockroach Labs outlines the pricing structure for CockroachDB Cloud on their website, offering three plans—Basic, Standard, and Advanced—each with different compute models, scaling limits, and feature sets. The Basic plan is designed for smaller, bursty workloads with a free tier including 50 million request units (RUs) and 10 GiB of storage per month, scaling down to zero when idle. The Standard plan targets steady workloads with up to 200 vCPUs, providing instant scaling and private connectivity. The Advanced plan supports high-scale applications with advanced security features, offering unlimited scaling and multi-region support across AWS, GCP, and Azure. CockroachDB Cloud is primarily positioned for transactional workloads rather than analytics, using request units and provisioned vCPUs for compute, while integrating changefeeds for streaming data to analytics systems like ClickHouse via Tinybird. The pricing details emphasize transaction-focused operations with potential additional costs for analytics, requiring users to consider their workload needs and possibly integrate separate analytics solutions.

SaaS analytics, unlike traditional BI reports, are integral to a product's infrastructure, requiring real-time, multi-tenant data handling without compromising on speed or data isolation. ClickHouse, an OLAP columnar database, addresses these needs through schema patterns, multi-tenant isolation, and pre-aggregation strategies, ensuring fast and secure data queries for SaaS applications. The use of ClickHouse involves creating append-only events tables with tenant-specific sort keys to streamline analytics, employing materialized views for pre-aggregated metrics, and managing ingestion paths via HTTP or Kafka for seamless data flow. Tinybird enhances ClickHouse's capabilities by providing managed services with built-in JWT authentication and API endpoints, enabling SaaS teams to efficiently deploy real-time dashboards with low query latency. This infrastructure transition allows SaaS products to offer real-time usage visibility, enhancing customer satisfaction and supporting business operations as a critical component of the application rather than as an optional feature.

Real-time analytics differs fundamentally from batch reporting by continuously processing data and delivering insights within seconds, rather than relying on periodic updates. ClickHouse is designed specifically for this real-time workload, utilizing columnar storage, vectorized execution, and append-optimized ingestion to efficiently handle continuous data streams and rapid query responses. Essential features for real-time analytics include low ingestion and query latency, continuous write throughput, and flexible aggregation capabilities, which traditional row-oriented databases struggle to provide at scale. ClickHouse addresses these needs through its architecture, including append-only event tables with optimized sort keys, diverse ingestion paths like Kafka and HTTP, and advanced aggregation techniques such as materialized views for pre-computation. Tinybird further simplifies the deployment of real-time analytics by managing infrastructure and offering tools for seamless data integration and querying, allowing teams to focus on analytical SQL rather than system maintenance. The shift to real-time analytics enables dynamic dashboards and metrics without the delays and complexity of batch processes, transforming how teams interact with and benefit from their data.

Postgres and Oracle Database are both robust enterprise-grade relational databases with distinct differences that influence database choice for engineering teams. Oracle Database is commercial software with substantial licensing costs, while Postgres is open-source under a permissive license, allowing for cost-free deployment on any hardware or cloud. Postgres stands out with its rich extension ecosystem, adherence to SQL standards, and reduced vendor lock-in, whereas Oracle excels with advanced enterprise features such as Real Application Clusters (RAC), sophisticated partitioning, and deep integration with Oracle applications. Both databases face limitations in handling analytical workloads due to their OLTP optimization, leading organizations to use Change Data Capture (CDC) patterns to offload analytics to systems like ClickHouse via platforms like Tinybird, which offers real-time analytics without building traditional data warehouse infrastructure. Ultimately, the choice between Postgres and Oracle hinges on licensing considerations, feature requirements, and the balance between operational and analytical workload handling.

Monitoring systems rely on a loop of collecting signals, detecting anomalies, alerting operators, and investigating root causes, with query performance often being the bottleneck. ClickHouse is highlighted for its ability to handle the high volume and low latency demands of monitoring data, encompassing event schemas, metric aggregation, log searches, anomaly detection, and alerting architectures. The text describes how monitoring data spans different signal types like metrics, logs, and traces, each with unique schema requirements that allow for efficient querying and partitioning. It also details infrastructure metrics aggregation for real-time anomaly detection, utilizing Z-scores and pre-aggregated tables for quicker alert evaluation. Additionally, advanced features like security event correlation and trace analysis are discussed, emphasizing their importance in real-time monitoring scenarios. Tinybird is introduced as a managed ClickHouse solution that simplifies ingestion, storage, and query serving, allowing for rapid query responses even at high data volumes. The post concludes that fast query performance in monitoring systems enables more precise alerting, comprehensive incident investigation, and seamless integration across monitoring tools, changing the approach from reactive to proactive monitoring.

Postgres and MariaDB are both open-source relational databases, each with distinct strengths and use cases. MariaDB, a fork of MySQL, offers compatibility with MySQL applications and features like the Galera Cluster for multi-master replication and the ColumnStore engine for analytical queries. In contrast, Postgres is known for its extensibility, rich type system including JSONB support, compliance with SQL standards, and a robust extension ecosystem that includes tools like PostGIS for geospatial data and pgvector for vector search. While MariaDB excels in environments needing MySQL compatibility and simpler operational transitions, Postgres provides superior capabilities for complex queries and a more extensive range of extensions. Both databases can benefit from integrating with external systems like ClickHouse via Tinybird for handling large-scale real-time analytics, as neither offers native columnar analytics at scale. The choice between the two often depends on specific needs such as JSON/document storage, geospatial data handling, and whether the environment requires MySQL compatibility or advanced SQL features.