June 2025 Summaries

Tinybird has introduced compute-compute separation for populate operations, significantly enhancing how resource-intensive data transformations and backfills are managed within its platform. This architectural change eliminates resource contention by assigning dedicated ClickHouse instances for populates, ensuring that heavy operations do not compete with live queries and analytics workloads, thereby maintaining consistent response times. This separation allows for ephemeral, right-sized resource provisioning, reducing costs associated with over-provisioning and aligning infrastructure use with actual demand. Additionally, it enhances reliability by preventing failures due to resource exhaustion, allowing populate operations to scale to their full resource needs without impacting production environments. When a populate operation is initiated, Tinybird automatically provisions the necessary resources, which are decommissioned upon completion, ensuring users only pay for actual usage time. This feature is available to all clients via an opt-in model, providing them control over adoption and configuration to suit their specific needs.

At Tinybird, real-time analytics depend heavily on efficient data ingestion processes, particularly with the conversion of JSON data to ClickHouse®'s RowBinaryWithDefaults format, which permits missing data fields to be replaced by defaults. Originally implemented in Python with some C helper functions, the conversion process had limitations in performance and maintainability, especially as it duplicated ClickHouse®'s internal encoding functions. By leveraging ClickHouse®'s internal functionalities and adopting a new approach that included writing C++ code to handle JSON to RowBinary conversion, Tinybird aimed to improve performance while maintaining reliability and flexibility. This transition involved creating a JSON Path tree for efficient data extraction, using the fast simdjson parser, and addressing numerous conversion quirks that could impact existing customer workflows. The new implementation has resulted in a significant decrease in CPU usage, improved reliability by documenting ingestion quirks, and better alignment with ClickHouse®'s internals, although further work remains to fully optimize performance and handle legacy data conversion.

Tinybird's Forward deployments have evolved to enable efficient updates to data pipeline schemas without downtime by optimizing the migration of data. Initially, the algorithm required migrating all data, which was inefficient and time-consuming, especially for large data sources like a 14TB Kafka table. The initial approach was to migrate everything, creating auxiliary tables and using UNION views to ensure data availability during transitions. However, this proved cumbersome and didn't scale well. Recognizing this, Tinybird refined the process in three phases to dramatically reduce unnecessary data movement by focusing on smart migration triggers and isolating chain migrations to only the affected parts. The key was identifying the most upstream change in an ingestion chain, migrating from that point downstream, and employing cross-version bridging to maintain real-time data flow without migrating unchanged upstream tables. These optimizations significantly decreased deployment times for large workspaces, transforming the process from days to minutes. Looking forward, Tinybird aims to further enhance the deployment algorithm by avoiding downstream migrations when changes are purely additive and implementing TTL-based migration skipping for data sources with short time-to-lives, continuing to refine the process to only migrate absolutely necessary data.

Tinybird leverages Karpenter, an open-source Kubernetes cluster autoscaler, to optimize infrastructure scalability, reliability, and cost-efficiency by automating node provisioning based on workload demands. This approach, combined with AWS services like EKS and Spot Instances, has reduced manual intervention and AWS costs by 20%, with up to 90% savings on CI/CD workloads. Karpenter's intelligent management allows for seamless handling of high-volume data ingestion and spikes, improving application availability and reducing operational overhead. The platform further enhances scalability using KEDA for real-time autoscaling based on external metrics, facilitated by Tinybird's analytics capabilities. This setup prioritizes service resilience through architectural strategies like separate node pools for critical and stateless workloads, node diversity, and optimized AMIs like BottleRocket. By deploying Karpenter on AWS Fargate, Tinybird ensures continuous availability and operational independence, significantly reducing node and pod startup times. The integration of these technologies has enabled Tinybird to focus on delivering customer value, with a scalable and agile infrastructure that is prepared for rapid growth and unexpected demands.

Tinybird's platform faced a significant scaling challenge when one of their largest customers' data volumes began exceeding the capacity of a single ClickHouse writer instance, which necessitated the development of a multi-writer ingestion mode to distribute the load across multiple instances. This new mode, designed for real-time data ingestion, allows specific tables or workspaces to be routed to different ClickHouse instances within the same cluster, aiming to alleviate pressure on individual writer instances and ensure cluster stability. By opting for a static routing approach with Varnish load balancer and a custom Varnish extension, Tinybird achieved a system that prioritizes simplicity, reliability, and predictability, enabling them to manage high-throughput data ingestion more effectively and scale with customer demands. The implementation has been rigorously tested and gradually rolled out to larger customers, successfully managing peak data loads and demonstrating the platform's capacity for future growth.

Tinybird developed a custom autoscaling system leveraging Kubernetes Event-Driven Autoscaling (KEDA) to address the unpredictability of real-time analytics workloads, which can experience up to tenfold traffic spikes during events like product launches. Traditional autoscaling methods, based on CPU and memory metrics, were inadequate due to their reactive nature and latency in scaling decisions. By integrating KEDA directly with Tinybird’s real-time metrics API, the company eliminated the delays associated with Prometheus-based metrics scraping and instead scaled based on live ingestion data. This approach allowed for more responsive and reliable scaling by focusing on critical metrics like Kafka lag, which better reflected actual data processing demands. The system adapted to varying traffic patterns and ensured stability by tuning stabilization windows and combining multiple triggers, such as custom metrics and CPU utilization. This overhaul not only optimized infrastructure performance and cost-effectiveness but also provided a feedback loop that enhanced the overall product's reliability and usability.

Tinybird Forward has expanded its data export capabilities, introducing sinks for Amazon S3, Google Cloud Storage, and Kafka, enabling users to export processed data from Tinybird pipes to these external platforms either on a schedule or on-demand. Sinks are extensions of Tinybird pipes, allowing the results of SQL queries to be written to files or streams rather than exposed as API endpoints. These sinks support various file formats, including CSV, NDJSON, and Parquet, with options for advanced file partitioning and compression tailored to the needs of downstream systems like data warehouses and analytics tools. The new functionality includes enhanced partitioning strategies that facilitate efficient data organization, such as creating Hive-style partitioned structures for compatibility with tools like Athena, BigQuery, or Spark. Execution of sinks can be managed through cron expressions for scheduled runs or triggered manually for on-demand operations, with comprehensive logging and monitoring features to track performance and data transfer costs. These sinks are available to users on Developer and Enterprise plans, with detailed setup guides provided for each destination type.

Tinybird has announced full Windows support for its CLI, addressing previous compatibility issues that left Windows developers unable to use the tool effectively. Initially, the onboarding script, which used a curl command to download a .sh file, was incompatible with Windows systems. To resolve this, a new PowerShell installation script was created, allowing Windows users to install and configure the Tinybird CLI seamlessly. The development team tackled technical challenges such as file path handling, browser authentication on Windows Subsystem for Linux (WSL), and file upload path normalization to ensure cross-platform compatibility. With these updates, Windows developers can now enjoy the same functionality and user experience as macOS and Linux users, facilitating the deployment of projects directly from Windows environments. This enhancement aligns with Tinybird's mission to broaden its developer community and maintain a consistent CLI-first approach across all operating systems.

Tinybird has launched Forward, a new version focused on a CLI-centric experience with AI-powered features, while maintaining its high-performance analytics infrastructure. This shift from the UI-guided Classic version required a reevaluation of the onboarding process to improve user retention. The initial approach of using a command to create a complete project proved overwhelming, leading to a more focused onboarding sequence that starts with creating a data source. New CLI wizards guide users through setting up data sources and connections, with added support for Windows users and improved documentation that separates content for Forward and Classic. The company continues to refine its onboarding and documentation based on user feedback, as part of ongoing improvements introduced during Launch Week.

The Tinybird OpenTelemetry Exporter is a new tool that allows users to stream telemetry data, including logs, traces, and metrics, directly into Tinybird data sources using the OpenTelemetry framework, which is widely recognized for its vendor-neutral observability capabilities in distributed systems. This exporter simplifies the process for existing Tinybird and OpenTelemetry users by eliminating the need for additional code or steps, and it offers compatibility with ClickHouse® exporter formats, enabling easy migration from other backends. Officially integrated into the OpenTelemetry Collector Contrib repository as of version v0.131.0 and marked as alpha, it supports real-time data streaming into Tinybird by configuring the exporter with specific API host details. The integration allows users to connect Tinybird with Grafana for enhanced observability and to build custom analytics APIs, such as those for performance benchmarks or operational monitoring. The tool facilitates the creation of real-time analytics applications and is designed to support a variety of use cases, including tracking website performance and building custom application performance monitoring solutions.

Birdwatcher is an AI agent integrated with Tinybird that facilitates data exploration, monitoring, and analysis through Slack, offering a conversational interface for querying data directly from the messaging platform. Users can install the Birdwatcher Slack App and run the /birdwatcher-config command to connect it to their Tinybird workspace, allowing them to use natural language to access organizational metrics and data insights. The app functions by routing Slack event subscriptions to the Birdwatcher agent, which processes requests using the Tinybird Remote MCP Server. Birdwatcher also supports hybrid mode, combining notifications with interactive conversations to monitor and address issues, such as CPU spikes in Tinybird clusters, using GitHub Actions for deployment. This setup enables users to receive alerts and conduct further investigations by interacting with the Birdwatcher agent in Slack, leveraging tools like the explore_data feature for detailed data analysis.

Tinybird has launched the MCP Server, a remote, hosted service that enables AI agents and large language models (LLMs) to directly connect to Tinybird workspaces, making real-time data LLM-ready without requiring additional infrastructure. This server facilitates analytics and generative UI applications by exposing Tinybird resources through language-understandable by LLMs, allowing agents to explore data, generate SQL queries, and access API endpoints. The MCP Server is secured using token-based authentication, ensuring access control and data protection, particularly in multi-tenant environments. It supports Streamable HTTP as the transport protocol, allowing seamless integration with various agent frameworks and SDKs. The service offers tools like explore_data and text_to_sql, enhancing agents' ability to perform complex data queries and analysis while maintaining observability by tracking tool usage through Tinybird's service data sources. By offering deterministic API endpoints alongside text-to-SQL tools, the server simplifies interaction with data, addressing LLMs' challenges in generating valid SQL and enhancing efficiency in data-driven applications.

Birdwatcher is an AI agent designed to streamline data exploration, monitoring, and real-time analysis in Tinybird, functioning autonomously, on a schedule, or interactively through a CLI Slack bot. By leveraging the Agno framework, Birdwatcher is equipped with multiple LLM backends and a tool integration system, enabling it to perform complex data analysis tasks using Tinybird's MCP Server. The agent can be customized via specific missions and prompts, allowing it to conduct tasks such as web analytics, cluster health monitoring, or data exploration by understanding data schemas, generating SQL queries, and providing analytical responses. Birdwatcher's adaptability is further enhanced by integrating with Slack for notifications and GitHub Actions for scheduled operations, demonstrating its capability to address domain-specific problems effectively. This setup allows users to quickly develop analytics agents suited to their needs, promoting efficient data-driven decision-making.

Deciding between using Model Context Protocol (MCP) and traditional APIs for building AI agents involves understanding their respective strengths and scenarios of application. MCP acts as a universal adapter that allows AI systems to autonomously discover and use external services through natural language, adding a conversational layer to existing APIs. It excels in rapid prototyping, dynamic tool selection, agent autonomy, and multi-tool workflows, making it suitable for scenarios where AI needs to reason independently. Conversely, direct API calls are preferred for deterministic operations, high-performance, and real-time requirements due to their efficiency and security in regulated environments. A hybrid approach combining MCP for flexible, on-the-fly tool use and APIs for efficient, bulk operations is often the most effective strategy. The rise of MCP highlights the need for robust, well-documented APIs designed with AI consumption in mind, as it enforces consistency and allows for precise control over agent operations. At Tinybird, the focus is on creating low-latency, secure, and scalable platforms to support the development of data-intensive AI agents, emphasizing real-time access and robust security features.

Apache Iceberg is a robust tool for high-performance analytics, particularly excelling in batch processing and complex ETL scenarios, but its adaptation for real-time analytics requires careful consideration of its architectural trade-offs. Engineers often make mistakes by neglecting fundamental principles like partitioning and sorting, optimizing without understanding, and assuming more features equate to better performance. Effective partitioning, sorting, and compaction are essential for maximizing Iceberg's capabilities, with partitioning strategies tailored to specific query patterns and sorting designed to enhance data retrieval efficiency. While Iceberg can handle batch analytics with infrequent writes, its limitations in real-time scenarios stem from issues such as small file explosion, metadata bloat, and concurrent writer conflicts. Specialized real-time analytics platforms, like Tinybird, may be more suitable for applications demanding high concurrency and sub-second query latency, as they offer the ability to handle high-frequency streaming writes and require different indexing strategies to support multiple query patterns and incremental pre-aggregations.