April 2025 Summaries
159 posts from Tinybird
Filter
Month:
Year:
Post Summaries
Back to Blog
Earlier today, a new Logs Explorer template was launched, showcasing how to build and optimize a scalable logs explorer that can handle trillions of logs with sub-second refresh times in a Next.js application. The process involves preemptive optimization for performance and cost, focusing on the data layer without altering web components. Key strategies include using materialized views for filtering logs and sidebar counters, optimizing time series charts, and enhancing free text search and interaction design to manage large data efficiently. The development workflow is iterative, allowing for local testing and integration with tools like Tinybird and Next.js, ultimately supporting complex data migrations and efficient deployment. The logs explorer is designed to cater to high-concurrency scenarios, ensuring users can effectively navigate and search through vast amounts of log data.
Apr 24, 2025
1,757 words in the original blog post.
The text examines and compares the architecture of a view counter service inspired by Reddit's 2017 system with a streamlined approach using Tinybird, highlighting the differences in complexity, cost, and operational overhead. The Reddit architecture involves multiple components like Kafka, Redis, and Cassandra, requiring specialized personnel and significant infrastructure costs due to its complexity and maintenance needs. In contrast, the Tinybird approach simplifies the process by consolidating data ingestion, processing, and storage into a single system, resulting in lower costs and operational requirements, as well as quicker development and troubleshooting. The comparison underscores that while both systems can manage the required scale, Tinybird offers a more efficient solution with faster feature development and reduced operational risk, ultimately transforming infrastructure management into a more straightforward, cost-effective process.
Apr 24, 2025
1,348 words in the original blog post.
Tinybird has shifted its pricing for self-serve paid plans from a model based on processed data to one based on infrastructure usage, specifically vCPU-hours, to address challenges users faced with the previous system. The change aims to offer more predictable costs, reduce the pressure to optimize queries early, and better align with developers' familiarity with cloud infrastructure pricing. This decision was informed by positive feedback from enterprise customers already using infrastructure-based pricing, which resulted in better cost predictability and faster development cycles. The new pricing model includes features like autoscaling, improved usage monitoring, and flexible plan sizes, with ongoing adjustments based on user feedback to enhance the developer experience. This transition reflects Tinybird’s commitment to simplifying real-time analytics for developers by minimizing concerns about cost spikes and optimization pressure, thereby fostering an environment conducive to innovation and growth with large-scale real-time data.
Apr 24, 2025
1,435 words in the original blog post.
Geospatial operations, like identifying polygons that contain a specific point, can be accelerated using the geohash technique, which encodes Earth's surface into grid-shaped cells as short alphanumeric strings, with longer strings indicating greater precision. This approach is implemented using ClickHouse® functions for geohash encoding and indexing, which involve creating a materialized view that maps geohashes to polygon IDs. The process involves calculating the geohash of a point, retrieving candidate polygons from a geohash index table, and verifying the point's inclusion within these polygons using the pointInPolygon() function. Although geohash serves as a lossy filter, it significantly reduces the number of polygons that need to be tested, thus enhancing query efficiency. The introduction of experimental_geo_types in ClickHouse®, including Point, Ring, Polygon, and MultiPolygon, promises even faster future queries. Spatial indexing is highlighted as a method to further speed up queries by reducing the number of polygons tested, similar to how sorting keys optimize data source queries.
Apr 24, 2025
520 words in the original blog post.
In a recent overhaul of the Tinybird application, the team unified the frontend under a single URL, app.tinybird.co, and enhanced the authentication system to streamline user experience and reduce internal technical debt. This complex refactoring involved breaking up the monolithic architecture into a decoupled system using Next.js for the frontend, deployed on Vercel, while relying on Vercel's edge network for caching and authentication, which improved performance and reduced latency issues. The restructured architecture allowed for more efficient and faster deployments, cutting down deployment times significantly and enabling more rapid development iterations. The changes addressed both external user challenges, such as confusing regional URLs and authentication edge cases, and internal technical challenges, like deeply coupled deployments and service maintenance inefficiencies. The transformation has improved the developer experience and positioned Tinybird to handle its rapidly growing user base, which processes massive amounts of data and handles billions of API requests annually.
Apr 24, 2025
3,296 words in the original blog post.
Tinybird, an enterprise-grade data platform, is designed to handle large-scale data processing with a focus on real-time analytics, security, performance, and scalability. The platform is built around the real-time database ClickHouse and utilizes several database replicas for efficient load management, which can increase or decrease depending on customer needs. Tinybird's scaling philosophy prioritizes optimizing SQL queries to minimize resource usage before expanding infrastructure, as hardware is costly while logical improvements are not. The platform supports high-concurrency, low-latency APIs by scaling query concurrency through adding database replicas or CPUs and by optimizing ingestion processes with shared storage and efficient data handling before database entry. This approach enables Tinybird to manage significant data loads, such as supporting a top-five global clothing retailer during Black Friday with 4.3 billion events and maintaining low latency. Through features like Materialized Views for pre-calculating aggregates, Tinybird aids clients in enhancing query efficiency without additional hardware. The company continues to refine its methods for scaling, supported by contributions to the open-source ClickHouse project and ongoing improvements in handling high-concurrency systems.
Apr 24, 2025
1,448 words in the original blog post.
Ariel's article addresses the challenges of optimizing PostgreSQL for analytics and highlights the signs that indicate it's time to consider transitioning to a dedicated OLAP database. After previously discussing various optimization techniques, the article focuses on recognizing when such efforts become counterproductive due to technical and organizational strains, such as increased query complexity, replication lag, and team burnout. Ariel emphasizes the importance of monitoring both technical metrics and team health indicators, like deployment frequency and satisfaction scores, to plan an analytics migration proactively rather than reactively. The article suggests that moving from PostgreSQL to a specialized analytics platform is not a failure but a strategic decision to enhance efficiency and focus on building features. Ariel concludes by introducing Tinybird, a real-time data platform designed for low-latency analytics, as a potential solution for those ready to transition.
Apr 24, 2025
1,032 words in the original blog post.
In the realm of data storage and manipulation, efficiently searching text within vast datasets poses significant challenges, especially in real-time scenarios. Tinybird, built on ClickHouse®, addresses these challenges by enabling scalable real-time data product development through SQL transformations and APIs. Conventional text search methods often result in inefficient full scans, but ClickHouse®'s Data Skipping Indexes, specifically Bloom filters, offer a more efficient alternative. Bloom filters are probabilistic data structures that help determine if an element exists within a set, thus enhancing search performance by reducing unnecessary operations. They work by splitting text into chunks, like n-grams, which can then be indexed, enabling more granular and efficient searches. Performance testing is crucial for optimizing Bloom filter configurations, as demonstrated in a case study where Bloom filters significantly reduced query times and scan sizes while increasing storage requirements. While Bloom filters improve search efficiency, they require a careful balance between performance gains and storage costs, highlighting the necessity for tailored configurations based on specific use cases. Tinybird, leveraging ClickHouse®, provides a platform for real-time analytics, although Bloom filters are not yet generally available within its ecosystem.
Apr 24, 2025
3,864 words in the original blog post.
In a blog post, the author details the process of creating a custom web analytics platform to replace Google Analytics using Retool and Tinybird, focusing on replicating the Content Drilldown feature. The Content Drilldown is a Google Analytics view that aggregates metrics for subdirectories within a domain, allowing users to explore session metrics for entire sections of a website rather than individual pages. The author describes how they used Tinybird to capture and transform web events data into a format that mimics the Content Drilldown's functionality, with a focus on privacy by avoiding the use of session cookies. They detail the steps for creating SQL queries in Tinybird to extract and aggregate pageviews by subdirectory, publish these queries as API endpoints, and utilize Retool to build an interactive user interface that mimics Google Analytics' drilldown functionality. The project culminates in a Retool application that allows users to drill down into specific subdirectories and view aggregated pageviews over time, providing a Google Analytics-like experience without compromising data privacy.
Apr 24, 2025
2,484 words in the original blog post.
Real-time data visualization is crucial for delivering quick, up-to-date insights, which can significantly enhance decision-making and problem-solving across various industries such as healthcare, finance, and e-commerce. The post discusses the factors contributing to slow dashboards, often stemming from underlying data architecture issues, and offers strategies for improving dashboard performance through optimized data models. Essential practices include optimizing SQL queries, leveraging materialized views, caching, and adopting real-time streaming architectures to ensure data freshness and low-latency response times. Choosing the right technology, such as real-time databases and data platforms, is pivotal for handling complex aggregations over large datasets efficiently. Additionally, the post emphasizes the role of business intelligence tools and frontend frameworks in creating interactive dashboards, highlighting the importance of a well-designed data pipeline to support real-time analytics. By implementing these strategies, businesses can build faster, more responsive dashboards that provide valuable insights from real-time data.
Apr 24, 2025
3,348 words in the original blog post.
Data engineers face significant challenges when managing Snowflake as a data warehouse for their organizations, particularly due to the high costs and resource demands associated with its operation. Snowflake's pricing model, which charges for compute usage per second, can lead to unpredictable expenses, especially when supporting high-concurrency, low-latency applications. Data engineers also need dedicated resources to manage Snowflake, which often becomes a bottleneck for feature development. Additionally, granting access to Snowflake data involves considerable red tape, as data engineers must control access to prevent cost overruns. Tinybird offers a solution by acting as an intermediary layer that allows data teams to efficiently sync Snowflake data, providing high-concurrency, low-latency APIs for application development at a fraction of the cost. This setup enables software engineers to build features without overburdening data teams or incurring excessive Snowflake expenses, thus streamlining data access and application development across organizations.
Apr 24, 2025
2,920 words in the original blog post.
Event sourcing is a data design pattern that determines the state of an application by replaying a sequence of events, rather than storing the latest state directly, offering advantages in auditability, scalability, and system debugging. This approach contrasts with traditional transactional databases and is particularly suited for large, distributed systems due to its ability to provide an eventually consistent view of current states. Kafka is an ideal tool for implementing event sourcing due to its real-time event streaming capabilities, strong consistency guarantees, and robust ecosystem. By using Kafka with platforms like Tinybird, developers can efficiently manage event logs, create state snapshots to optimize performance, and leverage real-time analytics to derive actionable insights. Snapshots are crucial for scaling event sourcing systems as they minimize computational overhead by reducing the need to replay all events when recalculating the current state. Implementing event sourcing can be complex, but tools like Tinybird simplify the process by offering SQL-based operations and seamless integration with Kafka, allowing for effective handling of event data and real-time analytics.
Apr 24, 2025
3,316 words in the original blog post.
Tinybird has introduced the jobs_log, a new Service Data Source designed to enhance the visibility and analysis of background jobs in its real-time analytics platform for user-facing applications. This feature allows users to query background job data as if it were a table, providing comprehensive insights into job runs, statistics, queue timings, and errors. It addresses previous limitations of the Jobs API by offering extended log retention of up to 12 months and an unlimited number of records. Available in all Tinybird Workspaces as tinybird.jobs_log, it enables centralized monitoring and can be used to create API endpoints and visualizations, such as time series or bar charts, to track job status and activity across different Workspaces. Organization Admins can leverage the organization.jobs_log to monitor jobs across all Organization Workspaces, enhancing production-level monitoring capabilities. Tinybird offers new users a free trial without requiring a credit card and provides extensive documentation and a Slack community for additional support.
Apr 24, 2025
443 words in the original blog post.
Companies facing challenges in productizing data stored in BigQuery due to its limitations in handling numerous real-time requests can integrate Tinybird to enhance data availability via APIs. This approach is exemplified by Acme.co, a B2B2C company that seeks to provide its enterprise clients with a real-time transaction dashboard, enabling cost reduction, fraud prevention, and revenue improvement through data segmentation. By leveraging Tinybird, Acme.co can expose BigQuery data in real-time, allowing for fast query times and efficient data consumption via APIs. The integration involves creating data sources in Tinybird, performing bulk uploads and streaming ingestion using Google DataFlow, and applying real-time transformations to optimize API performance. Acme.co publishes API endpoints to facilitate custom client dashboards and uses tools like Tableau for visualization and Vega for data quality testing, enabling a robust real-time analytics solution for their BigQuery datasets.
Apr 24, 2025
851 words in the original blog post.
Mockingbird is a free, open-source mock data generator that allows users to define data schemas in JSON and control data generation frequency to stream mock data to Tinybird and other HTTP-enabled destinations. Developed to address limitations in using public APIs for data streaming, Mockingbird offers flexibility and precision for creating custom data streams without relying on external data sets. It supports integration with Tinybird's Events API, allowing seamless data streaming with optimal schema generation, and also supports third-party destinations like Upstash Kafka REST Producer API. The tool can be used as a headless library, CLI, or UI, facilitating its incorporation into various projects and CI/CD pipelines. Mockingbird encourages community contributions for new data types and schema templates, enhancing its versatility and applicability in different scenarios.
Apr 24, 2025
1,299 words in the original blog post.
The principles of DataOps outlined in the text emphasize a focus on simplicity, speed, and developer experience in the development of Tinybird, a data platform. Key concepts include treating data projects like source code by using serialization and version control, ensuring rapid continuous integration and deployment, and prioritizing data quality assurance. The approach advocates for using tools that facilitate easy data sharing and analysis, maintaining observability with automated monitoring systems, and optimizing queries continuously. Additionally, the text highlights the importance of creating reusable data projects and publishing auto-documented APIs to enhance collaboration between data and development teams while supporting traditional tools like spreadsheets for comprehensive data management.
Apr 24, 2025
616 words in the original blog post.
Tinybird utilizes pytest as a crucial component of its CI/CD pipeline, enhancing testing and performance monitoring to ensure fast and reliable code deployment. With over 3,000 tests and frequent code updates, Tinybird emphasizes the importance of CI/CD observability to maintain agility and confidence in shipping code rapidly. By implementing a data-driven approach and integrating additional observability tools, Tinybird reduced CI execution times by over 60%, helping to identify and address bottlenecks and flaky tests. The company developed the pytest-tinybird plugin to facilitate real-time analytics and visualization of CI metrics, utilizing Tinybird's platform for capturing, analyzing, and automating CI data. This approach allows for improved resource allocation and test execution order, contributing to a more efficient CI pipeline. The plugin, open-sourced for public use, is designed to send pytest data to Tinybird, enabling users to harness SQL for insights and publish metrics as APIs, which can be visualized in tools like Grafana and Datadog to monitor CI health and performance.
Apr 24, 2025
2,336 words in the original blog post.
Building a real-time dashboard is crucial for providing users with instantaneous data access, which traditional dashboards often fail to achieve due to slow, batch-processing architectures. This guide walks through creating a real-time dashboard from scratch using Tinybird for real-time data ingestion and processing, Tremor for data visualization, and Next.js for a seamless and interactive user interface. Tinybird stands out for its ability to handle large data streams, execute complex queries swiftly, and scale according to demand, making it an ideal choice for real-time analytics. The tutorial covers setting up the tech stack, creating API endpoints with Tinybird pipes, and visualizing data with Tremor in a Next.js application. It emphasizes the importance of using optimized queries and scalable architectures to overcome common pitfalls of sluggish dashboards, offering a versatile framework that can be adapted to various real-time data use cases beyond the example of a document signature SaaS application.
Apr 24, 2025
4,383 words in the original blog post.
Tinybird has achieved compliance with the Health Insurance Portability and Accountability Act (HIPAA), a significant step that allows it to expand its services to a broader community of data engineers and developers, particularly in the healthcare sector. This compliance, alongside SOC 2 Type II compliance, underscores Tinybird's commitment to stringent data security and privacy, implementing measures such as data encryption, access control, and incident response to protect Protected Health Information (PHI). HIPAA compliance is crucial for handling PHI, as it ensures legal standards are met to safeguard sensitive health information, distinguishing it from Personally Identifiable Information (PII) that pertains to general privacy. This milestone reassures existing and potential healthcare customers that Tinybird's platform can securely process real-time analytics without compromising patient confidentiality, enabling healthcare organizations to focus on care delivery while Tinybird manages data processing complexities. The company highlights its ongoing commitment to security, encouraging customers to learn more through the Tinybird Trust Center or engage in Business Associate Agreements for further collaboration.
Apr 24, 2025
785 words in the original blog post.
Real-time analytics is revolutionizing data interaction by enabling the immediate processing and visualization of data as it is generated, providing a significant competitive edge for modern businesses. Unlike traditional batch analytics that process data in intervals, real-time analytics handles data in seconds or less, making it ideal for operational intelligence and user-facing applications. This approach relies on five core facets: data freshness, low query latency, high query complexity, query concurrency, and long data retention, and is distinct from streaming analytics, which lacks the capacity for complex, high-concurrency queries and long-term data storage. Real-time analytics is increasingly being applied across various industries, such as sports betting, travel, and web development, to enhance customer experiences through personalized interactions and immediate insights. Despite the challenges of cost, tool adoption, and managing scale, real-time analytics offers numerous benefits, including faster decision-making, improved customer experiences, and competitive differentiation. Platforms like Tinybird are advancing the field by providing integrated solutions that streamline the real-time analytics process, reducing the need for multiple tools and simplifying the development and deployment of real-time applications.
Apr 24, 2025
7,184 words in the original blog post.
Rockset's acquisition by OpenAI and the subsequent shutdown of its services in September 2024 have led many customers to seek alternatives, with Tinybird emerging as a compelling option. Tinybird is a cloud-based, SQL-first analytics platform designed for real-time, operational, and user-facing analytics, offering low-latency streaming data ingest and high query-per-second capabilities. It features a range of managed ingest connectors, supports materialized views for efficient data aggregation, and provides REST APIs for seamless integration. Tinybird is built on ClickHouse®, allowing for reduced vendor lock-in and easy data export. It also supports JWTs for secure data access in multi-tenant applications. The platform is recognized for its developer-friendly interface, enabling rapid deployment of features without the burden of infrastructure management, and offers tools like Tinybird Charts for quick data visualization.
Apr 24, 2025
1,225 words in the original blog post.
Creating a scalable and cost-effective solution for displaying real-time viewer statistics during major live events poses significant challenges, such as managing 3.3 million concurrent viewers and processing 350,000 events per second. The text explores two different architectural approaches to tackle this challenge: an AWS-based solution and an alternative using Tinybird. The AWS approach involves a complex, multi-layered architecture utilizing services like Kinesis Data Streams, Lambda functions, and CloudFront to ensure reliability and scalability, but it results in higher costs and complexity. In contrast, Tinybird offers a simpler, more streamlined solution that reduces operational overhead by relying heavily on SQL for feature development and data processing, leading to a 20% reduction in costs compared to AWS. Both approaches ensure reliable performance but differ in their operational simplicity and cost efficiency, with AWS being more suitable for those already invested in its ecosystem, while Tinybird offers faster development and simpler maintenance for new projects.
Apr 24, 2025
2,032 words in the original blog post.
Selecting the appropriate application database for real-time applications poses a challenge for developers due to the need for high performance, scalability, and versatility in handling complex queries. Real-time databases are designed to store, process, and analyze data as it occurs, enabling timely decision-making and user interactions. Popular databases such as MongoDB, PostgreSQL, and Tinybird each have unique strengths and limitations for real-time analytics. MongoDB offers scalability and a powerful aggregation framework but struggles with complex analytical queries and lacks full SQL support. PostgreSQL is familiar and easy to use but isn't optimized for large-scale real-time workloads, while Tinybird, built on ClickHouse®, excels in real-time data ingestion and processing, offering robust SQL support and ease of collaboration. Additional databases like ClickHouse®, Apache Pinot, and Apache Druid are also viable options depending on specific use cases, with ClickHouse® noted for its columnar storage and real-time query performance. Ultimately, the choice of a real-time database hinges on specific requirements such as OLAP workload support, scalability, event-driven data processing, and ease of maintenance.
Apr 24, 2025
3,442 words in the original blog post.
Snowflake is a leading cloud data warehouse known for its scalability, security, and cost-efficiency, but it faces challenges in supporting real-time data analytics due to latency and concurrency issues inherent in its architecture. Although Snowflake is suitable for batch processing and business intelligence applications, its handling of queries in a job pool and limited query concurrency per warehouse make it less ideal for real-time applications. Tinybird's Snowflake Connector addresses these limitations by enabling the creation of low-latency, high-concurrency APIs on Snowflake data, significantly reducing costs and development time. By allowing for near-real-time data ingestion and SQL-based API publication, Tinybird offers a cost-effective solution for building real-time data products, which can be up to ten times cheaper than using Snowflake alone. Tinybird's integration allows developers to bypass traditional bottlenecks by providing a ready-made publication layer, thereby accelerating the development process and reducing compute costs associated with real-time data use cases.
Apr 24, 2025
2,492 words in the original blog post.
Dub is an open-source link management and conversion tracking platform that utilizes Tinybird for its real-time analytics and webhooks feature, celebrated its first anniversary and handles significant data volumes efficiently. The platform's webhook event logs are stored in a Tinybird data source, organized by a sorting key that includes webhook_id and timestamp for efficient filtering and reading. Dub employs the zod-bird client to interact with Tinybird's APIs, using functions like recordWebhookEvent to post JSONL formatted payloads to Tinybird, and getWebhookEvents to fetch and validate data through Tinybird pipes. This data retrieval is facilitated by Tinybird's SQL query pipes, allowing dynamic filtering with parameters like webhookId to ensure fast, relevant data access. The Dub UI displays the event logs to users by retrieving data via an API and mapping it into components for a streamlined visual representation. Dub's integration with Tinybird supports various use cases, enabling users to connect their analytics with broader development tools and create customized workflows.
Apr 24, 2025
941 words in the original blog post.
Tinybird initially supported CSV as its primary data ingestion format due to its widespread use as an exchange format in databases and applications, despite CSV's lack of a standard specification leading to interoperability challenges. These challenges include issues with encoding, inconsistent column separators, header presence, and data type ambiguity. In response, Tinybird has embraced JSON as the de facto standard for web-based data communication, due to its semi-structured format, ease of parsing, and ability to handle nested data. However, JSON is not without its criticisms, particularly regarding its inefficiency compared to alternatives like Apache Avro and Protobuf. Tinybird's approach focuses on creating a format and transport agnostic ingestion framework that enhances developer experience by using an API-centric framework integrated with their dashboard and CLI. This framework facilitates better data type guessing, efficient JSON event streaming, and adaptable schema handling, allowing for seamless and scalable data analytics in real-time scenarios.
Apr 24, 2025
645 words in the original blog post.
The text discusses the challenges and solutions related to selecting databases for real-time analytics, emphasizing that traditional databases like Postgres, MySQL, and MongoDB are not designed for such tasks due to limitations in handling real-time data ingestion, complex joins, and concurrent queries. It highlights the strengths of specialized databases such as ClickHouse®, Apache Druid, and Apache Pinot, which are built for real-time analytics and excel in high data freshness, low query latency, and high concurrency. The text also mentions how general-purpose databases can be optimized for "almost real-time" analytics through strategic tuning, separating analytics roles, and using replication for read-heavy workloads. Additionally, it introduces modern alternatives like RisingWave and Materialize, which offer unique features such as SQL compatibility and strong consistency. Ultimately, the text underscores the importance of choosing a database that not only provides speed but also enhances development efficiency, with options like Tinybird offering managed solutions to simplify deployment and maintenance while leveraging the performance of open-source ClickHouse®.
Apr 24, 2025
6,319 words in the original blog post.
Anomaly detection is a crucial aspect of data analytics aimed at identifying outliers or unusual patterns within datasets, especially time-series data, to manage potential risks, system failures, or capitalize on business opportunities. A simple statistical method like the Z-score can effectively detect contextual anomalies by measuring how far a data point deviates from the mean, using the standard deviation as a reference. This approach involves analyzing a specific time context, calculating the Z-score for a series of data points, and flagging anomalies based on threshold values. Real-time anomaly detection systems are particularly valuable for their ability to respond instantly to unexpected events, requiring features such as handling large datasets, customizability for domain-specific needs, and seamless integration with alerting and visualization tools. A practical example of this is seen with Tinybird, whose system enables a large retailer to monitor sales transactions in real-time, detect anomalies using SQL-based logic, and trigger alerts through platforms like Microsoft Teams, demonstrating the broader potential for real-time data utilization across various applications.
Apr 24, 2025
884 words in the original blog post.
In collaboration with a major A/B testing platform, strategies were explored to enhance the efficiency of processing nearly a petabyte of data daily by employing downsampling techniques. Downsampling, the process of transforming raw data into a more compact form while retaining core characteristics, was evaluated to reduce computational resources and costs without sacrificing statistical accuracy. The approach involves selecting a representative subset of data, ensuring statistical rigor is maintained, especially critical for A/B testing. A practical example from a Tinybird customer use case demonstrated the application of a downsampling strategy, which included hashing user IDs to maintain a consistent sample size of 10%, helping to preserve user-level data sequences critical for analysis. This resulted in significant performance improvements, reducing data processing requirements while maintaining a manageable level of precision. The process involves balancing trade-offs between precision and performance and iteratively testing and optimizing sampling strategies to determine the most effective approach for specific needs.
Apr 24, 2025
2,818 words in the original blog post.
Anthropic AI's recent launch of the Model Context Protocol (MCP) has prompted developers to create context-aware applications using an open-source framework with a standard server-client architecture. A team quickly adopted MCP, building their first server just two days post-launch, and implemented basic observability infrastructure to track server usage, errors, and performance metrics. Currently, MCP Servers are locally installed, with plans for future remote production server SDKs; hence, developers have built local logging and analysis tools using Python/Typescript, Tinybird, and Grafana. These tools include a dashboard for visualizing server metrics and allow developers to send observability logs to a Tinybird Workspace, where data is managed and can be queried for building metrics. The project, designed for ease of deployment and integration with observability tools like Grafana, provides sample Tinybird Pipes for transforming logged events into metrics. Utilizing Tinybird's data infrastructure, developers can extend their MCP Server's functionality and build observability dashboards or user-facing applications.
Apr 24, 2025
771 words in the original blog post.
Tinybird's evolution from batch to real-time data ingestion necessitated a strategic approach to schema migrations, particularly when dealing with streaming data. Initially, Tinybird allowed users to ingest CSV files into ClickHouse® clusters, but as customer needs shifted towards real-time data processing, the company integrated streaming connectors like Kafka and Kinesis. This change introduced complexities, especially in managing schema migrations without disrupting data flow or impacting user-facing products. To address these challenges, Tinybird implemented a git integration, enabling version control, CI/CD, and automated testing to facilitate safer and more efficient schema migrations. This approach allows for testing changes in a production-like environment and ensures that any mistakes can be easily rolled back. The integration with git and the use of Materialized Views helped Tinybird execute complex data operations without compromising the stability of its services, ultimately reducing customer debugging time significantly. This approach underscores the importance of applying software development best practices to data engineering challenges, ensuring that schema migrations in high-throughput streaming systems are manageable and reliable.
Apr 24, 2025
2,914 words in the original blog post.
Embedding analytics dashboards into multi-tenant SaaS applications presents distinct challenges, primarily ensuring each customer sees only their data to maintain trust and privacy. Multi-tenant analytics, which combines multi-tenancy and user-facing analytics, involves collecting data from multiple customers and displaying tenant-specific metrics through shared infrastructure. User-facing analytics integrate real-time data visualizations directly into software applications, differing from traditional BI by offering on-demand data interactions. Key to secure multi-tenant analytics is row-level security, which restricts data access at the database row level, and JSON Web Tokens (JWTs) that provide a secure way to enforce these restrictions by ensuring only authorized data is accessible. Platforms like Tinybird simplify this process by enabling developers to stream data, define metrics with SQL, and publish analytics as low-latency REST APIs. Tinybird's approach, used by companies like Vercel and Canva, combines multi-tenancy and user-facing analytics, leveraging JWTs to ensure data security and isolation across tenants.
Apr 24, 2025
1,293 words in the original blog post.
Observability plays a crucial role in the software development lifecycle, and Tinybird has enhanced its integration capabilities by supporting the Prometheus endpoint format for all published API Endpoints. This addition allows developers to build endpoints for Prometheus, facilitating the monitoring of Tinybird infrastructure and other related resources, as Prometheus is widely adopted by leading observability platforms such as Grafana and Datadog. The article provides examples of observability projects, including Tinybird Organization Monitoring and MCP Server Analytics, showcasing how the Prometheus format can be used to monitor Tinybird Workspaces and locally-deployed servers through visualization tools like Grafana. Tinybird continues to prioritize observability by offering free querying of Service Data Sources and providing dashboards within its interface, encouraging users to explore further applications of the Prometheus endpoint format in their observability projects.
Apr 24, 2025
478 words in the original blog post.
Tinybird is leveraging the speed of the columnar database ClickHouse® to enhance query performance on large datasets by exploring the use of parallel replicas, a feature introduced in ClickHouse's April 23.3 release. Parallel replicas allow for query execution to be distributed across multiple servers, each holding a full copy of the data, offering a blend of sharding's performance benefits and replication's fault tolerance. This approach is particularly beneficial for complex queries over massive data sets, often involving JOINs, which are essential for Tinybird's real-time data platform. Despite the initial lack of JOIN support in parallel replicas, Tinybird contributed a solution to the ClickHouse® community, enabling INNER JOINs through broadcasting techniques. This advancement significantly reduces execution time for complex queries, as demonstrated by a test that cut a JOIN operation on 64 billion rows from 47 seconds to under 8 seconds using parallel replicas. While this method isn't always advantageous for simple queries due to overhead, it shows promise for scaling ClickHouse® clusters to handle trillions of rows at sub-second latency, marking a potential leap in real-time data processing capabilities.
Apr 24, 2025
2,350 words in the original blog post.
Tinybird has introduced support for the PostgreSQL Table Function, enabling developers to seamlessly import and sync data from a Postgres database into Tinybird Data Sources using SQL. This feature simplifies the process of migrating analytics data from Postgres, which, despite its popularity and versatility as a general-purpose database, struggles with high-scale, real-time analytics workloads. By leveraging Tinybird's Copy Pipes, developers can efficiently manage data synchronization, using strategies like replace or append, to maintain real-time analytics without needing to set up new infrastructure or learn new tools. The PostgreSQL Table Function utilizes Tinybird’s new Environment Variables API to securely manage credentials and variables, allowing developers to backfill historical data, perform incremental updates, and enrich event data with Postgres dimensions for scalable, user-facing analytics. Currently in Private Beta, this feature is part of Tinybird's ongoing efforts to provide easy solutions for developers to build real-time analytics platforms, with resources and community support available for new users.
Apr 24, 2025
1,333 words in the original blog post.
Effective data management in cloud environments is crucial to avoid excessive costs, exemplified by the potential $40,000 daily expenditure from inefficient data queries. Data engineers emphasize the importance of optimizing data pipelines to minimize processing and storage costs. A practical example involves an initial inefficient query that processes 6.67 GB of data, costing over $40,000 daily, which is improved through several optimization techniques. By applying filters before joins, using appropriate data types, and optimizing schemas, the data processed is reduced significantly, cutting costs to $400 a day. Further, utilizing materialized views reduces costs and processing time further to just $18 a day, demonstrating the substantial financial benefits of adhering to data management best practices. These optimizations not only deliver immediate savings but also establish a foundation for future efficiencies, underscoring the long-term value and return on investment of careful data pipeline design.
Apr 24, 2025
1,289 words in the original blog post.
dbt revolutionized data management by enabling data engineers and analysts to efficiently organize and process data in warehouses through batch analytics, whereas Tinybird offers a platform optimized for real-time analytics and low-latency API use cases, appealing to developers seeking a streamlined solution for building data-intensive applications. Tinybird distinguishes itself with its focus on speed and freshness, leveraging ClickHouse® for fast analytical queries and integrating data ingestion, transformation, API publishing, and observability into a single workflow. While dbt is well-suited for batch processing with its comprehensive stack involving separate tools for various stages of data handling, Tinybird simplifies the process by offering APIs as first-class citizens and reducing the complexity associated with multiple moving parts. Migrating from dbt to Tinybird requires adapting to a real-time processing mindset, emphasizing the design of materialized views and efficient data source schemas to ensure optimal performance. Despite the challenges, for those with real-time needs and an interest in consolidating their data operations, Tinybird presents a compelling alternative or complement to traditional dbt workflows.
Apr 24, 2025
1,770 words in the original blog post.
Steven Tey, a developer advocate for Vercel, discusses his project Dub, an open-source link management tool designed to help marketing teams create, share, and track short links. Initially built using Redis Sorted Sets for storing click data, Dub faced challenges with scalability and performance, prompting Steven to switch to Tinybird for analytics. Tinybird provided a more efficient and faster solution, enabling Steven to handle a growing number of clicks with improved speed and reduced bandwidth consumption. The integration with Tinybird allowed for dynamic API queries using SQL, enhancing Dub's analytical capabilities. With Tinybird as the backbone of Dub's analytics, Steven plans to continue scaling the tool while also focusing on marketing efforts. He endorses Tinybird for its powerful features, affordability, and excellent developer experience, recommending it to those developing analytics-driven applications.
Apr 24, 2025
1,090 words in the original blog post.
Tinybird utilizes ClickHouse® to allow customers to query massive datasets in real time, but achieving optimal performance requires careful query design and data processing strategies. The company provides a platform for storing, processing, and querying data, with performance monitored to meet enterprise SLAs. Users can define dynamic API endpoints, which may require optimization to avoid computational expense and slow response times. Tinybird offers tools like the tinybird.pipe_stats_rt service to monitor request durations, response times, and error codes, helping users identify which query parameters impact performance. By analyzing correlations between parameters and response times, Tinybird can optimize data schemas and queries, such as partitioning tables and adjusting sorting keys, to improve efficiency. Visualization tools like Google Sheets can be used to interpret correlation matrices, and strategic changes have resulted in a 40% decrease in response times. Tinybird emphasizes the importance of real-time analytics and offers resources for building scalable data products.
Apr 24, 2025
917 words in the original blog post.
Data engineers are increasingly turning to Change Data Capture (CDC) as a solution to manage the growing demand for real-time data processing and decision-making. CDC is a method that efficiently tracks and routes changes in a database to downstream systems in real-time, thereby minimizing latency and reducing the load on the source database. It is particularly useful in real-time, event-driven architectures, supporting data integration and analytics by streaming updates as they happen. CDC tools, such as Debezium, leverage database transaction logs to capture data changes, which are often sent to message queues like Apache Kafka for further processing. While CDC offers significant advantages, including efficiency and immediate data updates, it also presents challenges such as complexity and resource intensiveness. Despite these challenges, CDC remains a vital component for enabling real-time data pipelines and analytics, often used alongside ETL processes for comprehensive data management strategies. Real-time data platforms, like Tinybird, enhance CDC's capabilities by providing scalable solutions for capturing and processing change data in real-time, enabling organizations to maintain up-to-date insights and improve decision-making processes.
Apr 24, 2025
3,149 words in the original blog post.
Columnar databases, also known as column-oriented databases, offer a specialized approach to data storage that optimizes for analytical queries by storing data in columns rather than rows, which is common in traditional databases. This storage format enhances performance for analytics tasks that involve scanning or aggregating large datasets with minimal columns, reducing I/O costs and time. Columnar databases, such as Amazon Redshift, Google BigQuery, and Snowflake, are suited for real-time analytics, complex queries, and time-series data, while row-oriented databases, like MySQL and PostgreSQL, are more efficient for transactional workloads that involve frequent single-row updates or deletes. The choice between columnar and row-oriented databases depends on the specific use case, data volume, and whether the primary need is for analytics or transactions. Columnar databases, while having a steeper learning curve, can provide significant performance benefits for big data analytics, although they are not typically optimized for tasks requiring strict ACID compliance. It's common to use both database types in tandem within an organization to cover a broad range of data requirements.
Apr 24, 2025
2,439 words in the original blog post.
Tinybird Forward offers a streamlined approach to data deployments, aiming to bring the same level of simplicity and reliability typically enjoyed in software development to data workflows. Unlike traditional fragmented processes that often lead to errors and inefficiencies, Tinybird Forward's tb deploy command simplifies the deployment lifecycle, allowing users to deploy changes with a single command while integrating seamlessly with CI/CD pipelines. The platform ensures zero downtime and data consistency during schema migrations by employing techniques such as parallel processing and using auxiliary tables to maintain data flow even during complex transitions. Features like pre-deployment validation, automatic backfills, and the innovative FORWARD_QUERY instruction facilitate smooth schema evolution without the cognitive overhead or potential pitfalls associated with manual processes. Additionally, Tinybird Forward provides robust safeguards, allowing users to discard changes during deployment if necessary, thereby ensuring a fail-safe environment. As the platform evolves, it continues to enhance its observability and validation processes, aiming to further optimize the deployment experience for users dealing with large-scale data transformations.
Apr 24, 2025
1,393 words in the original blog post.
The blog post outlines a step-by-step guide to creating a personalized, open-source alternative to Google Analytics using Tinybird's Starter Kit, emphasizing its privacy-first approach. It begins by setting up a Tinybird workspace, which acts as a serverless data backend, using a JavaScript tracker to collect website traffic data and generate a real-time analytics dashboard. Users can install a tracking snippet on their websites to capture pageviews and other events, which are then processed through Tinybird Pipes and API Endpoints to populate the dashboard. The guide also explains customizing the dashboard by defining custom events, creating new metrics, and developing new visualization widgets using technologies like Next.js and React. Users are encouraged to ensure compliance with data privacy laws such as GDPR and can choose their frontend tools to display analytics data.
Apr 24, 2025
2,127 words in the original blog post.
Tinybird Local is a free Docker image that replicates the Tinybird cloud service for local development, providing developers with a local environment to build and test applications before deployment. This local version allows users to create pipes and data sources, ingest data, manage tokens and workspaces, but lacks connectors, scheduled copy jobs, a browser UI, and sinks. The image is updated continuously and is not intended for production use due to its limitations with larger data sizes. Despite initial hesitations, the release of Tinybird Local addresses the need for faster builds and reduced flakiness in CI pipelines by eliminating external dependencies and allowing for easy creation of test environments. Although it runs multiple services in a single container, which deviates from standard Docker practices, this approach is designed for simplicity and ease of use, aligning with the developer preference for local builds. Feedback from users will guide future enhancements, with the intention to enhance developer velocity and align with remote work trends.
Apr 24, 2025
541 words in the original blog post.
ClickHouse® has introduced window functions as an experimental feature, enabling users to perform calculations across related table rows without grouping them into a single output row, thereby simplifying query writing and expanding functional possibilities. These functions, already familiar to users of databases like Postgres and MySQL, allow for cumulative sums, averages, moving counts, and more advanced operations, utilizing clauses such as WINDOW, PARTITION BY, and RANGE to define how data is processed. Additionally, ClickHouse® supports ranking functions like RANK, ROW_NUMBER, and DENSE_RANK, with recent updates addressing issues related to sorting with partition expressions. The roadmap for ClickHouse® includes promising features like enhanced support for nested and semi-structured data, separation of storage and compute, a PostgreSQL table engine, projections for efficient aggregations, and improved backup capabilities. Tinybird offers dynamic endpoints for real-time analytics on ClickHouse® data, providing additional utility for users looking to leverage these new features.
Apr 24, 2025
886 words in the original blog post.
The text discusses the challenges and considerations involved in migrating analytical workloads from PostgreSQL to real-time OLAP databases. It highlights the limitations of traditional data warehouses like Snowflake and BigQuery for handling high-concurrency and low-latency requirements, suggesting newer cloud data warehouses like Firebolt or real-time OLAP solutions such as StarRocks, DuckDB, and Doris as potential alternatives. The document outlines key evaluation criteria, including performance and scalability, SQL compatibility, developer experience, and integration capabilities. It stresses the importance of assessing non-negotiable requirements, data freshness needs, development velocity, and migration efforts when choosing a new system. The text also mentions the benefits of using Tinybird for real-time data analytics, offering it as a solution optimized for low-latency, high-concurrency workloads, and highlights the importance of building a cohesive OLTP and OLAP stack for efficient data processing.
Apr 24, 2025
3,228 words in the original blog post.
Flexibility in data exploration is crucial, and using Tinybird, users can easily rearrange transformation nodes within a data pipe by dragging and dropping them to optimize query readability and logic flow. Pipes, which support multiple transformation nodes, help maintain concise and clean queries that can source data from a Data Source or previous node results, similar to sub-queries. This rearrangement capability allows users to refactor code or incorporate additional elements as needed, enhancing the real-time analysis and actionable insights derived from data at any scale. Users can access these features through their Tinybird account or engage with the community via a Slack channel.
Apr 24, 2025
172 words in the original blog post.
Amazon Simple Storage Service (S3) is a popular object storage service known for its scalability, durability, and cost-effectiveness, making it ideal for storing large amounts of various data types. Tinybird has introduced the S3 Connector to facilitate the ingestion of data from Amazon S3 into its real-time analytics platform, allowing users to build, publish, and share near-real-time analytics with ease. This connector bridges the gap between S3 and real-time analytics by enabling data engineers and developers to sync S3 files into Tinybird, where they can be queried using SQL and transformed into low-latency analytics APIs. Unlike AWS Redshift, which is more suited for long-running analytics, Tinybird's platform is optimized for real-time use cases, providing high-concurrency, low-latency data products without the need for additional infrastructure or code management. The S3 Connector supports various file formats and can be set up via Tinybird's intuitive UI or CLI, offering automation for data ingestion and robust monitoring tools to ensure efficient data processing.
Apr 24, 2025
1,439 words in the original blog post.
As the number of views on posts increases to trillions, simple systems for counting unique viewers become insufficient, prompting the need for scalable solutions. The challenges arise from the sheer volume of data and the limitations of functions like uniqExact, which, while providing perfect accuracy, can lead to significant performance degradation due to memory and CPU constraints. To address these issues, optimizations such as using uniqCombined64, which adapts based on data scale, and pre-aggregation with materialized views are proposed. These methods help manage memory usage and improve query performance, with uniqCombined64 offering a balance between accuracy and scalability, and pre-aggregation reducing query times by maintaining exact counts in real-time. A combined approach utilizing both uniqCombined64 and pre-aggregation is recommended for handling massive scale while ensuring efficient resource use, enabling teams to maintain performance without sacrificing accuracy significantly.
Apr 24, 2025
1,070 words in the original blog post.
Building a data-intensive Next.js application is simplified by using tools like Tinybird and Cursor, which streamline the creation of scalable apps capable of processing large datasets. These platforms allow developers to quickly set up infrastructures for managing backend tasks such as data persistence, authentication, and caching. Tinybird provides an analytics backend for handling massive data, transforming logs into secure, scalable API endpoints while reducing the complexity associated with traditional databases. Integrations with AI-assisted development environments, like Cursor, facilitate rapid prototyping and testing, enabling the creation of functional applications, such as a leaderboard app, that efficiently handle millions of data rows. The process involves setting up a local Tinybird project, using Docker for development, and deploying to the cloud with CI/CD workflows, using environments like Vercel for deployment and management. This approach results in high-performance applications that maintain efficiency even with large volumes of data, showcasing the ease of building and deploying complex web applications with modern development tools.
Apr 24, 2025
1,738 words in the original blog post.
Tinybird has introduced the ability to create Materialized Views (MVs) directly within its UI, which previously required using the CLI and specific ClickHouse expertise. This enhancement aims to streamline the development process for users who prefer a graphical interface by automatically suggesting optimal parameters for MVs, thus enabling faster and more efficient creation of performant endpoints. Materialized Views are key in reducing query response times and data scan sizes, making data delivery faster and more cost-effective. The process involves using Pipes to define pre-aggregated or filtered results that can be easily converted into a Materialized View, with the UI offering features like automatic parameter recommendations and real-time incremental materialization. This update is designed to simplify the creation of MVs, offering an intuitive process that ensures reliable and real-time data sources for enhanced application performance.
Apr 24, 2025
457 words in the original blog post.
Vercel is a web platform designed for rapid software deployment, and it utilizes webhooks to export data that can be integrated with Tinybird's Events API, enabling efficient data analysis. By using Vercel webhooks, developers can seamlessly transfer deployment data to Tinybird, where SQL transformations are applied to create API endpoints, which are then visualized through a Next.js app using shadcn charts. This setup allows for real-time insights into deployment speed and efficiency, offering metrics such as the number of deployments, their durations, and individual developer contributions. Additionally, the ability to analyze various Vercel event types enables developers to track and mitigate deployment errors, thereby enhancing overall performance. The process is streamlined and flexible, allowing for the rapid creation of new visualizations and insights, with community support available for troubleshooting and expansion.
Apr 24, 2025
541 words in the original blog post.
In an endeavor to address an ongoing issue with their ClickHouse® cluster, Tinybird engineers managed to resolve a longstanding problem that had limited query concurrency and underutilized CPU resources. Initially, despite high demand, the system's CPU usage remained below 20%, prompting a series of temporary fixes over the course of a year. The breakthrough came when they identified a spike in ContextLockWait events, which led to a significant code refactor involving the replacement of a global mutex with read-write mutexes to reduce contention in the ClickHouse® database. This refactor, coupled with a new metric to monitor Context lock impact, resulted in a dramatic performance improvement, increasing query throughput and CPU utilization to 100% in testing. Although the engineers do not expect a fivefold improvement in production due to potential bottlenecks, even a 1.5x increase in performance will significantly benefit Tinybird's infrastructure and its clients. With these changes incorporated into the ClickHouse® 23.10 release, the company anticipates enhanced performance for their most demanding clients.
Apr 24, 2025
1,704 words in the original blog post.
Building fast dashboards is crucial to enhancing user experience and involves addressing common performance issues such as poorly constructed SQL queries, using inappropriate databases, and failing to pre-filter or cache data. By optimizing SQL queries, moving raw data to real-time databases, creating pre-filtered and pre-aggregated views, caching query responses, hosting data near users, adopting event-driven architectures, and utilizing real-time processing engines, dashboards can be significantly sped up, offering lower latency and fresher data. Real-time platforms like Tinybird facilitate these processes by allowing users to capture streaming data, process it in real-time using SQL-based pipes, and store pre-aggregated data in materialized views, which can be accessed through low-latency APIs. These strategies collectively contribute to faster, more responsive dashboards that can efficiently handle concurrent queries and provide timely data updates.
Apr 24, 2025
1,395 words in the original blog post.
Postgres, traditionally viewed as more suited for OLTP workloads, can effectively handle OLAP workloads with the right techniques and optimizations. This involves understanding the fundamental differences between OLTP and OLAP queries, such as the complexity and resource demands of the latter, and implementing strategies like separating workloads, using dedicated analytics replicas, optimizing schema design with materialized views and partitioning, and leveraging Postgres extensions for columnar storage. Additionally, employing advanced query optimization techniques, like window functions and parallel execution, and utilizing specific indexing strategies, such as BRIN indexes, can significantly enhance performance. However, these approaches require careful planning, monitoring, and adjustments, and there are inherent limits to what Postgres can handle, necessitating consideration of specialized analytics solutions when those limits are reached.
Apr 24, 2025
2,980 words in the original blog post.
JR the Builder, an indie-hacker based in the Bay Area, co-founded Beam Analytics with Leng Lee to offer an affordable, GDPR-compliant alternative to Google Analytics. Beam Analytics aims to address gaps in the market by providing advanced features like cohort retention and conversion funnels at a lower cost, compared to other options like Plausible and Fathom. To achieve this, JR leveraged Tinybird and ClickHouse® to handle real-time analytics and privacy-first web analytics, enabling rapid and cost-effective development without worrying about infrastructure management. Tinybird's serverless architecture and predictable pricing model facilitated Beam's ability to quickly scale and focus on product development. With Beam currently deployed on 200 websites, the team is focused on expanding to 1,000 and enhancing its product analytics capabilities. JR emphasizes that Tinybird simplifies data infrastructure management, allowing companies to concentrate on their core product features, and encourages others to explore Tinybird's potential through its Starter Kits.
Apr 24, 2025
1,141 words in the original blog post.
Amazon DynamoDB is a widely used NoSQL database on AWS, known for its speed and efficiency in handling real-time transactional workloads. However, it is not optimized for online analytical processing due to its limitations with wide scans and aggregations. To address this, the text explores three methods to enhance DynamoDB with other AWS technologies for real-time analytics. These include leveraging AWS Lambda to calculate aggregates and store them in either another DynamoDB table or ElastiCache for Redis, and using Tinybird's analytics platform to create dynamic REST APIs by streaming data from DynamoDB. Each approach has its pros and cons, with the Tinybird method offering the most flexibility and performance by enabling SQL-based queries on replicated DynamoDB data, thus providing a seamless way to build user-facing analytical features.
Apr 24, 2025
2,670 words in the original blog post.
Tinybird has introduced a rate limiting feature for its API Endpoints, allowing users to control the frequency of API requests on a per-endpoint or per-user basis through JSON Web Tokens (JWTs). This enhancement aims to simplify the management of API request limits, offering an integrated solution that eliminates the need for backend proxies, which previously increased latency and complexity. By leveraging JWTs, developers can set specific rate limits for different users or globally across multiple clients, enabling flexible API support tiers based on pricing models. Tinybird's new rate limiting feature enhances security and efficiency by replacing static tokens with dynamic JWTs, streamlining the process of building scalable, real-time analytics applications without the added overhead of external services.
Apr 24, 2025
661 words in the original blog post.
Tinybird has introduced Tinybird Charts, a set of built-in visualization components that allow users to quickly transform real-time data into various chart formats, enhancing the speed and efficiency of data handling within the platform. Available to all Tinybird customers across different pricing plans, these charts are designed to integrate seamlessly into applications, demonstrated through an example of exporting a chart as a React component and embedding it in a Next.js app within seconds. The platform supports eight visualizations initially, with plans for rapid expansion based on user feedback collected via Community Slack. Tinybird Charts utilize Apache ECharts, an open-source JavaScript visualization library, offering users the ability to customize charts to fit their design and branding needs. The company encourages users to consult the documentation for detailed guidance and invites community input to help prioritize future developments.
Apr 24, 2025
379 words in the original blog post.
Tinybird has announced the launch of its BigQuery Connector, which allows developers to efficiently integrate Google BigQuery data into Tinybird for real-time processing and application development. This connector is designed to facilitate the creation of low-latency, high-concurrency APIs, enabling developers to sync BigQuery tables and views to Tinybird with ease, thanks to its fully managed and serverless architecture. The integration leverages the Tinybird Connector Development Kit (CDK), offering users the ability to develop custom connectors for other data products. With a focus on observability, security, and ease of use, the connector streamlines the process of ingesting BigQuery data, making it ideal for building high-performance applications that require up-to-the-second data. The BigQuery Connector is available through the Google Cloud Marketplace as part of the Google Cloud Partner Advantage Program, allowing users to start using the service immediately.
Apr 24, 2025
1,638 words in the original blog post.
Tinybird has launched JWTs for API Endpoints into public beta, allowing developers to call Tinybird APIs directly from browsers without needing additional middleware or backend components. This feature facilitates front-end applications in accessing real-time analytics data directly, enhancing performance by reducing latency and eliminating the need for proxies, which previously required extra code and infrastructure. JWTs are integrated into existing app authentication flows, allowing secure and direct API calls from the client-side by using a shared secret for JWT signing. Initially, JWTs support read-only scopes and fixed parameters, enabling row-level security and flexible per-token parameters in multi-tenant applications. Future enhancements might include expanding write scopes, implementing caching, rate limiting, and supporting custom domains, based on user feedback and needs. Users can begin using JWTs in Tinybird Workspaces for free, with comprehensive documentation and community support available to assist in the transition.
Apr 24, 2025
1,083 words in the original blog post.
The text delves into the complexities of optimizing ClickHouse® for efficient data handling, focusing on both read and write operations. It highlights the challenges of managing different types of queries, such as real-time and long-running queries, and the necessity of balancing hardware resources to avoid latency issues. The importance of query design is emphasized, with specific strategies for optimizing queries, including the use of sorting keys and managing memory settings. It also discusses the intricacies of backfilling materialized views and the potential pitfalls of using certain ClickHouse® features. The author advises on monitoring key metrics and handling cluster operations to ensure stability and performance, while acknowledging the inherent difficulties in operating a large-scale database system. Additionally, the text provides insights into managing materialized views and avoiding server crashes, ultimately offering practical advice for those navigating the challenges of database management with ClickHouse®.
Apr 24, 2025
2,040 words in the original blog post.
Tinybird has introduced a feature allowing users to cancel queries, addressing a limitation in their system that previously prevented this capability. The challenge stemmed from technical complexities associated with query cancellation, particularly when queries originate from clients like the Tinybird web app or CLI. Traditionally, database query IDs are assigned at the API layer, meaning the UI is unaware of these IDs until the query is complete, rendering cancellation efforts ineffective. To resolve this, Tinybird now delegates query ID assignment to the UI, which then communicates this ID to the API server, enabling the client to initiate a query cancellation request. This approach overcomes earlier limitations by allowing safe and targeted query cancellation, carefully avoiding disruption of critical background processes through a unique query ID strategy, endpoint restriction, and targeted server cancellation. These measures ensure that only specific queries can be safely terminated, maintaining system integrity without overloading replication services like ZooKeeper. This feature highlights the complexity behind seemingly simple functionalities and offers a framework for other products considering similar implementations.
Apr 24, 2025
908 words in the original blog post.
The text discusses a method for calculating the sum of amounts from previous and following days for a specific date in a table, using the example of the date 2020-01-05. It suggests utilizing the function arrayCumSum along with other array techniques to efficiently calculate these sums without needing to compute the values repeatedly for each day. The author implies there might be a simpler method to achieve the same result but is unsure of what that might be. The text points to a snapshot for a detailed, step-by-step explanation of the approach.
Apr 24, 2025
116 words in the original blog post.
Tinybird's platform offers a seamless way for developers to create REST API endpoints from SQL queries, emphasizing simplicity and real-time data interaction. The core feature is its support for parameterization using a templating language within Tinybird Pipes, which allows users to define query parameters directly in SQL, extending the capabilities of the APIs with complex expressions that go beyond basic SQL constraints. Tinybird facilitates powerful data manipulation techniques, such as dynamic aggregations and variable column selections, and supports advanced templating functionalities, providing a flexible environment for building customized APIs. This approach not only streamlines the development process but also enhances the adaptability and scalability of data-driven applications.
Apr 24, 2025
658 words in the original blog post.
A newly announced open-source Logs Explorer template offers a free, deployable solution using a Next.js app and Tinybird backend, designed as a simple alternative to Datadog for building user-facing logs analytics features in apps or services. This template, which can be deployed to Tinybird and Vercel in under five minutes, is suitable for those interested in developing a performant logs explorer by providing a foundational setup that can be modified and scaled to handle trillions of rows and thousands of queries per second. The development process involves using Tinybird's Events API for log streaming, version control, and CI/CD, alongside tools like Cursor.ai, Claude Sonnet 3.5, and GitHub Actions. The template supports end-to-end workflows with APIs for filtering, aggregating metrics, and performing text searches, while the Next.js integration uses zod-bird for API validation and @tinybirdco/charts for visualizations. The project emphasizes a streamlined approach to setup, testing, and iterative development, allowing users to optimize as needed for scalability, and encourages customization to fit specific use cases.
Apr 24, 2025
1,944 words in the original blog post.
Tinybird, a real-time data platform, enables users to ingest, query, and publish data using SQL, offering a flexible solution for companies seeking to automate data-driven insights without extensive coding. The blog post shares a practical application of Tinybird paired with Make, a no-code automation tool, to solve a communication gap between sales and data engineering teams at Tinybird. The author, a Sales Engineer, leveraged these tools to automate the process of notifying sales teams when customers reached usage milestones, improving customer relationship management and operational efficiency. By integrating data from Salesforce into Tinybird, querying it with SQL, and using Make to automate Slack notifications, the solution reduced the need for manual data checks and allowed the sales team to be more proactive. This approach underscores the utility of low-code/no-code platforms in bridging technical gaps and streamlining workflows in fast-paced startup environments.
Apr 24, 2025
2,178 words in the original blog post.
Apache Kafka is a widely used architecture for building scalable and fault-tolerant streaming applications, making it an essential component for enterprises handling event data. Tinybird, a platform for developers to create low-latency APIs, faced a challenge with its Kafka connector due to the exponential increase in Kafka costs with only linear customer growth. Initially, Tinybird's Kafka consumers were optimized for throughput and availability, but this approach resulted in scalability issues due to too many Kafka connections. To address this, Tinybird implemented rendezvous hashing, a method allowing them to reduce the number of Kafka connections while maintaining high throughput and availability. This solution distributed agents across hundreds of topics, minimized rebalancing during changes, and significantly reduced infrastructure costs. Tinybird now supports ingestion from numerous concurrent topics, providing a cost-efficient and scalable solution for developers building analytics on Kafka data.
Apr 24, 2025
2,275 words in the original blog post.
The text explores the process of integrating OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing) systems, specifically using databases like MySQL and ClickHouse for a hybrid architecture. It provides insights on various data ingestion methods, such as streaming integration through Change Data Capture (CDC) and Event-Driven Architecture, as well as batch integration techniques. The text discusses the pros and cons of each approach depending on factors like data freshness and operational complexity. Additionally, it covers data egress strategies and the importance of comprehensive monitoring across distributed systems to manage potential points of failure. Emphasizing minimalism to reduce the complexity tax, the text suggests starting with simple integration patterns and scaling complexity only when needed. It also highlights emerging patterns like Zero-Copy Analytics with Iceberg for potential future simplification of OLTP and OLAP integration.
Apr 24, 2025
3,037 words in the original blog post.
In the rapidly growing landscape of data generation, expected to reach over 450 exabytes daily by 2025, Materialized Views offer an efficient solution for managing and querying large volumes of real-time data. Unlike regular Views, which act as virtual tables, Materialized Views store the results of a query either in memory or on disk, improving query performance and reducing latency. They are especially beneficial in scenarios requiring complex analytical queries, as they precompute and store results, thus reducing computational load at query time. Platforms like Tinybird leverage Materialized Views to unify batch and streaming data into indexed stores, facilitating quick and consistent access to real-time analytics via low-latency APIs. While Materialized Views enhance speed, simplicity, and consistency in data handling, they do require additional storage and may need manual refreshing depending on the database system used. However, for real-time analytics, databases like Tinybird optimize these processes to ensure that data is always fresh and available, making them a valuable tool for developers aiming to enhance application performance and maintain consistent business logic across various data applications.
Apr 24, 2025
2,291 words in the original blog post.
The discussed data ingestion API is designed to be user-friendly and compatible with web technologies, accepting NDJSON and JSON formats while potentially considering Parquet support. It emphasizes a schema-based approach to enhance efficiency by transforming attributes into columns in a columnar database, significantly improving storage and processing performance compared to schemaless methods. The API provides an acknowledgment when data is received and stored, ensuring reliability through idempotency, allowing retries within a five-hour window without duplicating data entries. It incorporates buffering to optimize performance and manage database overloads, making it suitable for handling high query per second (QPS) loads and large payloads, while still enabling near real-time data availability, typically within four seconds or less. The API's design aims to address common ingestion challenges while inviting feedback on its effectiveness.
Apr 24, 2025
484 words in the original blog post.
In the digital marketing landscape, real-time data has become essential for businesses to maintain a competitive edge, as relying solely on overnight data processing and dashboards is no longer sufficient. Tinybird, a real-time data analytics platform, allows data teams to ingest, process, and expose data via APIs, enabling automated and immediate decision-making across various business functions such as software engineering, customer support, and marketing. By leveraging Tinybird APIs, the company's marketing team has integrated real-time data analytics into their MarTech stack, resulting in enhanced performance metrics such as a significant increase in product sign-ups and user conversions. This integration allows for personalized and automated marketing campaigns, A/B testing, and the identification of user behavior patterns to optimize customer engagement and retention. The company also uses Tinybird to detect anomalies in usage patterns to preemptively address customer issues and avoid unexpected costs. As real-time data becomes a standard requirement in marketing, Tinybird provides the tools necessary for companies to innovate and refine their strategies, ultimately driving business goals with speed and accuracy.
Apr 24, 2025
1,987 words in the original blog post.
An individual enhanced the Auth0 activity page by integrating Auth0 logs with Tinybird and incorporating language model (LLM) capabilities. By utilizing Auth0 webhooks to send log data to Tinybird's JSON-compatible endpoint, they created a real-time data infrastructure that facilitates insights into product usage. To address the complexity of Auth0's 139 different log schemas, the author educated an LLM about these schemas, enabling it to assist with SQL queries without needing explicit schema definitions. The project involved building a dynamic dashboard using Tinybird for data infrastructure, Next.js for the frontend, Vercel for deployment, and additional UI components. They implemented various features like filtering by tenant, application, and connection, as well as time-based aggregations, to enhance data analysis capabilities beyond the standard Auth0 offerings. Despite not being a frontend developer, the author leveraged AI tools to rapidly develop and iterate the dashboard, resulting in a functional prototype in less than a day. The project's code is publicly available as part of the Tinynest initiative, which includes dashboards for various developer tools, inviting others to explore and expand on the work.
Apr 24, 2025
1,214 words in the original blog post.
Tinybird has launched an updated user interface to enhance developer productivity by streamlining access to and exploration of Data Sources. The new interface features a full-width view for Data Sources, removing the previous pop-up modal to provide a more spacious and functional workspace with improved visibility of ingestion charts. A split-screen functionality has been introduced, allowing users to view Data Source schemas while building Pipes without needing to switch contexts. The Data Source menu has been reorganized with simplified tabs for easier navigation, and a dedicated schema tab has been added for direct interaction with Data Source files. Additionally, a new copy button simplifies copying values from tables, enhancing the ease of testing filters or adding query parameters. These improvements aim to make the platform more intuitive and efficient for all Tinybird users.
Apr 24, 2025
685 words in the original blog post.
Tinybird has introduced a new pricing model aimed at providing greater predictability and flexibility, with pricing now based on vCPU hours rather than processed data. The Free plan remains available without a credit card requirement, but all Free Workspaces must now belong to an Organization. Developer plans, previously known as Pro plans, have various tiers based on vCPU size and include features like autoscaling and burst mode, with pricing ranging from $25 to $299 per month. All existing Pro Workspaces need to transition to the new Developer plan by April 29, 2025, and Tinybird provides tools to help users estimate new plan costs based on previous usage. For Enterprise users requiring dedicated infrastructure, pricing is now credit-based, and they will receive guidance on migration from their support representatives. The introduction of Organizations allows users to manage Workspaces under a single billing plan, and Tinybird has enhanced its web UI to offer better pricing observability to track plan usage and limits.
Apr 24, 2025
1,470 words in the original blog post.
Real-time streaming data has become essential for modern data analytics, allowing organizations to process and analyze vast amounts of data as it is generated, providing immediate insights and actions. This shift from traditional batch processing to real-time data architectures is driven by the need for data freshness, low-latency, and high-concurrency access, enabling businesses to respond swiftly to changing conditions and capitalize on opportunities. Key components of real-time data architectures include data sources, event streaming platforms like Apache Kafka, stream processing engines such as Apache Flink, real-time databases like ClickHouse®, and APIs that expose real-time analytics to downstream consumers, including user-facing applications and machine learning models. The scalability challenges of handling data volume, velocity, and variety in real-time systems can be addressed through strategies like scaling out, scaling up, partitioning, and utilizing serverless compute models. Real-time data platforms, such as Tinybird, streamline the integration and deployment of these components, enabling organizations to build effective real-time data pipelines and applications. As real-time architectures evolve, they are increasingly used for applications like user-facing analytics, real-time machine learning, and operational intelligence, driven by the need for immediate and accurate data-driven decision-making.
Apr 24, 2025
3,553 words in the original blog post.
Processing large volumes of data in real-time can be optimized through a set of best practices that enhance query performance and efficiency. These practices, drawn from extensive experience with real-time systems, are encapsulated in the "5 Rules of Fast Queries" by Tinybird. The first rule emphasizes minimizing data writes to avoid unnecessary memory usage and costs. The second rule suggests prioritizing data filters to expedite query execution by reducing the dataset size early in the process. The third rule highlights the importance of sequential reads, which can be significantly accelerated by proper indexing aligned with query patterns. The fourth rule advises processing only essential data columns to decrease query time and resource consumption. Finally, the fifth rule recommends deferring complex operations like joins and aggregations to later stages in the processing pipeline, ensuring they're applied to a reduced dataset. These strategies, complemented by additional guidance on avoiding full scans and optimizing joins, aim to facilitate faster and more cost-effective data queries. Tinybird offers a platform for real-time data ingestion, SQL querying, and the publication of optimized queries as low-latency APIs, available for free without time constraints or credit card requirements.
Apr 24, 2025
1,569 words in the original blog post.
Dub serves as a model for modern small SaaS companies, particularly in how it facilitates real-time, user-facing analytics for marketing teams through its open-source platform. The company manages and shortens links to track performance and conversions, processing over 150 million events annually. Dub's technical architecture includes storing link metadata and click events using NextJS, PlanetScale, Prisma, and Tinybird, employing an event-driven approach to efficiently handle CRUD operations. Dub utilizes materialized views and columnar databases to optimize low-latency queries, choosing simplicity over complexity by using managed services like Tinybird's Events API for high-throughput ingestion. The company employs best practices such as filtering first, deferring joins, and selectively pre-aggregating data to maintain performance while offering dynamic filtering in its analytics APIs. This allows Dub to serve thousands of users in real-time, providing a flexible API that supports dynamic query patterns without affecting the frontend experience. Dub's approach emphasizes the importance of aligning data preparation with querying needs to ensure scalability and efficiency, making it a valuable case study in SaaS analytics.
Apr 24, 2025
3,856 words in the original blog post.
Since the 1960s, the computing power of devices has increased exponentially, yet many organizations still rely on outdated batch ETL processes for data management. The advent of real-time data processing offers transformative opportunities that batch processes cannot match, such as optimizing supply chains and enhancing customer experiences. Despite its potential, real-time data has not been widely adopted due to the comfort and familiarity associated with batch processing, giving a competitive edge to early adopters in industries like healthcare, financial services, and e-commerce. Implementing a successful real-time data strategy requires understanding the specific needs of business users and data engineers, ensuring it complements existing data management strategies without compromising financial performance. Organizations must evaluate their current infrastructure, team skills, and data sources to transition to real-time data effectively, considering tools like Tinybird that facilitate real-time data processing with SQL-based interfaces and managed services to minimize operational burdens.
Apr 24, 2025
4,437 words in the original blog post.
Amazon's DynamoDB is a highly efficient NoSQL key-value database optimized for real-time, distributed data storage within the AWS ecosystem, known for its exceptional read/write performance and scalability. As a serverless solution, it integrates seamlessly with other AWS services, offering high availability, fault tolerance, and multi-region replication, making it ideal for modern web applications that require fast data access without managing infrastructure. DynamoDB's strengths lie in its ability to handle high-frequency workloads with low-latency writes and reads, thanks to features like the Log-structured Merge tree and DynamoDB Accelerator. It is particularly suited for use cases such as gaming, content streaming, banking, mobile and web apps, and IoT, where real-time data processing and scalability are crucial. However, DynamoDB is not built for analytical processing or complex data modeling, as it lacks SQL support and can become cost-ineffective for wide scans and aggregations. Despite these limitations, it remains a robust choice for many applications, although alternatives like MongoDB, Apache Cassandra, and cloud-specific options are recommended when DynamoDB's capabilities do not align with specific needs.
Apr 24, 2025
2,610 words in the original blog post.
A conversation transcript with Claude, using the Model Context Protocol (MCP) Server for Tinybird Workspaces, explores the dynamics of the Bluesky social media platform. The transcript reveals insights into user engagement, network growth, and content creation by analyzing various data metrics such as follow activity, posting behavior, and account creation trends. It highlights the substantial network growth with peaks of 1.2 million follows per hour and the role of content creators in driving this growth. The findings show that active creators with 50 or more posts significantly boost follow activity, contributing to over 2.3 million follows in a day. A comprehensive dashboard was created to visualize these insights, featuring real-time charts and metrics on user engagement and platform dynamics.
Apr 24, 2025
1,196 words in the original blog post.
Tinybird has introduced a beta version of Tinybird Self-Managed, a deployable version that allows users to run the platform within their own cloud environments, providing complete control over their analytical backend while maintaining the platform's renowned speed, scalability, and ease of use. This version can currently be deployed on AWS infrastructure, with support for GCP and Azure forthcoming, and is designed to integrate with existing private data sources while allowing users to manage performance, scaling, and upgrades independently. Tinybird Self-Managed addresses the needs of customers requiring data sovereignty, deeper integration, infrastructure optimization, cost efficiency, and operational control, offering an OLAP database with ingestion APIs, an API gateway, observability, and backpressure mechanisms. It runs as a containerized deployment using Kubernetes for a straightforward yet scalable setup, with plans to expand its capabilities to include high availability, enhanced monitoring, and automated upgrades. While free during the beta phase, users must consider infrastructure costs, and the platform is particularly suitable for teams needing self-hosted solutions to meet regulatory requirements or specific business needs.
Apr 24, 2025
727 words in the original blog post.
The recent introduction of two pull requests in ClickHouse® 24.4 has significantly enhanced the performance of JOIN operations, with some queries experiencing speed increases by multiple orders of magnitude. The first pull request implemented a more sophisticated predicate pushdown technique using equivalence classes, which allows predicates to be applied to both sides of a JOIN operation, enhancing query performance by reducing the data processed. This technique leverages the equivalence of JOINed columns to optimize the logical query plan, improving the execution of INNER, LEFT, and RIGHT JOINs. The second pull request introduced a mechanism for automatically converting OUTER JOINs to INNER JOINs when predicates filter out non-joined rows with default values, further optimizing query execution by enabling more scenarios for predicate pushdown. These optimizations have resolved preexisting issues within ClickHouse® and have resulted in substantial improvements in query performance, making data processing more efficient for users.
Apr 24, 2025
1,095 words in the original blog post.
Tinybird's new feature, Versions, introduces integrated version control and CI/CD for real-time data projects, applying proven software engineering principles to data pipelines. This development addresses previous challenges such as lack of clear workflows, collaboration issues, and the risk of breaking changes in production, by allowing data teams to iterate and release projects directly from Git. Versions enhances the user experience by ensuring safer and more reliable deployments through features like branches for testing, Git integration, and CI/CD actions, which automate quality assurance and deployment processes. Tinybird aims to make real-time data product development more efficient and consistent, with feedback from early adopters indicating a preference for its simplicity and elegance over other tools. As Tinybird rolls out this feature to Enterprise and Pro customers, it seeks to further refine and expand on these capabilities, emphasizing the importance of maintaining data projects with the same rigor as traditional software projects.
Apr 24, 2025
2,729 words in the original blog post.
Log Analytics is an essential tool for developers to gain insights into their software systems, surpassing traditional Log Analysis by not only searching logs but also identifying trends and patterns. The Tinybird Log Analytics Starter Kit aims to simplify the process of building custom Log Analytics solutions by offering a language-agnostic, easy-to-deploy template that leverages the scalability and real-time processing capabilities of Tinybird. It allows developers to efficiently manage massive volumes of log data, thereby enhancing their ability to proactively monitor, diagnose, and address system errors. By providing a user-friendly interface and utilizing familiar SQL queries, the Starter Kit enables developers to create meaningful metrics and dashboards without extensive database or SQL expertise, thus facilitating more efficient, productive, and proactive software development processes.
Apr 24, 2025
2,368 words in the original blog post.
The recent enhancements to the pipe_stats_rt service Data Source include additional columns that capture detailed information about each request, such as the name of the requested pipe, bytes processed, number of rows read, request ID, token ID, and token name. Furthermore, the pipe_stats service, which aggregates data daily, now includes new columns for pipe name, processed bytes sum, and read rows sum. These improvements allow users to filter QueryAPI requests by pipe_id or pipe_name and to utilize the Tinybird UI for performance exploration. A new dashboard feature enables users to compare published Pipes by showing average and total processed data, while a graph illustrates the average processed data over time for a detailed view of individual Pipes. These updates are crucial for monitoring data processing, optimizing endpoint performance, identifying potential data leaks, and analyzing parameter efficiencies, with additional guidance available for optimizing API Endpoints.
Apr 24, 2025
336 words in the original blog post.
Tinybird has introduced a new feature that allows users to share Data Sources across different workspaces in a read-only mode, addressing customer demands for data reuse across projects and environments without repeated data ingestion. This feature supports teams in either reusing data for different projects such as real-time dashboards and alerts or separating data management responsibilities among team members. Users can share Data Sources with workspaces they have access to, using workspace names or IDs, and track sharing activity including who shared what data and when. Shared Data Sources are visually distinguishable with a special icon, and users can create Pipes or Materialized Views from them. Notifications are sent via email when changes occur to shared Data Sources, such as renaming or deletion, and users are informed of the shared status when listing Data Sources. While the feature currently does not support sharing through the CLI, it allows access to specific logs, but read-only users cannot view data quarantines.
Apr 24, 2025
475 words in the original blog post.
Resend is a developer platform designed to streamline the process of sending transactional and marketing emails by capturing and managing various email events such as sent, delivered, and bounced, among others. Through the integration with Tinybird's Events API, users can efficiently track and analyze these events using SQL queries, allowing them to assess the effectiveness of their email strategies, such as the performance of different email introductions. The platform supports webhooks for event data transmission and provides a dev stack analytics template featuring dashboards for popular developer tools. While focusing on important events like sent, delivered, and bounced, Resend advises against using tracking pixels for opened and clicked events to avoid being marked as spam. In the upcoming week, a series of videos will demonstrate the creation of a Resend dashboard, detailing each query and chart component, with the complete code accessible in the dev stack analytics template.
Apr 24, 2025
489 words in the original blog post.
chronark, a software engineer at Upstash, developed Planetfall, a global latency monitoring platform that enables developers to identify and resolve API latency issues across different regions to ensure optimal user experiences. The platform was inspired by the need for applications to perform well globally, not just in select regions, and leverages Tinybird for data aggregation and billing processes. While initially using MySQL for storage, chronark transitioned to a hybrid approach where Tinybird handles data aggregation to overcome MySQL's scaling limitations. Despite building Planetfall independently, chronark acknowledges the support and valuable feedback from beta testers and praises Tinybird's developer-friendly environment for accelerating development and innovation. Future plans for Planetfall include enhancing the debugging experience and integrating automatic checks within CI environments, with a continued reliance on Tinybird for feature development.
Apr 24, 2025
1,779 words in the original blog post.
Tinybird has launched Copy Pipes as a general availability feature, allowing users to seamlessly transfer the results of Tinybird Pipes into new or existing Data Sources, enhancing data management and application building capabilities without the need for APIs. Initially released in beta, Copy Pipes have quickly become popular, accounting for over 10% of the data written in Tinybird, and they support various use cases such as consolidated inventory snapshots, real-time OLTP enhancements, and change data capture (CDC). Unlike Materialized Views, which can be inappropriate for certain use cases, Copy Pipes provide a more efficient method for creating periodic data snapshots or deduplicating records. Users can now create, trigger, pause, and resume Copy Pipe jobs via Tinybird's UI and CLI, offering flexibility in scheduling and parameter usage. The official launch aims to improve data processing efficiency while addressing the needs of users seeking to streamline intermediate data handling in their data engineering workflows.
Apr 24, 2025
1,397 words in the original blog post.
User-facing analytics, also known as customer-facing analytics, involves embedding real-time data visualizations and data-driven features into software applications to provide end-users with immediate insights. Unlike traditional Business Intelligence (BI), which caters to a few internal stakeholders with complex queries and high latency, user-facing analytics delivers on-demand, low-latency analytics to a vast number of concurrent external users, often using real-time databases like ClickHouse® for fast data ingestion and complex query handling. This approach differs from embedded analytics by emphasizing performance and user experience, integrating analytics seamlessly into the software application without relying on slow, external dashboards. It poses challenges such as maintaining high data freshness, low query latency, and high concurrency, while also managing historical data retention. Companies like Canva, FanDuel, Vercel, Factorial, and Dub have implemented user-facing analytics to enhance user experience by providing live engagement statistics, real-time personalization, and dynamic data-driven interactions. The article highlights the importance of choosing the right architecture and technology, such as real-time databases, to ensure scalability and manage costs, while platforms like Tinybird simplify the process by integrating data ingestion, storage, querying, and API layers into a single platform, allowing developers to efficiently build user-facing analytics systems with just SQL skills.
Apr 24, 2025
3,380 words in the original blog post.
Tinybird is a platform that enables developers to create real-time analytics solutions across various use cases such as in-product analytics, usage-based billing, and personalized user experiences. It is trusted by a diverse range of companies, from startups like Vercel to large enterprises like FanDuel and Canva, due to its foundation on ClickHouse, a fast real-time analytics database, and its robust security features. The platform recently introduced the "Organizations" feature, which enhances enterprise readiness by allowing company administrators to manage multiple Workspaces, track data consumption, monitor API usage, and oversee members within their organization. This feature includes an Organizations Monitoring API for integrating metrics into external tools, offering admins a comprehensive view of cloud spend and user activity. Administrators can keep track of both billable and non-billable Workspaces, ensuring efficient resource allocation and management. Tinybird offers a free Build plan for small projects, with the option to upgrade to a Pro plan for more advanced features.
Apr 24, 2025
951 words in the original blog post.
Tinybird's "tb create" feature is part of the new Tinybird Forward UX, designed to streamline the process of building and deploying data projects by leveraging AI for efficiency and practicality. The tool enables users to quickly initiate data projects by automating tasks such as setting up databases, creating tables, configuring API endpoints, and establishing CI/CD workflows, which traditionally require significant time and effort. By embedding AI selectively and purposefully, Tinybird aims to enhance productivity without succumbing to AI hype, allowing data engineers to focus on solving real problems while adhering to best practices. The "tb create" command is central to this approach, offering a structured and opinionated framework for launching projects, whether starting from scratch or adding to existing ones, and it supports a variety of workflows including AI-generated project scaffolding, data source creation from existing files, and integration with IDEs for more customized control.
Apr 24, 2025
774 words in the original blog post.
Developers often face challenges when working with production data due to issues like compliance, data format, and accessibility, necessitating the use of mock data streams that mimic real-world data. Mockingbird, an open-source library, was launched to simplify the generation of these streams, allowing users to define data schemas in JSON and send data to platforms like Tinybird. Building on this, Tinybird has introduced AI-powered Demo Data, a feature that uses generative AI to create mock data schemas based on natural language prompts, thereby easing the schema generation process. This innovation leverages OpenAI's GPT-3.5 to generate schemas for mock data streams, allowing developers to efficiently create real-time applications by describing desired data patterns to AI. The tool is available across all Tinybird pricing plans, including a free tier, and encourages users to explore AI's potential in generating mock data for testing and development.
Apr 24, 2025
1,610 words in the original blog post.
DynamoDB, a fast and scalable NoSQL key-value store, is widely used for real-time applications but lacks native support for aggregations, prompting developers to employ various alternative methods to perform such operations efficiently. These methods include using Amazon Redshift for batch analytics, which integrates easily with DynamoDB for large-scale analytical processing but is unsuitable for real-time analytics. Another approach involves using Apache Hive on EMR, which offers flexibility for complex queries but requires a complex setup and only supports batch processing. For real-time aggregation, AWS Lambda can be used in conjunction with DynamoDB Streams, providing low latency for user-facing applications, though it offers limited query flexibility and potential inconsistencies if Lambda fails. Tinybird offers a real-time solution with SQL-based dynamic queries and scalable API endpoints, yet it is a managed service that may not suit those preferring self-hosted options. Each method has distinct advantages and limitations, catering to different needs based on the desired flexibility, performance, and cost considerations.
Apr 24, 2025
2,376 words in the original blog post.
Tinybird is focused on enhancing the developer experience by enabling the swift creation and iteration of real-time data pipelines, from ingestion to API deployment, without the complexities of traditional data handling methods. The company prioritizes usability and efficiency, addressing common pain points in data engineering such as fear of altering pipelines due to potential downtime or increased costs. Tinybird integrates with platforms like Kafka, BigQuery, and Snowflake, allowing developers to manage data projects in various environments, including popular IDEs. The platform emphasizes the importance of workflow optimization over quick technical fixes, advocating for transparent changes, version control best practices, and reliable testing strategies to facilitate continuous delivery with data. With a commitment to flexibility and adaptability, Tinybird aims to empower developers to iterate data solutions efficiently, aligning with industry standards and community feedback to continually enhance their platform's capabilities.
Apr 24, 2025
1,976 words in the original blog post.
Tinybird, a company focused on simplifying real-time analytics for data and engineering teams, has raised $30 million in a Series B funding round led by Balderton Capital, with contributions from existing investors CRV, Singular, and Crane. The company aims to enable developers to build fast, user-facing applications by providing an end-to-end platform that facilitates the ingestion, transformation, and visualization of real-time data, significantly reducing query latencies and infrastructure requirements. Tinybird's platform allows developers to integrate complex data infrastructure into applications seamlessly, offering features such as high-concurrency, low-latency REST APIs and real-time data visualizations. By focusing on the developer experience and making real-time data scenarios accessible and cost-effective, Tinybird is catering to the growing demand for immediate, data-driven user experiences, facilitating the creation of dashboards, personalization, and anomaly detection services. The company is expanding its capabilities across multiple cloud platforms and continues to add features to enhance its offerings, while also engaging with the developer community through workshops and hiring initiatives.
Apr 24, 2025
836 words in the original blog post.
Tinybird has introduced a significant update to its Command Line Interface (CLI) to create a local-first experience that simplifies and automates the deployment of data infrastructure projects. This new CLI is designed to eliminate the manual, error-prone processes traditionally associated with deploying changes to production, like creating and backfilling materialized views, by allowing developers to make, test, and deploy changes locally with a single command. Inspired by frameworks like Next.js, Tinybird's new CLI offers an intuitive onboarding process, enhanced by leveraging Large Language Models (LLMs) for generating project scaffolding and tests. The CLI uses a reduced set of primitives—data sources and pipes—to allow developers to efficiently manage and transform data, while also providing fast iteration and feedback loops. The update includes features such as environment differentiation for testing and deployment, instant feedback on project changes, and comprehensive test generation, all aimed at improving developer experience and efficiency. Codenamed FORWARD, this CLI is in beta and reflects Tinybird's broader initiative to provide an analytics backend that minimizes complex infrastructure work for developers.
Apr 24, 2025
1,185 words in the original blog post.
James Devonport, founder of UserLoop, discusses his journey in building a platform aimed at enhancing Shopify merchants' understanding of their customers through direct feedback and zero-party data. Initially built on Bubble, UserLoop faced challenges with latency when handling large datasets, prompting Devonport to explore other analytics solutions. He discovered Tinybird, which provided a cost-effective, efficient, and user-friendly platform for managing analytics over large data volumes, significantly improving performance and reducing costs. The integration of Tinybird allowed Devonport to run complex analytics and publish them as APIs, enhancing UserLoop's capabilities. Tinybird's ease of use, coupled with excellent support, helped overcome previous limitations faced with other platforms. Looking ahead, UserLoop aims to expand its features, including AI capabilities and video surveys, with Tinybird as a central component in their data architecture, ensuring scalability and performance. Devonport emphasizes Tinybird's suitability for SaaS developers needing quick deployment and scalability, highlighting the value of its support team and community.
Apr 24, 2025
1,290 words in the original blog post.
The blog post discusses practical and efficient methods for implementing AI features in production, focusing on five key functionalities: vector search, AI-based filtering, visualization, auto-fix, and explanation of complex concepts. Vector search is used to find similar items in a database by calculating and comparing embeddings, while AI filtering allows users to use free-text input to refine data dashboards through structured query parameters. Visualization with AI provides customizable data views based on natural language queries, enhancing user interaction with analytical APIs. The auto-fix feature leverages AI to correct errors in code development, minimizing downtime caused by syntax issues. Lastly, the explanation feature utilizes large language models (LLMs) to aggregate information from multiple sources, facilitating support and documentation processes. The post emphasizes the value of these AI-driven functionalities and encourages developers to prioritize building practical solutions over being swayed by industry hype.
Apr 24, 2025
2,155 words in the original blog post.
Tinybird, originally developed to address internal challenges of managing large volumes of real-time data, has evolved over five years into a product aimed at simplifying data handling through a local-first development workflow. This new approach aligns with typical software development practices, allowing users to write, test, and deploy code locally before pushing it to production, thereby eliminating the need for browser-based UI setup. The upcoming Tinybird local experience promises features such as local project building, mock data generation, full project validation, end-to-end API testing, seamless CI/CD deployment and rollback, and live schema migrations without data loss. Additionally, it integrates with AI-powered IDEs to streamline development processes and offers built-in observability for scalable cloud operations. The beta version of this local-first experience will soon be available for testing, with updates to be shared via Tinybird's social media and community platforms.
Apr 24, 2025
470 words in the original blog post.
Tinybird has launched Forward, a new user experience aiming to accelerate the development of software with big data requirements by providing instant feedback through tb dev, a local data development environment. This tool offers real-time validation of data schemas, SQL queries, and API endpoints, facilitating a fast and fluid development process similar to hot reloading in frontend development. By enabling immediate feedback on changes to data projects, tb dev simplifies debugging and iteration, allowing developers to work efficiently with both local mock data and production data. It supports instant SQL query iteration, local API endpoint testing, and real-time debugging, promoting a rapid and intuitive development loop. Additionally, it offers commands for building and testing projects, ensuring valid deployments through CI/CD integration, thereby aiming to eliminate the traditionally slow feedback loops associated with big data systems.
Apr 24, 2025
1,047 words in the original blog post.
A seemingly minor inefficiency in an integration test for Tinybird, a real-time data platform, led to significant system optimizations. Initially, a single test took a second instead of a millisecond, prompting the team to investigate and discover inefficiencies in Python's Global Interpreter Lock (GIL) and the use of ProcessPoolExecutor for parallel processing. By shifting CPU-intensive tasks to C++ extensions and using the ThreadPoolExecutor with refined control over the GIL, they achieved a 50% reduction in global memory usage, decreased CPU usage by 10-20%, reduced the number of threads and processes by 60-70%, and virtually eliminated I/O traffic. This optimization not only improved performance but also simplified system management, demonstrating the value of questioning existing assumptions and exploring alternative solutions to enhance application efficiency.
Apr 24, 2025
2,670 words in the original blog post.
TheirStack.com is a market intelligence tool designed for B2B SaaS companies to identify potential buyers by analyzing technologies mentioned in job postings from numerous companies. Initially using PostgreSQL for text search, the team encountered limitations due to case sensitivity, tokenization, and lack of custom dictionary support, leading them to explore alternatives such as regex matching and trigram indices. Despite some improvements, data growth rendered these methods inefficient, prompting a switch to ClickHouse® via Tinybird, providing faster pattern matching on larger datasets. The use of the multiMatchAllIndices function allowed for efficient parsing of numerous patterns in a single query, significantly reducing data read and query count. This approach facilitated the transition from 180,000 monthly queries to a drastically lower number, enhancing performance and efficiency in identifying relevant technologies for TheirStack's clients.
Apr 24, 2025
2,764 words in the original blog post.
Typeform, a platform for creating no-code forms, quizzes, and surveys, utilized a hackathon to enhance their product by incorporating real-time tracking of UTM parameters with the help of Tinybird. This allowed for improved data analysis of form interactions, such as views, submissions, and completion rates, across various platforms like Facebook and Instagram. The challenge was to integrate disparate data points into a cohesive, real-time dashboard for customers. Using Tinybird, Typeform's team developed an API to group and filter UTM parameters by different metrics, achieving significant performance improvements through the use of materialized views and snapshots. Over 3 million events were processed in 12 hours, maintaining low latency and enhancing user experience with almost instantaneous data filtering capabilities. The hackathon not only demonstrated the efficiency of Tinybird's serverless architecture but also highlighted the ease of integration and configuration, enabling Typeform to deliver valuable new functionality quickly.
Apr 24, 2025
855 words in the original blog post.
Real-time data ingestion is a crucial component in modern data architectures, enabling the capture and utilization of data as soon as it is generated to maximize its value for downstream analytics and business applications. This process is essential for building event-driven architectures and is distinguished by the concept of "freshness"—ensuring data is available for use almost immediately, unlike traditional batch processing methods. The article details how real-time data ingestion powers various industries such as retail, finance, and transportation by enabling applications like inventory management, fraud detection, and logistics tracking, respectively. It distinguishes between real-time data ingestion and event streaming, explaining that while event streaming involves moving data in flight via platforms like Apache Kafka, real-time ingestion focuses on storing this data efficiently for immediate access and processing. The piece also explores the importance of monitoring and observability to ensure data reliability, emphasizing the need for robust metrics, logs, and traces to maintain system health. Additionally, it highlights the need for performance, scalability, and cost-efficiency in real-time pipelines, suggesting strategies like tiered storage and unifying batch and streaming pipelines for optimal operations. With tools like Tinybird providing end-to-end solutions for real-time data platforms, the article underscores the transformative potential of real-time data ingestion for businesses looking to leverage data for competitive advantage.
Apr 24, 2025
3,893 words in the original blog post.
Tinybird has launched Forward, an evolution of its platform aimed at enhancing the experience of building real-time data applications by aligning with modern development practices and AI-native workflows. This update addresses previous user challenges, such as managing schema changes and aligning with CI/CD workflows, by introducing a local-first development experience, AI-assisted features for rapid prototyping, and seamless integration with AI coding assistants like GitHub Copilot. Forward offers faster build times, automated schema migrations, and the ability to deploy Tinybird on personal infrastructure, currently supporting AWS, with plans for other clouds. It retains Tinybird's core strengths of speed and scalability while making it more accessible for developers and enabling a smoother transition into AI-assisted development workflows. Existing users can upgrade to Forward or continue using the classic version, as Tinybird aims to cater to the future needs of developers in creating sophisticated, AI-driven, real-time data projects.
Apr 24, 2025
926 words in the original blog post.
Many modern applications, typically centered on transactional operations like creating and updating records, are increasingly incorporating analytical experiences to enhance user engagement, such as tracking price changes or post engagement metrics. While databases like Amazon DynamoDB are optimal for handling transactional workloads due to their speed and scalability, they are not suitable for analytical processing, which requires aggregating large datasets. To address both transactional and analytical needs within an application, a combined approach using DynamoDB for CRUD operations and Tinybird for analytical workloads is effective. DynamoDB excels in maintaining consistent performance for high-throughput, low-latency operations by utilizing partition and sort keys, while Tinybird handles real-time data ingestion and efficient analytical queries through its columnar storage model and partitioning strategies. This dual-database architecture, leveraging DynamoDB's transactional strengths and Tinybird's analytical capabilities, supports diverse user demands and future scalability while maintaining architectural simplicity and performance.
Apr 24, 2025
2,537 words in the original blog post.
Tinybird and Vercel have established a strong partnership, enhancing developers' ability to build and iterate quickly using the combined strengths of both platforms. Tinybird, known for its analytics capabilities, now offers a first-party integration with Vercel through the Vercel Integrations Marketplace, making it the first purpose-built analytics backend available there. This integration simplifies the process of connecting Tinybird to Vercel projects, allowing developers to manage analytics without worrying about infrastructure, scale, or security. The integration supports various use cases, including web analytics, structured logging, and anomaly detection, and offers Starter Kits, such as a Web Analytics Starter Kit and a Log Analytics Starter Kit, to help developers quickly deploy and customize analytics solutions. Notably, projects like Beam Analytics, Dub, and Planetfall have successfully utilized the synergy between Tinybird and Vercel to deliver innovative analytics solutions.
Apr 24, 2025
867 words in the original blog post.
As businesses grow and their datasets expand from gigabytes to terabytes, managing and querying large-scale data in Postgres presents several challenges, including slower queries, increased I/O operations, index inefficiency, longer maintenance operations, and increased memory usage. To address these issues, various strategies can be employed, ranging from basic maintenance tasks like regular VACUUM and ANALYZE operations to advanced techniques such as table partitioning, sub-partitioning, cascading materialized views, and vertical partitioning. Partitioning helps manage large tables by splitting them into smaller, more manageable chunks, although it increases the complexity of database structures and queries. Sub-partitioning offers even finer control but requires robust database administration. Vertical partitioning can improve query performance by separating infrequently accessed columns, though it complicates schema design and data consistency management. For more extreme cases, sharding distributes data across multiple instances, enabling horizontal scaling but introducing significant complexity in application logic and system architecture. Recognizing when Postgres is nearing its limits is crucial, prompting consideration of distributed SQL databases, NoSQL solutions, or OLAP systems. Continuously monitoring database performance and adapting architecture as needed is essential to effectively manage terabyte-scale data.
Apr 24, 2025
2,480 words in the original blog post.
Incorporating Large Language Models (LLMs) into real-time dashboards can enhance their functionality, particularly through natural language filtering. This approach replaces traditional complex filter menus with a simple text input that an LLM interprets to generate structured filter parameters, which are then applied to the dashboard. The process involves creating an API route that accepts user input, uses a system prompt for the LLM to produce the necessary filter parameters, and updates the dashboard based on these new filters. The tutorial details the use of Tinybird's API for efficient data querying and illustrates how the LLM Performance Tracker template can be employed for implementing natural language filters. This method is advantageous for handling large datasets with multiple dimensions, and while performance challenges remain, such as the LLM response time, strategies like using WebLLM can mitigate these. The tutorial emphasizes the transformative impact of AI on data visualization, providing resources like the LLM Performance Tracker for practical implementation.
Apr 24, 2025
1,593 words in the original blog post.
The Hotels Network (THN) collaborates with Tinybird to enhance real-time personalization for hotel booking websites by segmenting visitors and processing data through API endpoints. THN faced challenges in optimizing performance due to high read and write demands, making over 120 requests per second. Efforts to improve involved simplifying API endpoints, adjusting data partitioning strategies, and reducing query complexity by minimizing the number of data sources. While some strategies like partitioning by user key initially caused disk throttling, transitioning to a quarterly partition with a time-to-live (TTL) setting helped manage data accumulation. Further optimizations included reducing index granularity and altering join methods, leading to a significant reduction in endpoint response time from 340 ms to 170 ms. Despite these improvements, ongoing performance enhancements are pursued, such as experimenting with compact parts and profiling tools to identify additional areas for refinement, underscoring the importance of understanding how ClickHouse® merges parts for optimal performance.
Apr 24, 2025
1,367 words in the original blog post.
The blog post offers a detailed walkthrough of designing and implementing a public-facing data API to deliver near real-time weather conditions across 175 US cities, utilizing Tinybird's core features like Data Sources and Pipes. It illustrates the process of capturing weather data from OpenWeatherMap and ingesting it into Tinybird for analysis, followed by steps to develop and publish the API. The author emphasizes the importance of thoughtful API design, focusing on endpoint selection, response formatting, supported HTTP methods, and query parameters, ensuring they are well-documented and user-friendly. The post explains how Tinybird's building blocks, such as Workspaces, Data Sources, Pipes, Nodes, and API Endpoints, facilitate scalable and performant API creation. Additionally, it highlights Tinybird’s capabilities in serverless hosting, data storage, scaling, security, and support, making it a viable choice for developing high-concurrency, low-latency APIs for various real-time data use cases.
Apr 24, 2025
3,334 words in the original blog post.
The blog post provides a comprehensive guide on building a real-time React dashboard on top of Google BigQuery using Tinybird, Next.js, and Tremor components, addressing the limitations of traditional dashboarding methods like Looker and Looker Studio, which struggle with high concurrency and low-latency demands. It explains the significance of real-time dashboards in providing fresh data and supporting numerous concurrent users, which traditional approaches fail to accommodate due to BigQuery's inherent design for batch processing and limited concurrency capabilities. The tutorial demonstrates how Tinybird enhances the process by replicating BigQuery tables into a real-time analytics platform, enabling rapid SQL-based metric creation and API publishing for seamless integration into frontend applications, thereby overcoming BigQuery's latency and concurrency challenges. Through a step-by-step approach, it details the setup and implementation using Tinybird's BigQuery Connector, Pipe creation, and API endpoint publication, culminating in a Next.js app utilizing Tremor components to visualize the data in real-time, showcasing the effectiveness of Tinybird in delivering high-performance dashboards.
Apr 24, 2025
2,275 words in the original blog post.
Tinybird, a real-time data platform, aims to bridge the gap between data engineering and software engineering by integrating best practices such as version control, testing, and CI/CD into its user experience. Despite the challenges faced by many data engineers, such as the absence of systematic testing and deployment processes, Tinybird offers solutions through its Versions feature, which leverages Git for version control and provides a consistent project structure for data pipeline code. This approach facilitates collaborative development, enhances reliability, and reduces deployment risks by automating CI/CD operations with improved command line tools and integrating data testing strategies. Tinybird's platform enables data teams to manage data pipelines as code, ensuring traceability and auditability, and provides tools for testing with production data, making it easier to identify potential regressions. By aligning its platform with established software engineering principles, Tinybird empowers organizations to build and deploy real-time data products more efficiently and confidently.
Apr 24, 2025
2,309 words in the original blog post.
The blog post provides a detailed tutorial on setting up an event-driven real-time data ingestion system using AWS S3 Event Notifications and AWS Lambda, with the aim of sending file change events to the data platform Tinybird. It outlines the process of creating an S3 bucket and a corresponding Lambda function configured with Python 3.10 runtime on an arm64 architecture for cost efficiency. The tutorial guides the user through configuring IAM permissions, writing Lambda code to generate presigned URLs for file sharing, and sending these URLs to Tinybird's API for data ingestion. The post also includes instructions for setting up S3 Event Notifications to trigger the Lambda function upon new file uploads, specifically targeting CSV files. The author emphasizes the flexibility of event-driven architectures and suggests alternative methods, such as using message queues like SNS or SQS, for processing file changes. The post concludes by encouraging readers to explore Tinybird's features and community resources to enhance their understanding and implementation of real-time data workflows.
Apr 24, 2025
2,173 words in the original blog post.
Tinybird has introduced Tinybird Local, a Docker container that allows developers to run a full instance of the Tinybird platform on their laptops, thereby enabling local development, testing, and deployment of data applications. This shift to a local-first approach was driven by user feedback and the need for greater control and flexibility, particularly in scenarios where offline development is necessary or when testing data pipelines within CI environments without relying on cloud services. Tinybird Local is designed to mirror the cloud version closely, maintaining compatibility while optimizing for developer experience with features like streamlined architecture, reduced memory footprint, and CPU usage controls. It does, however, lack some cloud-specific functionalities, and its single-node execution may affect the behavior of distributed queries. Despite these differences, Tinybird Local offers a robust environment for experimentation and learning, with the possibility of deploying it to production settings. The development team is actively seeking community feedback to enhance the platform further, with future plans including better storage options and additional deployment configurations.
Apr 24, 2025
1,413 words in the original blog post.
Building AI features presents unique challenges due to their stochastic nature and non-deterministic costs, making observability crucial for performance monitoring and iteration in production. The LLM Performance Tracker is an open-source solution designed to help AI engineers and developers track the performance of large language models (LLMs) across applications, offering real-time visualization of key metrics like cost, time-to-first-token, and total requests. Built with a tech stack including Next.js, Tinybird, and Vercel, this tool allows for multi-dimensional filtering, drilldowns, and multi-tenancy support, enabling users to analyze usage patterns, evaluate model efficiency, and assess the impact of parameters like "temperature" on response quality and costs. The app is customizable and extendable, supporting integration with observability platforms for alerting on usage anomalies, while also providing a template for deploying analytics dashboards with advanced features, such as user authentication and multi-tenant dashboards, using Clerk and Tinybird. Users can easily fork and adapt the project to fit specific needs, and for those seeking more advanced observability options, Dawn offers comprehensive AI monitoring solutions.
Apr 24, 2025
1,218 words in the original blog post.
A major U.S.-based eCommerce retailer, facing challenges with Google Analytics, developed its own privacy-first, real-time analytics platform to address scalability, data freshness, and legal compliance issues. Despite being a large entity with significant daily web traffic, the limitations of Google Analytics, such as data latency and incompleteness due to ad-blockers and cookie consent, prompted the retailer to create a first-party tracker. This tracker utilizes Kafka for event streaming and Tinybird for real-time analytics, allowing them to capture 100% of traffic and ensure GDPR compliance by using first-party cookies and anonymized session IDs. The new system enables the retailer to maintain control over data flow and attribution models, achieve near-instantaneous data freshness, and adaptively respond to real-time customer demands, especially during peak periods like Black Friday. With this bespoke solution, the company not only addresses privacy and data accuracy concerns but also discovers new operational use cases, positioning itself to proactively meet future analytical needs.
Apr 24, 2025
1,937 words in the original blog post.
Tinybird offers a solution for quickly uploading and querying large CSV files without the need for additional software installations or reliance on cumbersome online tools. Users can upload CSVs from local sources or remote URLs, perform swift SQL queries directly in their browser leveraging ClickHouse technology, and easily share results via snapshots or dynamic API endpoints. Tinybird supports a variety of file types, including NDJSON and Parquet, and enables users to execute complex operations like joins and aggregations efficiently. The platform allows for the creation of API endpoints from query results, facilitating data sharing in multiple formats such as CSV, JSON, and Parquet. Additionally, Tinybird provides a free plan allowing users to store up to 10 GB of data and perform up to 1,000 API requests daily without cost, making it a versatile tool for managing and analyzing extensive datasets.
Apr 24, 2025
605 words in the original blog post.
Modern applications rely on real-time data for decision-making, with platforms like Confluent leading in streaming data services by extending beyond Apache Kafka. Companies such as Tinybird leverage Confluent for various applications, including inventory management and sports betting enhancements. Real-time data often requires enrichment from additional sources to maximize its value, such as combining sales streams with inventory data for automated stock decisions. The newly launched Tinybird Confluent Connector facilitates the integration of Confluent streams into Tinybird's system, enabling the creation of low-latency APIs for user-facing scenarios like business intelligence and personalization. Users can set up the connector through Tinybird's UI or CLI, allowing effortless data ingestion and processing. Tinybird offers a free plan and encourages engagement with its community for support and feature requests, while also promoting learning opportunities through events like their Launch Week Release Round-up.
Apr 24, 2025
848 words in the original blog post.
The post explains how Tinybird can be used internally to analyze Nginx traffic in real-time and automatically, providing a guide for users to do the same. To follow the process, users need a Tinybird account, code from a specified GitHub repository containing necessary configurations, and the tbtail tool to stream logs to Tinybird. The service, currently in private beta, requires users to contact Tinybird for account activation. Once the required tools are set up, users can clone the data project locally, set it up on Tinybird, and begin streaming logs. The post details how to install tbtail and use it to automatically send logs to Tinybird, allowing users to access real-time traffic insights via a dashboard and utilize the query_grouped_requests pipe for aggregated data queries.
Apr 24, 2025
316 words in the original blog post.
Tinybird has significantly enhanced its data ingestion capabilities, transitioning from handling monthly data volumes to processing petabytes daily, enabling developers to efficiently create powerful applications. The platform's ease of use, based on SQL queries, allows users to transform and enrich data streams quickly, fostering rapid development of new real-time products. Tinybird's recent updates include a revamped ingestion interface and plans to integrate more native data source connectors, alongside a Connector SDK for custom integrations. High-profile users like an eCommerce giant, The Hotels Network, and Vercel leverage Tinybird's native connectors and Events API for personalized services and analytics. As Tinybird continues to innovate in data ingestion, its focus remains on expanding support for a wider array of data sources and maintaining robust customer support.
Apr 24, 2025
1,547 words in the original blog post.
Tinybird, a platform for building real-time data products, is now available on Amazon Web Services (AWS) for self-service Free and Pro plans, offering customers the flexibility to deploy in AWS regions such as US East and EU Central. This expansion allows businesses to leverage Tinybird's capabilities for real-time data analytics, enabling faster time-to-market and innovation through real-time data engineering. Tinybird facilitates the integration and transformation of streaming and batch data sources using SQL and Git to create scalable APIs, making it easier for organizations to access and analyze data in real-time. With native connectors to AWS services like MSK, S3, and Kinesis, Tinybird streamlines data ingestion and transformation processes, ensuring high performance and scalability. Companies such as FanDuel and Factorial have successfully used Tinybird on AWS to enhance their data-driven offerings, citing its simplicity, performance, and integration capabilities as key benefits. As Tinybird becomes available in the AWS Marketplace, it simplifies billing and invoicing, making it an attractive option for businesses aiming to optimize cloud spending while enhancing their real-time data strategies.
Apr 24, 2025
1,353 words in the original blog post.
The Tinybird CLI, a crucial tool for interacting with Tinybird APIs, has reached a significant milestone with the release of its stable version, v1.0.0, now available on PyPi. This development follows over three years of beta testing and 448 incremental releases, which, while allowing rapid updates based on customer feedback, also resulted in version noise and confusion due to the lack of semantic versioning. The new stable release introduces a semantic versioning scheme (MAJOR.MINOR.PATCH) to help users easily discern changes, whether they are breaking, backward-compatible, or bug fixes. Customers can now expect regular weekly updates, enhancing the CLI's functionality and user experience while maintaining transparency about the nature of each release. The release is a testament to the collaborative effort between Tinybird and its customers, whose feedback and testing have been pivotal in refining the CLI, ensuring it remains a powerful and user-friendly tool within Tinybird’s real-time data platform.
Apr 24, 2025
536 words in the original blog post.
Tinybird has secured a $37 million Series-A investment from CRV, Singular, and Crane, with additional support from business angels, to further its mission of enabling developers to build scalable realtime data products using familiar tools like SQL and APIs. The investment will be used to enhance developer experience through improved Git integration, schema management, and materialization, and to expand serverless and multi-cloud capabilities to address the increasing demand for fast, reliable data queries. Tinybird aims to simplify data ingestion from various sources and improve monitoring and observability, with plans to release new connectors in the coming months. The company emphasizes the importance of speed in decision-making and product development, striving to enhance its infrastructure's elasticity and reactivity while addressing challenges such as data egress costs.
Apr 24, 2025
835 words in the original blog post.
Time series predictions are a valuable tool for companies aiming to anticipate future events and make informed decisions, with various methods ranging from sophisticated machine learning algorithms to simpler statistical approaches. An example of a straightforward method is using the NYC taxi dataset to predict taxi pick-ups, where data from 2017 is used for training and 2018 for validation, leveraging the Tinybird Datasource API. The prediction assumes that the number of pick-ups mirrors the previous year, adjusted for average annual growth, and involves techniques like using the addYears function for date matching and adjusting for the day of the week. The model performs well with deviations typically under 10%, and improvements could include accounting for outlier dates and analyzing model performance to rectify inaccuracies. Further analysis could enhance the model's utility by examining pick-up patterns per taxi zone or hour, and future discussions will explore these advanced analyses.
Apr 24, 2025
856 words in the original blog post.
Tinybird, a real-time analytics platform built on ClickHouse, offers an efficient solution for processing streaming data by providing an intuitive SQL-based environment for data teams to build high-concurrency, low-latency APIs. The platform excels in managing the complexity of streaming data, which continuously flows from various sources at high volume and velocity, and ensures data durability, reliability, and real-time processing capabilities. By integrating with Confluent, a cloud-native streaming service built on Apache Kafka, Tinybird empowers data engineers and developers to build differentiated features faster, leveraging both platforms' strengths. The partnership, highlighted by the "Connect with Confluent" program, simplifies the development of real-time data pipelines, enabling businesses to gain insights and competitive advantages by quickly processing and analyzing data. Tinybird's model allows users to connect batch and streaming data sources without additional tooling, offering a seamless way to enhance real-time analytics use cases.
Apr 24, 2025
1,272 words in the original blog post.
The text explores the challenges of handling large datasets for data visualization, specifically focusing on rendering a candlestick chart using an API endpoint from Tinybird. It highlights issues such as client-side rendering performance, data privacy, and the limitations of tools like Excel when analyzing massive datasets. The solution involves creating a dynamic API that delivers OHLC (Open, High, Low, Close) data and trading volumes for any specified date range using Bitcoincharts' data on the Coinbase BTC/USD pair. By converting the unixtime to a DateTime format and implementing a manageable data ingestion process, the API ensures efficient data delivery and visualization without overwhelming the client-side rendering pipeline. The text concludes by stating that Tinybird allows dynamic responses from analytics API endpoints, offering greater control over data transfer for performance and privacy considerations.
Apr 24, 2025
696 words in the original blog post.
The current data integration landscape, dominated by poll-based ETL pipelines, creates challenges due to the added load on source data systems and the resulting outdated data in downstream systems. These traditional pipelines often require full table scans, leading to performance issues and high costs, while also failing to support real-time data applications due to inherent latency. As a solution, the text explores event-driven architectures, which trigger data ingestion based on events, allowing data to be processed immediately and reducing the strain on source systems. Event-driven architectures use approaches like event streaming, message queuing, and serverless functions to provide fresher data and support real-time use cases. Change Data Capture (CDC) is also discussed as a viable alternative when backend code modifications are not feasible, leveraging database logs to capture changes without additional load on the application database. The text highlights the benefits of using modern cloud storage services for event-driven file ingestion, which enhance data freshness and system robustness. Although message queues are preferred for their flexibility and durability, serverless functions can trigger processes directly for simpler use cases, though they may lack the flexibility of message queues. Ultimately, transitioning to event-driven architectures can enable real-time data processing, crucial for applications requiring immediate data access.
Apr 24, 2025
3,598 words in the original blog post.
Real-time personalization is a dynamic approach that tailors user experiences based on live data from individual user preferences, behaviors, and contextual information, offering significant benefits over traditional web personalization by using in-session data to increase conversion rates and customer satisfaction. This practice is particularly valuable for digital enterprises, such as online retailers and travel agencies, aiming to enhance user experiences and boost revenue without relying on costly or inflexible off-the-shelf software solutions. Instead, real-time data platforms like Tinybird allow companies to build customizable real-time personalization systems efficiently by handling data ingestion, processing, and API publication, reducing the overhead of managing complex infrastructures. Examples across industries include personalized shopping offers, tailored travel experiences, customized sports betting, and social media content feeds, all of which demonstrate the effectiveness of real-time personalization in increasing customer engagement and brand affinity.
Apr 24, 2025
3,182 words in the original blog post.
Jamie Barton, a Developer Relations professional at Grafbase, discusses the advantages of using Tinybird for real-time analytics in a conversation with Tinybird. Grafbase, founded by Fredrik Björk in 2021, is a serverless data platform designed to simplify the creation of GraphQL backends by unifying data sources into a single endpoint. When building their analytics feature, Grafbase sought a solution that avoided the complexities of setting up a data warehouse and found Tinybird to be the ideal fit, providing real-time data processing and low-latency analytics. Tinybird's ability to integrate with multiple data sources, such as Rudderstack, Cloudflare, and DynamoDB, and its user-friendly API design were important factors in Grafbase's decision. The platform allows Grafbase to swiftly publish and query data, which supports both customer-facing analytics and internal business intelligence. Looking forward, Grafbase aims to enhance its offerings by incorporating real-time functionality into various GraphQL components while relying on Tinybird for usage tracking and API management. Jamie emphasizes the importance of development speed and the benefits of using a well-designed, integrated platform like Tinybird, recommending it to other developers for its efficiency in bringing products to market.
Apr 24, 2025
1,178 words in the original blog post.
PostgreSQL is a popular choice for startups due to its open-source nature, robustness, and user-friendly SQL dialect, making it a versatile tool capable of handling various tasks, from transactional to basic analytical workloads. Its rich ecosystem and ACID compliance make it an economical and reliable option, especially with the availability of managed and serverless solutions that simplify deployment. However, as startups scale, they may encounter performance challenges such as slow query execution, increased I/O wait times, higher CPU utilization, and lock contention, leading to degraded user experiences and increased error rates. To address these issues, proactive strategies like regular performance audits, caching, schema optimization, connection pooling, and data partitioning are recommended to maintain optimal performance. As data volumes and user concurrency increase, businesses may need to explore more advanced solutions for handling analytics workloads, possibly transitioning some processes off Postgres. Future articles and solutions like Tinybird, designed for real-time data infrastructure, offer additional insights and tools for managing these challenges.
Apr 24, 2025
2,014 words in the original blog post.
When developing applications, traditional relational databases like Postgres and MySQL, or NoSQL databases such as MongoDB, are commonly used for their general-purpose capabilities and transactional features. However, for real-time analytics and large-scale data queries, columnar databases like ClickHouse, Pinot, and Druid significantly outperform traditional databases due to their optimization for online analytical processing (OLAP). These column-oriented databases store data in columns rather than rows, enhancing data locality and allowing for faster query processing, especially for aggregation and filtering tasks typical in analytics. Concepts such as compression, vectorization, caching, sorting, and parallelization further optimize performance by ensuring data is processed swiftly and efficiently. Additionally, features like probabilistic data structures, fast writes with log-structured merge-trees (LSMTs), incremental materializations, and specialized functions for time series data make columnar databases particularly suitable for handling big data and real-time analytics. While they may present management challenges due to their capacity to handle vast amounts of data, columnar databases offer a compelling alternative for projects requiring extensive analytical capabilities, with platforms like Tinybird simplifying their integration and use.
Apr 24, 2025
2,189 words in the original blog post.
Feature stores are integral components in modern machine learning operations (MLOps) as they provide a centralized repository for machine learning features, enhancing model accuracy and operational efficiency by allowing data teams to share and reuse features across different projects. They address common challenges in machine learning deployments, such as data consistency and redundancy, by ensuring that models are trained and served with the same data set, akin to a database enforcing data integrity. Feature stores are typically divided into online and offline variants; offline stores focus on batch data processing for model training, while online stores facilitate real-time data access for model inference, crucial for applications like personalized recommendations and fraud detection. Tinybird emerges as an innovative serverless solution for online feature stores, offering real-time data processing, a SQL-based development environment, and instant API access, thereby reducing infrastructure overhead and enabling rapid deployment of machine learning features. By integrating with platforms like Databricks and Confluent, Tinybird enables companies across various industries, such as e-commerce, hospitality, and sports betting, to deliver personalized user experiences, optimize recommendations, and manage real-time data flows efficiently.
Apr 24, 2025
3,829 words in the original blog post.
Real-time data systems, which often handle immense data volumes and serve numerous concurrent users with low-latency requirements, rely heavily on load testing to ensure performance stability during traffic surges. Tinybird’s experience with load testing highlights its importance in predicting system behavior, ensuring Service Level Objective (SLO) compliance, and preventing downtime during peak usage periods. Load testing evaluates infrastructure response under increasing traffic, focusing on metrics like response times and stability. It becomes crucial in scenarios such as traffic surges from marketing campaigns or significant events, the introduction of new endpoints, and validation of existing infrastructure. Proper load test planning involves defining objectives, selecting representative API calls, and understanding the distribution and type of queries to simulate real-world conditions. Key variables include queries per second, request latency, and processed data volume. By examining the performance during load tests using metrics like the 99th percentile latency, systems can be optimized through strategies like query optimization and infrastructure scaling. Conducting iterative tests helps refine these optimizations, ensuring systems can handle increased loads efficiently without compromising user experience.
Apr 24, 2025
3,081 words in the original blog post.
In exploring efficient methods for counting post views and unique viewers, the text contrasts Reddit's 2017 complex system using Kafka, Redis, and Cassandra with a simpler solution implemented using Tinybird. Reddit's approach involved multiple components, including HyperLogLog for space-efficient approximations, while the author proposes a streamlined method that stores raw events in a single location, counts uniques with a single SQL query, and delivers real-time results with high efficiency. This alternative approach is demonstrated through a Tinybird project, which provides a REST API for counting unique post viewers and emphasizes ease of deployment and scalability, evidenced by tests handling large volumes of data with minimal latency. The author acknowledges potential trade-offs, such as increased raw data storage and the necessity for query optimization at extreme scales but highlights the absence of complex data pipelines and distributed system challenges. The text invites readers to try the solution themselves and teases further discussion on scaling to even larger data volumes in future posts.
Apr 24, 2025
820 words in the original blog post.
Real-time dashboards are increasingly valuable for operational decision-making and customer experiences, offering immediate insights and allowing for rapid response to data as it arrives. Despite claims that traditional dashboards are becoming obsolete with the rise of AI-powered analytics, they remain essential, especially as data quality improves. The primary challenges with dashboards are often due to slow data pipelines and outdated data infrastructure, which are not designed for real-time processing. Real-time dashboards, which display data that is almost instantly updated, help organizations make quick, informed decisions across various sectors such as retail, healthcare, and finance, providing a competitive advantage. Building effective real-time dashboards involves focusing on clarity and usability, and using appropriate tools for real-time data collection, processing, storage, and visualization. By leveraging real-time data platforms and modern visualization tools, organizations can create dashboards that not only support internal operations but also enhance customer interactions, ultimately driving top-line growth and customer satisfaction.
Apr 24, 2025
3,422 words in the original blog post.
Anomaly detection, also known as outlier detection, is a crucial technique for identifying data points that significantly deviate from expected patterns across various domains. This practice has evolved from traditional, static dataset analysis to real-time applications, driven by the increasing volume and velocity of data. Real-time anomaly detection is essential for systems handling continuous data streams, such as credit card fraud detection, IoT monitoring, and cybersecurity, with techniques ranging from simple out-of-range checks to complex machine learning algorithms. The field is divided into supervised and unsupervised methods, each with distinct advantages; supervised methods rely on labeled data for training, while unsupervised methods identify anomalies in real-time without predefined labels. Recent advancements in real-time databases, like Apache Druid and ClickHouse®, enable high-speed analytical queries, facilitating the implementation of real-time anomaly detection systems. Examples include SQL-based algorithms for out-of-range, timeout, rate-of-change, interquartile range (IQR), and Z-score anomaly detection, which help detect anomalies nearly instantaneously while adapting to changing data trends.
Apr 24, 2025
1,953 words in the original blog post.
Guilherme Oenning, a software engineer with over 15 years of experience, founded Aptabase, a privacy-focused telemetry and analytics service for native app developers, after realizing the lack of open-source, privacy-first analytics tools for mobile and desktop apps. In building Aptabase, Guilherme chose Tinybird for its serverless ClickHouse® capabilities, which provided a scalable and easy-to-deploy analytics solution without the complexities of managing database infrastructure. Tinybird's platform, with features like Pipes and Materialized Views, allowed Guilherme to rapidly develop and deploy Aptabase while maintaining focus on product development. The open-source project offers SDKs for various programming languages, enabling developers to track app usage and enhance user privacy. Looking forward, Aptabase aims to expand its SDK offerings and introduce customizable dashboards, with strong community support playing a key role in its development. Tinybird has been instrumental in Aptabase's growth, offering a supportive community and technical capabilities that simplify data management and accelerate product development.
Apr 24, 2025
1,873 words in the original blog post.
Python is a favored language for data analysis and visualization due to its powerful libraries like numpy, pandas, matplotlib, plotly, and seaborn. The tutorial explores building a real-time dashboard in Python using Tinybird, a serverless real-time data analytics platform, and Dash, an open-source app-building framework. Tinybird captures and transforms streaming data using SQL, publishing transformations as REST APIs, while Dash provides visualization capabilities. The integration uses the Verdin Tinybird SDK for Python to visualize data in real time, incorporating interactive filtering through Dash's input components and Tinybird query parameters. The tutorial covers setting up a real-time dashboard that includes charts like a time series line chart, donut chart, and bar charts, emphasizing the simplicity and scalability of Tinybird's API model for handling large-scale data efficiently. It highlights Tinybird's advantages over other libraries, such as its ability to easily ingest data from various sources, offer scalability without being limited by RAM, and provide real-time data pipelines as REST APIs.
Apr 24, 2025
3,164 words in the original blog post.
ClickHouse® has introduced a new Database engine called Backup in its latest version 25.2, designed to streamline the process of working with databases in read-only mode directly from backups without performance issues. This engine supports both incremental and non-incremental backups, allowing users to attach tables and databases instantly to ClickHouse® and work with them as usual tables. The Backup engine addresses challenges encountered during client upgrades to new ClickHouse® versions, where performance issues and edge cases often require testing with real production data. The implementation includes components like BackupDisk and DatabaseBackup, which handle path translation and adapt the backup interface to the disk interface, ensuring seamless integration and operation. The engine is user-friendly, compatible with various backup engines like S3, Azure, File, or Disk, and can work with complex use cases such as validating upgrades, building time machine functionality, sharing datasets, and running ad-hoc queries. The future development plans for this engine include potential write capabilities and enhancements for multi-machine setups, with Tinybird planning to leverage this functionality for their upcoming "self-managed Tinybird" service.
Apr 24, 2025
1,280 words in the original blog post.
AWS tools like Kinesis, S3, Glue, and Athena are often combined to create data analytics architectures aimed at business intelligence use cases, capable of processing and analyzing large volumes of data. However, they fall short for real-time analytics due to limitations in concurrency, latency, and data freshness. Kinesis collects and processes data streams, S3 provides scalable storage, Glue transforms and enriches data, and Athena allows for SQL-based querying of data stored in S3. This architecture is ideal for strategic reporting where immediacy is not crucial, but it struggles with real-time operational needs. Tinybird offers a solution for real-time analytics with its platform that ingests, processes, and serves data with low latency, making it suitable for operational applications demanding quick responses. By integrating Tinybird with AWS tools, businesses can address both strategic and operational data needs, balancing long-term decision-making with immediate, data-driven actions.
Apr 24, 2025
1,653 words in the original blog post.
Revamping Tinybird's documentation was a comprehensive process driven by the need for a more dynamic, personalized, and scalable platform. The previous Sphinx-based system was deemed inadequate for future needs, lacking features like easy personalization, integration with Auth0, and a smooth workflow for the product and engineering teams. The team, led by Julia and Lucy Mitchell, opted to build a new system from scratch using Next.js for its flexibility and familiarity, supplemented by Markdoc for documentation, and hosted on Vercel with Tailwind CSS for styling. Key features such as personalized user experiences, dynamic code snippets, and streamlined search capabilities were prioritized, while maintaining accessibility and preparing for future enhancements like AI-powered help and advanced image optimization. The new documentation infrastructure is hosted in a dedicated repository to facilitate open-source collaboration and ensure seamless scalability, reflecting a strategic shift towards a more user-friendly and integrated documentation experience.
Apr 24, 2025
2,782 words in the original blog post.
The Tinybird DynamoDB Connector, now available in public beta, facilitates the seamless integration of data from Amazon DynamoDB to Tinybird, enhancing real-time analytics capabilities by overcoming DynamoDB's limitations in handling intense analytical workloads. This connector allows developers to effortlessly sync data without the need for additional code or external ETL processes, using initial backfills and continuous streaming upserts to maintain data synchronization with minimal latency. Security is managed through AWS IAM Roles and Policies, ensuring a secure connection between the systems, while observability is provided out of the box with a logging service for all write operations. Pricing is based on data processing and storage, with costs for write operations and monthly storage depending on data size and schema compression ratios. The DynamoDB Connector also offers a straightforward migration path for former Rockset customers, who can transition their data pipelines to Tinybird's platform with familiar processes and minimal disruption.
Apr 24, 2025
1,143 words in the original blog post.
Tinybird emphasizes the critical importance of data ingestion for its analytics services, offering mechanisms to ensure this process is reliable and observable. The platform now provides email notifications to alert users of data ingestion failures or when data is sent to quarantine, enhancing basic observability. Users can manage these notifications through the Member configuration of the Workspace, with admins receiving alerts by default. Additionally, Tinybird includes a Service Data Source called datasources_ops_log, allowing users to build custom observability functions and expand alerts into various platforms such as Slack or Datadog. This feature is designed to integrate seamlessly with existing observability services and encourages users to provide feedback or seek assistance through Slack for further enhancement of these capabilities.
Apr 24, 2025
315 words in the original blog post.
Real-time recommendation systems have become essential in enhancing online user experiences by providing personalized content suggestions based on immediate user interactions and historical data. These systems leverage intelligent algorithms, often incorporating machine learning, to deliver low-latency recommendations that adapt to changing user behavior, thereby surpassing traditional batch recommendation methods. By integrating real-time data platforms like Tinybird, which allow for the construction of recommendation engines using SQL, developers can create systems that analyze streaming data in conjunction with data stored in warehouses such as Snowflake or BigQuery. Real-time recommendation systems utilize both content-based and collaborative filtering to tailor suggestions, drawing on user preferences and similar user behaviors. Prominent examples include Netflix, TikTok, Twitter, Spotify, and Amazon, each utilizing these systems to boost user engagement and drive revenue. The development of such systems involves key steps like data collection, storage, preprocessing, algorithm development, and real-time implementation, with a focus on optimizing recommendations through continuous feedback and adaptation. As these systems evolve, they promise to significantly enhance digital interactions by making them more relevant and engaging.
Apr 24, 2025
2,603 words in the original blog post.
Last year, the Tinybird team faced challenges with their continuous integration (CI) pipeline, which had become unreliable and inefficient due to the rapid growth in their team, codebase, and test suite. To address these issues, they paused all production changes for a week to focus on fixing the CI system. They initially implemented parallelization and auto-scaling techniques to improve CI performance but soon realized these changes did not address underlying problems. By utilizing their own product, a real-time analytics platform, they collected and analyzed CI telemetry data, leading to insights that helped optimize test execution order, redistribute tests across workers, and tailor infrastructure resources to specific job requirements. These optimizations resulted in a 60% reduction in CI pipeline execution time and the elimination of random job failures. The process underscored the importance of data-driven decision-making, observability, and collaboration in engineering. Despite these improvements, the team continues to seek further optimizations and plans to share their solutions with the wider development community.
Apr 24, 2025
2,731 words in the original blog post.
The blog post describes how the author, initially surprised by Tinybird's decision not to use Google Analytics due to privacy concerns and GDPR rulings, embarked on creating a custom analytics system using Tinybird and Retool. Tinybird employs its own analytics infrastructure, using first-party cookies for enhanced data control and privacy adherence. The author, leveraging their technical skills in SQL and Python, recreated key Google Analytics widgets through a series of steps that involved defining sessions from web events, creating materialized views for efficient data processing, and using Tinybird Pipes to turn SQL queries into API endpoints. These endpoints were then visualized in Retool, an internal team data visualization platform, allowing the author to replicate Google Analytics functionalities such as metric dashboards and interactive charts. This process not only demonstrated the capability of Tinybird and Retool as effective tools for web analytics but also illustrated the ease of implementing custom data solutions that respect user privacy.
Apr 24, 2025
2,301 words in the original blog post.
Over the past eight months, the frontend team at Tinybird has modernized its code management and continuous integration (CI) process by implementing a monorepo structure alongside Turborepo and pnpm, significantly reducing execution times despite a large increase in code size and test coverage. Tinybird, a serverless real-time database platform, has improved its developer experience by consolidating its previously fragmented codebase into a single repository, which eliminated duplicated code and streamlined task management. Turborepo has enhanced their CI efficiency through intelligent task handling and caching, while pnpm has accelerated package management with its superior dependency resolution. These changes have allowed Tinybird to ship code faster and with greater confidence, cutting down CI times by about five hours each week, even as they continue to increase their lines of code and unit tests. The team plans to continue improving by fully migrating to TypeScript, splitting frontend from backend repositories, and transitioning from React to Next.js, further fostering development speed and feedback loops.
Apr 24, 2025
2,189 words in the original blog post.
In July 2021, catastrophic floods in central Germany highlighted the critical importance of effective flood-warning systems (FWSs) after technological failures in early warning systems resulted in over 150 fatalities, despite accurate forecasts. The blog post discusses how public APIs, which are vital components of FWSs, must be robust and resilient, featuring real-time data processing, scalability, and security. It emphasizes that designing such APIs involves mastering various components, including API design, programming language selection, data storage management, load balancing, performance monitoring, and security measures. The platform Tinybird is introduced as a solution that simplifies API development by handling complex backend tasks, offering features like real-time data ingestion, SQL-based data querying, and automatic scaling, thus allowing developers to focus on design and optimization. This enables the creation of scalable and resilient APIs for various applications, including flood warning systems, by reducing the burden of infrastructure management and ensuring high performance and reliability.
Apr 24, 2025
2,469 words in the original blog post.
A soccer enthusiast and content marketer at Tinybird delves into the intersection of soccer, data analytics, and Twitter, aiming to measure the emotional responses of fans during the World Cup by analyzing tweets. Inspired by a colleague's project, he utilized the Twitter API and the TextBlob Python library for sentiment analysis but ultimately found a more straightforward method by tracking the use of country flag emojis in tweets. This approach allowed the creation of a real-time sentiment analysis tool that captured the collective emotions of fans by counting flag emojis during matches, with data processed through Tinybird's platform and visualized using Retool. The project not only highlighted the emotional highs and lows of the World Cup but also included a unique feature tracking the length of "GOAL" tweets, reflecting the intensity of reactions to match events.
Apr 24, 2025
2,114 words in the original blog post.
In today's financially constrained tech environment, data and engineering teams are increasingly focused on creating revenue-generating data products by prioritizing real-time data platforms. These platforms are essential for developing real-time data pipelines that support user-facing applications and use cases by centralizing core components like streaming data ingestion, pipeline development, and real-time access. They enable the integration of streaming and batch data sources, providing low-latency access to transformed and enriched data for downstream applications. By leveraging technologies such as real-time databases and analytics engines, these platforms simplify tool deployment, reduce time to market, and allow engineers to concentrate on developing use cases rather than maintaining complex infrastructures. Real-time data platforms, such as Tinybird, offer managed solutions that integrate data ingestion, storage, processing, and access, allowing teams to efficiently build and scale their real-time data strategies without extensive resource investment.
Apr 24, 2025
2,054 words in the original blog post.
Tinybird employs a modified version of Amazon's "working backwards" approach, focusing on customer experience before technical details, to quickly develop data projects using a BP/FAQ (Blog Post/Frequently Asked Questions) method. This approach encourages building a "minimally viable endpoint" first and optimizing it later, prioritizing rapid prototyping over initial technical perfection. Tinybird simplifies data project creation by allowing users to ingest data into Data Sources, transform it in Pipes using SQL, and publish APIs in a streamlined manner. The process emphasizes starting with a prototype to validate use cases, with later iterations focusing on performance improvements through schema refactoring, indexing, and the use of Materialized Views. This methodology aligns with Tinybird's commitment to speed, both in endpoint performance and in the development process, promoting agility and efficiency in building data-driven applications.
Apr 24, 2025
1,533 words in the original blog post.
Tinybird has enhanced its platform to allow users to append CSV data to an existing Data Source, even if the column order in the CSV differs from the original Data Source, as long as the CSV header includes the correct column names. This improvement addresses the issue users faced when inconsistent column orders led to data being quarantined. Users can now append data through the UI, API, or CLI without concern for column order, enabling more flexible data integration. This update is exemplified by appending data to a Data Source of movie likes and dislikes, demonstrating that column order variability will not affect data appending as long as the headers are correct.
Apr 24, 2025
307 words in the original blog post.
As applications grow, managing increased user concurrency becomes a challenge, especially with Postgres, which wasn't designed for thousands of simultaneous connections. While vertical scaling aids data volume, connection management requires a different strategy. Tinybird offers a solution by allowing Postgres event tables to be imported, streamed, or connected via Kafka, using SQL to define real-time API endpoints with low latency. Postgres uses a process-based model for connections, leading to limitations like resource use, connection limits, and context switching. To manage more users, key strategies include reducing connection duration through query optimization, caching, and tuning configurations; increasing connection capacity via connection pooling with PgBouncer and read replicas; and decreasing connection demand with asynchronous processing, connection sharing, and application-level caching. Despite these optimizations, signs like connection timeouts or high CPU usage may indicate the need for more advanced solutions such as vertical scaling, sharding, or specialized databases. The text offers a comprehensive guide to scaling Postgres effectively, highlighting the importance of starting with basic optimizations before considering more complex solutions.
Apr 24, 2025
3,331 words in the original blog post.
Tinybird is a serverless, real-time data platform that allows users to query large datasets from Google Sheets using SQL, turning them into secure, interoperable APIs. This transforms Google Sheets into a real-time analytics hub, enabling users to perform complex queries, integrate with multiple data sources, and create advanced dashboards and automated workflows. Tinybird's architecture supports scalability, handling billions of rows quickly, and offers robust security protocols, ensuring data safety. Users can enrich Google Sheets data with external sources like CRMs or other APIs, enhancing analytics capabilities. With Tinybird, users can publish real-time metrics as API endpoints, facilitating integration with various visualization tools and other applications. The platform also features a time-unlimited free tier, allowing users to explore its capabilities without financial constraints.
Apr 24, 2025
1,764 words in the original blog post.
Tinybird, an analytics backend for software applications, is increasingly being used by developers to monitor Large Language Model (LLM) usage, costs, and performance as AI features become more prevalent. The company has open-sourced the LLM Performance Tracker app template, which includes both a frontend and backend capable of capturing and analyzing LLM calls in real time. A notable feature of this template is the AI Cost Calculator, which allows users to visualize LLM costs by passing user input through an API that generates structured parameters for the Tinybird data API. The system is designed for high scalability and speed, handling millions of LLM call logs efficiently. The app leverages various components like a Tinybird data source and pipe, React components, and an API route to generate structured parameters from user input, ultimately creating dynamic, user-generated data visualizations. While LLMs are utilized to convert free-text inputs into structured data, certain tasks like determining chart types and generating SQL queries are handled without LLMs to maintain performance, security, and observability. The template illustrates how a combination of LLMs and static APIs can facilitate real-time data visualization while ensuring data security and efficiency.
Apr 17, 2025
1,515 words in the original blog post.
Inbox Zero is an open-source AI personal assistant designed to help users efficiently achieve inbox zero by automating email management tasks. The platform leverages Tinybird to power its analytics, providing both internal platform insights and user-facing dashboards with real-time data. This dual-purpose system uses the aiCall data source to track AI interactions, enabling users to monitor their email processing activities and API usage in detail. Tinybird's architecture supports real-time processing and scalability, ensuring performance even as the user base grows. This setup allows Inbox Zero to offer transparency in both code and analytics, making it a useful tool for users to manage their email workflows and API costs effectively. The use of Tinybird's Docker container further enhances the platform's open-source nature, aligning with Inbox Zero's commitment to transparency and resource optimization.
Apr 04, 2025
664 words in the original blog post.