March 2025 Summaries

Real-time inventory systems play a critical role in preventing stockouts and overstocking, thereby ensuring a positive customer experience. However, creating a real-time inventory API from raw data is complex due to the need for flexible, context-specific data slicing and resource-intensive on-the-fly aggregations. Transactional databases, while often used for maintaining inventory systems, face limitations when managing real-time data at scale, necessitating a more sophisticated approach. Analytical databases, such as Tinybird, offer a solution by using columnar storage and specialized indexing for high-throughput, real-time analytics, though they come with tradeoffs like immutability and costly updates. The lambda architecture emerges as an effective strategy, combining batch and real-time data processing to maintain updated results efficiently, with Tinybird providing a unified platform to streamline this process. By integrating pre-aggregated snapshots with real-time transactional data, Tinybird enables the creation of a dynamic, real-time inventory API that balances historical depth with current data accuracy, allowing businesses to maintain an accurate inventory state seamlessly.

In recent years, Tinybird has increasingly integrated AI features into its platform, transforming it into an AI-native product with the latest release. These enhancements, powered by large language models (LLMs), automate tasks like project creation, test writing, mock data generation, and API iteration. Tinybird uses a dual-backend system: Python for complex AI functionalities and TypeScript for user experience improvements. The company employs Vertex AI and libraries like LiteLLM and Vercel AI SDK to facilitate AI feature implementation. Instrumentation of AI usage is critical for monitoring performance, costs, and user engagement, enabling the optimization of LLM performance and cost across various providers. Tinybird tracks detailed metrics through its Events API, allowing for real-time analysis and adjustments. A web app developed by Tinybird's team for internal use visualizes these metrics, and plans are in place to release it as an open-source tool. This comprehensive approach to AI instrumentation positions Tinybird to effectively navigate the evolving AI landscape by optimizing resource utilization and enhancing performance.

In a previous role at Planet, the author developed a data infrastructure system to monitor and alert for issues in satellite image collection, using an array of technologies including Python scripts, SQL, and dbt. This effort was challenging and required substantial time and resources, as it involved creating a complex data pipeline to transform data into metrics essential for the company's revenue-generating imagery products. Transitioning to Tinybird, the author embraced modern AI-assisted development tools, significantly reducing the time and complexity involved in rebuilding a similar anomaly detection system. By employing prompt engineering techniques and using Tinybird's capabilities, the author was able to create a functional prototype in just one day, contrasting with the extensive effort previously required. This shift highlights the transformative potential of AI tools in streamlining development processes and enhancing efficiency, particularly in building internal tools, although they may not yet be ideal for production-level applications.

Tinybird Forward introduces a new user experience for handling big data requirements in software development, addressing the lack of robust testing frameworks in data engineering by providing tools for generating test data and validating data transformations. The platform simplifies the process of creating realistic test data with the "tb mock" command, which uses an LLM to analyze SQL queries and data schemas to generate synthetic data. It allows users to prompt LLMs for more complex scenarios, storing these prompts for reuse. The "tb test" command offers a framework for testing SQL logic, generating YAML files for tests and updating expected results when logic or data changes. By integrating with CI workflows, Tinybird Forward aims to foster a testing culture in data engineering, making it easier to implement basic tests without significant time investment.