January 2025 Summaries

The Time Series Buying Guide for IIoT is a resource that helps businesses navigate the challenges of managing Industrial Internet of Things (IIoT) data. Leveraging IIoT data can significantly improve operational and business efficiency by providing actionable insights into machine performance, usage patterns, and environmental conditions. However, achieving these benefits requires tools that preserve and enhance the quality of datasets, ensuring they remain accurate and complete. Managing time series data poses unique challenges due to its scale, speed, and complexity, including massive scale, real-time action, and high cardinality. High-fidelity data is crucial for businesses to detect issues early, build accurate analytics, and create reliable models, but failing to address these challenges can result in incomplete datasets that lack precision. The guide offers practical advice on identifying the right tools and architectural practices necessary to achieve high-fidelity data sets and seamlessly integrating specialized IIoT tools into existing architectures and environments.

InfluxDB 3 Enterprise is now free for at-home use, offering capabilities previously limited to InfluxDB 3 Core. The new tier provides a rate-limited version of the full enterprise features, allowing users to write and query data across longer periods. This change addresses concerns from open-source users who require these capabilities. InfluxDB 3 Enterprise includes a compactor that rewrites files into larger blocks of time, enabling faster query response times on any time range spanning longer than an hour. The product offers a compelling set of features for real-time and recent data storage and querying, making it suitable as a data collector, ETL tool, or full historical time series database. This change is part of InfluxDB's business model, allowing the company to continue developing open-source projects like InfluxDB 3 Core while contributing to other initiatives in Apache DataFusion and related technologies.

The TIG Stack is a powerful solution for real-time analytics, observability, and monitoring. It consists of Telegraf for data collection, InfluxDB for storage and analytics, and Grafana for visualization. InfluxDB Core is the latest version of InfluxDB, introducing improvements in performance, usability, and cost-efficiency. To get started with the TIG Stack, you'll need to install Telegraf, InfluxDB Core, and Grafana, then follow a series of steps to write data to InfluxDB using Telegraf, visualize your data with Grafana, and finally configure the Grafana source to connect to InfluxDB Core. The process involves creating a configuration file for Telegraf, starting the Telegraf service, verifying successful writes with the InfluxDB CLI, and configuring the Grafana source to connect to InfluxDB Core.

InfluxDB 3 queries data in real-time by leveraging the FDAP stack, which includes Flight, DataFusion, Arrow, and Parquet technologies. The database's high-performance analytics are supported by a custom ingester, compactor, catalog, caching, and file organization that work together to deliver speed and efficiency. Real-time analytics for time series data involves analyzing and extracting insights from time-stamped data immediately after it is ingested, with minimal latency. InfluxDB 3's real-time analytics rely heavily on its custom ingester, which prioritizes speed over durability and ACID compliance. The database also uses advanced caching techniques, flexible Parquet files, and optimized data organization to ensure fast query performance and efficient storage. By combining the power of open source tools with bespoke optimizations, InfluxDB 3 delivers high-performance capabilities for real-time analytics and time series workloads.

The text discusses the use of Python with InfluxDB v3 Core, an open-source time series database. To start using it, one needs to create a token and write data to it using the `influxdb3` CLI or cURL requests. The Python Client Library is then used to interact with the database. It supports writing several types of data objects, including Pandas DataFrames. The library can be used to query the data as well. The text also provides instructions on how to install and use InfluxDB v3 Core, including creating a token, serving the instance, and managing databases. Additionally, it discusses the importance of using the `influxdb3` CLI or Client Library for future updates.

Today, InfluxDB 3 Core and Enterprise are publicly available in alpha, offering a new paradigm for time series data management with exceptional performance for real-time monitoring, data collection, and streaming analytics. The platform is designed to handle high-speed, high-cardinality data from the edge to the cloud, supporting critical use cases in observability, IoT/IIoT, and real-time monitoring and control. InfluxDB 3 delivers products capable of handling massive volumes of time series data with orders-of-magnitude improvements in performance and compression, while also addressing seismic shifts in how data is collected, transformed, and acted upon. The platform is built on the FDAP stack, embracing open source wherever possible to build around a larger community and accelerate platform maturity.

InfluxDB 3 Core, a recent-data engine for time series and event data, is now available for alpha testing as an open source product under the MIT/Apache 2 License. InfluxDB 3 Enterprise adds historical query capability, read replicas, high availability, scalability, and fine-grained security to Core's foundation. The products are built on more than four years of development using the FDAP stack (Apache Flight, DataFusion, Arrow, and Parquet) and deliver a powerful SQL query engine with unlimited cardinality, native object storage support, and fast recent data processing capabilities. InfluxDB 3 Core operates in "diskless" mode, using object storage as the only persistence layer, enabling more dynamic operating environments and seamless access to third-party systems. The product includes features such as fast recent data processing, last value cache, distinct value cache, and a plugin system with embedded Python for plugins and triggers, which enables custom data collection, real-time monitoring, integration with third-party services, scheduled task execution, downsampling, aggregation, and HTTP endpoint creation. InfluxDB 3 Enterprise adds high availability configuration, read replicas, enhanced security features, historical data compaction, row-level delete support, and an integrated admin UI. The products are designed for operational simplicity and compatibility with previous InfluxDB versions, while the FDAP stack enables a state-of-the-art, high-performance SQL engine with parser, planner, optimizer, and vectorized execution.

The article provides a comprehensive guide on building a modern, intelligent DevOps monitoring pipeline using open-source technologies. The pipeline consists of four key components: real-time data collection with Telegraf, time series storage with InfluxDB Cloud, pattern recognition and anomaly detection with Qdrant vector database, and visualization and alerting with Grafana. By integrating these components, teams can identify and fix issues before users notice them, ensuring faster response times and improved overall system reliability. The guide is designed to be hands-on, providing sample Python code and examples for readers to experiment with the system in action.

The year 2025 promises to be exciting for Apache DataFusion, with the project reaching an inflection point and accelerating in adoption as more systems are built on top of it. The journey from 0 to 1,000 projects is expected to be achieved, driven by significant investment from the community, major companies, and database products. DataFusion has matured beyond early adopters and is now a viable choice for anyone building highly performant analytic systems. Its adoption is set to surge as the industry embraces Open Data Lake architectures, with efforts focusing on simplifying adoption for downstream users, pushing the limits of performance, and reducing friction for users when adopting new versions of DataFusion. The community aims to balance innovation with stability while maintaining a rapid velocity of improvements, leading to a thriving ecosystem.

InfluxData, the creator of the leading time series platform InfluxDB, has appointed Pat Walsh as its new Chief Marketing Officer. Walsh brings extensive expertise in open source innovation, product management, and go-to-market execution to his role, having previously held leadership positions at Privitar, Tufin, Core Security, and Talend. At InfluxData, Walsh will spearhead the rollout of InfluxDB 3, a major leap forward in time series database technology that sets a new standard for performance. With InfluxDB 3, developers can power demanding workloads with speed and precision, driving significant cost savings. Walsh joins InfluxData at a pivotal moment, redefining what's possible with time series data, and is excited to unlock the impact of InfluxDB 3 across the market.

Apache DataFusion Meetup in Chicago was attended by around 25 enthusiasts who learned from project contributors and discussed ideas for the future, showcasing increasing adoption of the technology. The event featured four talks on various topics, including building a real-time data lake with DataFusion, practical data science in robotics using DataFusion, a disaggregated cache for DataFusion, and building InfluxDB 3.0 with the FDAP stack. The meetup also provided opportunities for attendees to meet others from companies like Pydantic, Relativity Software, and Embucket who are working on projects using DataFusion, fostering a sense of community and discussing topics such as project growth, secondary indexes, performance, and the future roadmap.

The article compares and contrasts SQL and InfluxQL as query languages for InfluxDB v3. InfluxDB v3 supports both SQL and InfluxQL, allowing users to choose the best tool for their specific needs. SQL is a widely-used language that offers flexibility and interoperability with existing analytics workflows, while InfluxQL is a domain-specific language optimized for time series data. The article highlights the key terminological differences between SQL and InfluxQL, including the concepts of databases, tables, columns, buckets, measurements, fields, and tags. It also explains the basic syntactic differences between the two languages, such as interval types, datetime formats, and aggregation functions. Additionally, the article discusses the user experience and query tools available for each language, including client libraries, HTTP APIs, and visualization tool support. The advantages of InfluxQL include its suitability for schema exploration and its use in data processing projects, while the advantages of SQL include its wider adoption, increased function support, and enhanced interoperability with other tools.

Verint, a customer experience automation platform, was facing scalability challenges with its previous database due to rapidly growing data volumes and the need for long-term analytical capabilities. The company evaluated several time series databases, including OpenTSDB, Amazon Timestream, and TimescaleDB, but ultimately chose InfluxDB Clustered for its on-prem deployment flexibility, maintainability, and performance benefits. Verint selected InfluxDB for its ability to store data in durable object storage, enabling robust backup capabilities and reducing latency. The company is currently ingesting hundreds of millions of data points per month into InfluxDB and plans to scale further to billions of data points per month in 2025, with future plans including integration with Apache Superset for dashboard creation and visualization.