September 2023 Summaries

InfluxDB and Kafka are complementary tools that enable companies like Hulu to handle high volumes and velocities of time series data by providing real-time queries and analytics with integration to cutting-edge machine learning and artificial intelligence technologies. A data streaming pipeline can be built using these tools, which includes sending data from a Kafka topic to InfluxDB Cloud Serverless using Telegraf, an open-source plugin-based tool. The process involves creating a Dockerfile, generating sample data, building the code file, setting up the InfluxDB Cloud Serverless account, and configuring mytelegraf.conf with necessary settings such as API tokens, URLs, organization, bucket, and Kafka brokers. After spinning up the containers using `docker compose up --build`, Telegraf displays messages indicating that it's reading and writing data to InfluxDB Cloud Serverless correctly. The Data Explorer can be used to query the data, making it simple to connect a Kafka cluster to InfluxDB Cloud Serverless for time series data management.

InfluxDB 3.0 has been released, providing 45x better write throughput and 5-25x faster queries compared to previous versions. One of the deprioritized features is the task engine, which led to the development of Mage.ai, an open-source data pipeline tool for transforming and integrating data. Mage is designed to be easy to use, with a customizable UX, templates for sharing pipelines, and AI-powered pipeline generation. It uses Polars and Parquet under the hood and has excellent community support. To run Mage, users need to deploy and manage it themselves, but the documentation provides detailed instructions for deployment on various cloud platforms. The tutorial demonstrates how to create a materialized view of time series data in InfluxDB Cloud v3 using Mage.ai, including creating triggers to schedule pipeline runs.

The commercial version of InfluxDB 3.0 is a distributed, scalable time series database designed for real-time analytic workloads, supporting infinite cardinality, SQL and InfluxQL as native query languages, and managing data efficiently in object storage using Apache Parquet files. The open source version, called InfluxDB Edge, will be released after the commercial version and will serve as a standalone process optimized for providing a queryable, real-time buffer of time series and observational data. InfluxDB Edge will support the InfluxDB 1.x and 2.x write APIs, Line Protocol, InfluxQL query API, and new APIs specifically built for 3.0, including FlightSQL for SQL queries. The open source InfluxDB Edge will be developed in the existing InfluxDB repo under a permissive MIT or Apache2 license, while a free community edition named InfluxDB Community with additional features not available in Edge will also be released. InfluxDB Edge is designed to fill a different spot in the toolbelt and infrastructure of companies working with time series data at scale, providing a powerful agent for collecting, processing, and monitoring data. The commercial version of InfluxDB 3.0 will provide all the functionality of Edge and Community while adding features for high availability and security.

The use of time series data is critical for various industries within the space sector, including satellite communications, Earth observation and remote sensing, and space exploration and navigation. These industries rely on time series data to monitor signal strength and quality, predict signal interference, track environmental changes, plan mission trajectories, and analyze celestial events. The sources of time series data in these industries include satellite sensors and instruments, radar and Lidar sensors, ground-based telescopes and observatories, and other devices that collect data on and monitor objects and events in space. However, managing this data can be challenging due to the high volume of data generated, device malfunctions or space weather events, and the need for sufficient storage and compute power. Time series databases like InfluxDB provide solutions for these challenges by offering features such as edge data replication, durable queues, and upserting late-arriving data.

Apache Arrow is a framework that enables in-memory columnar data representation, aiming to be the language-agnostic standard for columnar memory representation to facilitate interoperability among various processing engines. It was developed by several open-source leaders from companies working on Impala, Spark, and Calcite, including Wes McKinney, creator of Pandas. Arrow provides efficient columnar memory exchange, zero-copy reads, and faster data manipulation due to its column-based format. This standardization enables the transfer of data between systems with little overhead, reduces costly data serialization and deserialization, and supports multi-language development. Additionally, Apache Arrow Flight SQL is a client-server protocol that solves the problem of efficiently transferring in-memory data across networked services. It provides features like encryption, authentication, and parallel data access, making it an attractive solution for companies and projects using data analytics and storage solutions.

InfluxData and Dremio leverage the Apache ecosystem to enhance their product offerings, with InfluxDB 3.0 built on top of Apache Arrow, Parquet, and Flight, enabling efficient data analysis and storage. Both companies benefit from the Apache Software Foundation's open source approach, which promotes rapid innovation, security, and customization. The integration of various Apache technologies, such as Calcite, Iceberg, and Arrow Flight, facilitates efficient data processing and querying in Dremio's advanced data lakehouse platform. This collaborative development model allows developers to contribute different perspectives and skills, leading to more comprehensive and innovative software solutions.

LBBC Technologies has adopted a custom predictive maintenance program using InfluxDB, AWS, and MQTT to improve the observability of their machinery and provide better customer support. The company collects machine data from various sources, including PLCs, and stores it in InfluxDB for real-time ingestion, low-cost storage, and data visualization. To augment InfluxDB's capabilities, AWS Lambda functions are used to transform the data into line protocol format and send it to Servitly for historical data storage. The solution has enabled LBBC to write complex queries, identify specific issues with machinery, and provide data-driven predictive maintenance capabilities to their customers.

The Go InfluxDB v3 Client Library is a software package that allows developers to efficiently query and write time series data from/to InfluxDB 3.0, simplifying the integration of InfluxDB into Go applications. InfluxDB 3.0 offers significant performance improvements over previous versions, including 45x better write throughput and 4.5x better storage compression. The library provides a set of tools and functions for interacting with InfluxDB using the Go programming language, implementing writes via the /write API endpoint and utilizing Apache Arrow Flight Client Libraries and the Arrow Format and Flight gRPC protocol for efficient serialization and deserialization. To use the library, developers need to initialize the client by providing credentials and import the required packages into their main.go file. The library allows users to write data to InfluxDB synchronously or asynchronously using Point objects and supports upserts of fields but not tags. It also enables querying of InfluxDB 3.0 using SQL and InfluxQL, a SQL-like query language for time series data.

InfluxDB Clustered is a self-managed product for large-scale time series workloads that offers high performance, unlimited scale, and significant cost savings compared to its predecessor, InfluxDB Enterprise. It delivers real-time data analysis, native SQL support, and supports unlimited data cardinality, making it suitable for enterprise and compliance requirements. The product runs on Kubernetes-based containers with decoupled, independently scalable ingest and query tiers, providing high availability and scalability. With the separation of compute and storage, InfluxDB Clustered can handle high-speed, high-volume analytics in real-time, including managing high cardinality data without impacting performance. It also introduces multiple storage tiers, including a hot storage tier for immediate data availability, and a cold storage tier for long-term storage at lower costs. The product offers a 90% reduction in storage costs compared to previous versions, making it an attractive option for organizations looking to optimize their time series data analysis.

InfluxDB Clustered is a self-managed time series database for on-premises or private cloud deployments, built on InfluxDB 3.0's rebuilt database engine optimized for real-time analytics with higher performance, unlimited cardinality, and SQL support. It completes InfluxData's commercial product line developed on InfluxDB 3.0, offering customers the scale and flexibility of the cloud with the security and control of a self-managed infrastructure. Deployed natively in Kubernetes, InfluxDB Clustered combines the benefits of the reimagined InfluxDB 3.0 with the specific needs of enterprises deploying their own custom infrastructure. With features such as 100x faster queries, 45x faster data ingest, and a 90% reduction in storage costs, InfluxDB Clustered delivers significant improvements over its predecessor, InfluxDB Enterprise.

The use of time series databases in predictive analytics is particularly suited for anomaly detection and predictive maintenance, as they allow for processing high-speed and volume time-stamped data. Predictive analytics leverages big data, statistical algorithms, and machine learning techniques to anticipate future outcomes based on historical data, with applications across various industries such as finance, healthcare, retail, and marketing. A time series database, like InfluxDB 3.0, provides key functionality for performing predictive analytics by storing, retrieving, and processing time-stamped data at high speed and volume. By combining InfluxDB Cloud, Quix, and Hugging Face, organizations can create a predictive maintenance pipeline that uses anomaly detection and forecasting to stay proactive and informed, paving the way for enhanced efficiency, reduced risks, and improved decision-making. The pipeline utilizes Quix's streaming pipelines for analytics and machine learning, leveraging Keras Autoencoders for anomaly detection and Holt Winters from statsmodels for forecasting, with data written to two separate InfluxDB instances for storage and querying.

Self-hosting a database can be effective for many companies, but it also comes with challenges such as scaling and security risks, particularly when dealing with spiky or unpredictable traffic. Cloud-based self-hosting is another option that allows businesses to scale on demand without the need for in-house expertise. However, this approach still requires significant engineering resources and support from the cloud provider. The skills gap is also a concern, with many professionals needing to adapt to newer database technologies such as columnar databases, search engine databases, graph databases, and time series databases. InfluxDB 3.0 is a purpose-built time series database that offers real-time query responses and native SQL support, while InfluxDB Cloud Dedicated provides a fully managed, single-tenant instance of the database with built-in data durability and scalability. Ultimately, the decision to self-host or choose a managed database depends on an organization's specific needs, traffic flow patterns, engineering skills, and resources.