November 2022 Summaries

Amazon re:Invent 2022 keynote by AWS CEO Adam Selipsky highlighted the intersection of technology and physical challenges, emphasizing the importance of bridging the divide between the digital and physical worlds. The underlying factor is the exponential increase in data produced by systems and devices, which requires reliable and scalable solutions like InfluxDB Cloud. Selipsky also emphasized having the right tools for the job, with InfluxDB being a best-in-breed solution for time series data that plays nicely with AWS. Interoperability and data accessibility were also key themes, with AWS creating solutions to give people access to data and manipulate it. The keynote connected these ideas to use cases on the ground, including sustainability, security, Industry 4.0, edge computing, and showcasing companies like Engie, Aporeto, Siemens, and Prescient Devices that use InfluxDB for various applications.

Tracing is a form of observability that provides a view of how different components in a distributed system work together. It involves collecting timing information on the finer details of a request, using spans and child spans to represent subtasks within a trace. However, past versions of InfluxDB had limitations when it came to handling tracing data due to cardinality issues, which could impact query performance. InfluxDB IOx removes these limits by storing data in a more efficient format, allowing for infinite cardinality and unlocking new use cases such as tracing. The new storage engine and query tier enable fast query performance and scalability, making it possible to handle large volumes of tracing data without sacrificing performance.

InfluxDB is an open-source time series database that stores and queries large amounts of data over time, making it ideal for applications that require fast storage and retrieval functionality. In this article, you learned how to use Chart.js in tandem with InfluxDB Cloud to build time series data visualizations. You also learned how to set up a project with Chart.js, create basic charts such as bar and line charts, and visualize the data using a line chart and a scatter plot. Additionally, you discovered how to write data to your bucket using Flux, query the data from InfluxDB Cloud using JavaScript, and fetch the data from the API to display on a webpage. The article also provides resources for further learning about time series analysis methods, forecasts and projections, automations and workflows with InfluxDB, and more.

Apache Parquet is an open-source column-oriented storage format designed to improve performance and reduce costs in data storage and processing. It was developed by Twitter and Cloudera and has become a common interchange format for various projects, making it easier for users to import and export data with minimal disruption to their workflow. Parquet's innovative techniques, such as run-length and dictionary encoding, record shredding and assembly, and rich metadata, provide great performance by reducing the size of data on disk through compression and making reads faster for analytics queries. The format has been adopted by several companies and projects, including Hadoop, Apache Iceberg, Delta Lake, Apache Spark, and InfluxDB's IOx, which rely on Parquet to optimize their architecture and reduce storage costs, especially in the age of cloud computing where data processing and scanning are charged based on usage.

InfluxDB is a popular time-series database that allows developers to easily store and analyze large amounts of JSON data. The InfluxDB Client libraries provide a powerful method for sending, querying, and managing InfluxDB, giving developers the tools they need to quickly send read and transformed data into InfluxDB. The client libraries are available in many languages, including Python, JavaScript, Go, C#, Java, and others, and can be used to translate JSON data into line protocol, which is a text-based format used by InfluxDB. The libraries use Point helpers to generate valid line protocol data points, making it easy for developers to send their data to InfluxDB. By using the client libraries, developers can easily get their JSON data into InfluxDB and start analyzing it.

The author of this article, Wojciech Kocjan, aimed to improve his setup for monitoring power outages and connectivity issues that were causing his laptop battery to drain. He used InfluxDB, an open-source time-series database, along with Telegraf, a data collection agent, to store data locally on his laptop when connected to the internet but replicate it to a central location whenever possible. The author created a local instance of InfluxDB 2.0 and configured Edge Data Replication (EDR) to automatically send data from the local bucket to an instance of InfluxDB Cloud. He also added networking data, such as connectivity checks for his router and ISP, and power data, including battery state and charging information. By gathering this data, the author was able to identify the root cause of his issues, which were related to power outages affecting his switch and internet connection. With these insights, he improved his setup by running the local switch and printer behind an uninterrupted power supply (UPS), resolving most of the issues and making his friend's life easier. The author also learned how to set up EDR and monitor Windows devices using WMI.

** Flux, a full-fledged scripting language, has become the recommended query language for InfluxDB, offering advanced operations at the database level, but SQL and InfluxQL are also viable options depending on user needs and experience.

Partitioning can provide several benefits to a sharding system, including faster query execution, by splitting data into non-overlapping partitions that can be read and processed in parallel, reducing unnecessary reads and improving data retention policies. Partitioning effects include data pruning, intra-node parallelism, and fast deletion, allowing for efficient management of hot and cold data, with hot partitions containing recent data and cold partitions containing older data. Storing and managing partitions optimizes query workloads by minimizing the number of files in a partition, reducing I/O operations while reading data, and improving query execution speed, while optimizing ingest workloads through low latency ingestion and compaction processes that keep the accumulation of small files to a minimum. Hierarchical partitioning provides answers to questions about quantifying hot, less hot, cold, and more cold data, and can be relaxed by defining an active partition, reducing the number of partitions in the system, making it easier to manage, and reducing the number of partitions that need to be read when querying larger and older chunks.

InfluxDB has been benchmarked against MongoDB for time series workloads, revealing that it outperforms MongoDB in data ingestion by 1.9x, on-disk storage requirements by 7.3x, and query performance by 5x. In contrast, configuring MongoDB to handle time series data requires significant upfront decisions and planning, whereas InfluxDB is designed for time series workloads from the ground up with no additional configuration needed. The benchmarking tests demonstrate that InfluxDB is a clear winner for time series data ingestion, compression, and query performance, making it an ideal choice for developers and architects looking to manage metrics and analytics.

The OPC UA Client Listener Plugin is a new input plugin for Telegraf that offers improved performance and data integrity compared to the existing OPC UA Client Reader plugin. The plugin uses subscriptions instead of polling, which allows it to monitor nodes/tags for changes and notify the client of these changes, greatly improving network performance and guaranteeing the integrity of data. This plugin is highly recommended for most use cases but can be used with polling in certain scenarios. A working example config is provided, demonstrating how to set up and subscribe to a robot state using the OPC UA Client Listener Plugin.

In distributed databases, scaling throughput and performance are critical design topics that require careful consideration. Sharding can make scaling elastic or not, depending on how it is implemented. A database system should be designed to support elastic scaling of both query and ingest workloads, with the ability to scale throughput up and down easily based on traffic or other needs. Elastic scaling allows for flexibility in handling varying workloads, but may have limitations in terms of performance scaling. The key to achieving true elasticity is to split data into smaller shards, allowing for more flexible scaling. However, this approach also introduces additional complexity and overhead. Successful sharding requires careful consideration of the trade-offs between throughput, performance, and scalability, as well as the specific use case requirements.

In a bid to improve their incident management process, PagerDuty adopted a best-in-breed approach by utilizing InfluxDB for monitoring and Telegraf to collect data from various systems. This setup allows for high-volume time series data processing, intelligent alert dispatching, and automated incident resolution through runbooks. The integration of InfluxDB with Telegraf and PagerDuty enables real-time visualization, automation, and refinement of the application logic, ultimately reducing mean time to resolution and improving overall efficiency. By leveraging open-source and proprietary tools, PagerDuty was able to create a flexible solution that meets the demands of growing data volumes, various stakeholders, and infrastructure complexities.

InfluxDB uses scripts in the form of Flux code to help developers streamline and democratize application development, scale time-series workloads, and build lean integrations. Scripts can be used as a learning tool for development teams, allowing experts to share their knowledge with others. Scripts can also be used in conjunction with tasks to automate queries on specific schedules, making it easier to manage tasks and devices at scale. The ability to pass dynamic parameters into scripts streamlines the development and scaling processes, enabling developers to perform complex operations and reuse them across their entire solution. InfluxDB's scripting capabilities also enable smarter alerting based on time-series data, allowing developers to automate alerts and eliminate duplicate alerts. Additionally, scripts can be installed as invokable scripts, enabling external services or applications to access time-series data without the need for additional libraries or components. Overall, using scripts with a time-series database like InfluxDB helps developers build more efficient and effective solutions.

InfluxDB, an open-source time series database, offers various client libraries to facilitate easy integration into different programming languages and data analysis tools. The Python client library is particularly popular among developers, allowing users to leverage InfluxDB's real-time monitoring and alerting capabilities in their data science applications. By utilizing the pandas DataFrames format, users can efficiently query and manipulate data from InfluxDB, while also writing data back into the database with ease. This enables seamless incorporation of InfluxDB into Python-based projects, capitalizing on its strengths as a time series database.

InfluxDB is being used by telecommunications companies such as Cisco and Vonage to improve reliability, efficiency, and productivity. These companies are facing challenges in maintaining legacy infrastructure while modernizing, dealing with increased data generation from more devices, and supporting legacy hardware. InfluxDB helps solve these problems by providing a specialized tool for storing and analyzing time series data, enabling features such as real-time monitoring, automation, and security. With its ability to store and query data efficiently, InfluxDB enables companies like Cisco to monitor their internal SaaS solution and provide reliable service with a 99.999% uptime guarantee. Additionally, InfluxDB provides native integrations for ingesting data, automating tasks, and improving cybersecurity by tracking events and detecting anomalies.

Fluentd is an open-source data collector that can retrieve and receive event data from various sources, filter, buffer, and route data to compatible destinations. It supports multiple sources and destinations and can be deployed on multiple operating systems. Fluentd has a unified logging layer, is horizontally scalable, and has a robust plugin system, making it suitable for collecting log events from thousands of machines. InfluxDB is an open-source time-series database that stores and queries time-series data, offering performance optimization, improved data compression, and built-in tools for working with time-series data. The tutorial demonstrates how to install Fluentd on Ubuntu Linux, receive syslog data from a source, and send it to an InfluxDB instance, showcasing the benefits of using Fluentd as a centralized log event routing engine and InfluxDB for storing and querying time series data.

InfluxDB is an open-source time series database management system that specializes in storing, monitoring, analyzing, and visualizing time series data. It offers features such as support for multiple data types, no network protocol issues during data transmission, ability to encode metadata along with time series data, nanosecond precision in timestamps, and more. InfluxDB has helped various industries improve operational efficiency, monitor performance in real-time, identify seasonality and trends, and predict maintenance needs. To get started with InfluxDB, users need to set up an account, specify credentials, install the Python client, configure the client to communicate with InfluxDB Cloud, and fetch data from InfluxDB Cloud. The fetched data can then be used for time series forecasting using PyTorch Forecasting, which is built on top of PyTorch Lightning. The process involves feature engineering, building a model, fitting the data to the model, generating predictions, and visualizing the results. Overall, InfluxDB provides an efficient platform for storing and processing time series data, making it suitable for industries such as manufacturing, production, and quality assurance.

InfluxDB is being used to monitor air quality and set up an alert system to protect against wildfires, specifically in areas susceptible to wildfires. The Air Quality Template, a prepackaged configuration, includes a dashboard to visualize data and a Telegraf configuration to write data to InfluxDB. A connection to the AirNow API is established using Telegraf, allowing for real-time monitoring of air quality metrics such as PM2.5 levels and ozone quality. An alert system can be set up to notify users when air quality reaches dangerous levels, with InfluxDB sending alerts to various endpoints including Slack. The setup process includes installing InfluxDB, Telegraf, and the Air Quality Template, as well as configuring the AirNow API connection and setting up the alert system.

Telegraf 1.24.3 has been released, featuring updates to input plugins such as Directory Monitor Input and Kafka Consumer Input, improvements in processor and parser functionality, and bug fixes for various issues including float parsing in Zookeeper Input and regression with aggregators in Processor plugin. The release also includes a new feature allowing moving files across filesystem directories and provides an option to remove event metric tags in Splunkmetric Serializer. Users can download the latest release from InfluxDB's Downloads page and report any issues or questions through various channels.

The Telegraf custom builder is a tool that allows users to select which plugins to build the Telegraf binary with, reducing its overall memory and disk footprint. The tool was created to address the growing size of the Telegraf binary and provide a streamlined process for users to build a slimmer binary relevant to their use case. It can be used when a firm decision on which plugins has been made or when building from source is comfortable. The custom builder can help with future scalability of Telegraf, as adding more features over time increases its size. Users can create a customized Telegraf binary by pointing the tool at their Telegraf configuration file and running it through the `./tools/custom_builder/custom_builder` command. The resulting binary is significantly smaller than the full Telegraf binary, making it suitable for resource-constrained devices.

The InfluxData Community Award winners have been announced, recognizing outstanding contributions from community members who have made significant impacts on both the InfluxDB community and the company's products over the past year. The award is given to individuals who have demonstrated sustained involvement and commitment to helping other users of InfluxDB, Telegraf, and Flux through various channels such as forums, Slack, GitHub, and Stack Overflow. The winners are chosen by their peers and include notable contributors like Arvid Teichtmann for his support in the community forums, Thomas Casteleyn for his outstanding support of the Telegraf community, Alfonso Acosta for his significant contributions to the Flux language itself, and Giovanni Luisotto for his broad range of contributions across multiple platforms. The award is a testament to the growing strength and depth of the InfluxData community, which has helped drive growth in the number of users and engagement with the company's products.

InfluxDB's journey began with its focus on storing time series data, but as it evolved, it addressed issues like handling irregular events and high cardinality data sets. The latest iteration, powered by IOx, overcomes these limitations with a columnar structure, enabling efficient data compression and computation of time series on the fly from raw data. This update provides both storage and on-the-fly computation capabilities, as well as native support for SQL queries and edge federation, making it a more versatile time series database.

The upcoming Flux 1.0 release promises to bring several exciting features to users. One major change is the commitment to no longer making breaking changes to the Flux language, ensuring that existing scripts will continue to work without modification. The new `Editions` feature allows users to opt-in to new features without risking breaking changes, while `Modules` provide a way to share and reuse code within organizations. Label Polymorphism enables abstracting over data schema, and Dynamic Type allows for accurate representation of JSON data within Flux. These features are expected to be released behind specific editions or as part of the initial release, with some available today in experimental form. The Flux team has also emphasized their focus on quality and minimal changes, with a goal of releasing new editions only once per year.

InfluxData has announced IOx, a new time series engine for InfluxDB, which aims to improve performance and provide more querying options. The new engine removes cardinality limits, allowing users to handle large amounts of data without impacting performance. It also utilizes columnar database technology, enabling fast query execution and parallelized processing. Additionally, IOx supports real-time analytics, SQL compatibility with the PostgreSQL dialect, and partitioning for efficient data storage and retrieval. This update is designed to support a broader range of use cases, including observability and tracing, and will be available in the InfluxDB Cloud Beta program first.

InfluxData has announced significant product enhancements at its annual developer and community event, InfluxDays 2022, including the introduction of new features such as the InfluxDB Script Editor, Telegraf Custom Builder, and Flux 1.0 support, which aim to advance the developer-centric application building experience with time series data, allowing developers to build applications more efficiently with less code. These new features will be showcased at InfluxDays 2022 and are expected to accelerate application building with more capabilities and less code, improving collaboration, sharing, and flexibility in Flux, as well as enabling footprint reduction of Telegraf, the company's open source data collection agent.

InfluxDB has evolved over the past decade, focusing on making it easy for users to build valuable applications with their time series data. The platform now offers a wide range of tools, features, and resources to facilitate quick touchpoint experiences. At InfluxDays 2022, new features were announced, including Flux 1.0, which introduces collaboration, flexibility, and stability improvements. Flux 1.0 includes Flux Editions for controlled updates, modules for shared code, polymorphism for abstracting column names, and dynamic type to represent JSON data accurately. Additionally, a Custom Builder was introduced for Telegraf, allowing users to build custom agents with only the needed plugins. A new Query Experience was also announced, featuring an improved Script Editor with an integrated schema browser and multi-language support for Python and JavaScript.

InfluxDB has released a new engine that significantly increases performance and allows for unbounded cardinality, SQL support, and real-time analytics, making it ideal for use cases such as events, tracing, and ephemeral data. The new engine also supports native SQL queries, enabling users to connect using Postgres-compatible clients like psql and Grafana's Postgres data source. Additionally, InfluxDB has expanded the Data Explorer UI, made custom data retention easier, and added an onboarding wizard for the Arduino platform, while fixing various bugs to improve the user experience.

InfluxDB has introduced a new feature called Native MQTT, which allows users to collect time-series data from cloud-based pub/sub services without the need for custom code or additional hosting services. This feature is designed to accelerate collecting from MQTT brokers and enables businesses and applications to track real-time and historical change, contributing to forecasting and prediction. Native MQTT eliminates the need for orchestration of additional technology layers or infrastructure management, allowing users to focus on using their collected data. The feature is easy to set up and requires only three basic configuration steps, including specifying broker details, topic names, and parsing rules. It also supports dynamic subscription to topics using wildcards and allows for unstructured data payloads to be parsed using JSON path expressions and regular expressions.