September 2022 Summaries

InfluxDB has ranked #1 in the G2 Grid for Time Series Databases and held its leading position in the Momentum Grid for Time Series Databases. The company's time series intelligence capabilities have earned it a spot among leaders in an additional G2 Grid Report, with customers ranking InfluxDB #1 in four categories including Ease of Setup, Market Presence, Ease of Admin, and User Satisfaction. As an open source company, InfluxDB is thrilled to be recognized by its users who praise the platform's power and versatility, with many customers showcasing their success stories on the InfluxDB Customer Page.

InfluxDB Cloud has introduced features such as multi-org support, allowing users to switch between multiple organizations in a single account, and enhanced data ingestion capabilities, including improved AWS integration and support for PostgreSQL plugins. Additionally, the Enterprise version of InfluxDB has seen significant updates with features like enhanced retention policies, APIs, v2 buckets, and Flux. Other improvements include Adaptive Zoom, which provides more granularity on graphs, and bug fixes addressing various user experience issues.

Apache Kafka is a durable, high-performance messaging system that can be applied to various use cases including messaging, data integration, log aggregation, and metrics. It provides an event backbone for storing real-time events for aggregating, enriching, and processing. InfluxDB is a time series database (TSDB) optimized for monitoring, application metrics, IoT sensor data, and real-time analytics. It can be integrated with Apache Kafka using Telegraf to send or receive metrics or event data for both processing and monitoring. This integration allows for the streaming of real-time sensor data from Apache Kafka to InfluxDB.

InfluxDB's Flux is a powerful functional data scripting language used for querying, processing, analyzing, and acting on data, particularly time series data. It offers various features like custom functions, which allow developers to create reusable code snippets that can be applied to specific operations, making the code more efficient, readable, and maintainable. Custom functions in Flux are built using a simple syntax and can be used within scripts or pipelined together with other operators for complex calculations. This enables developers to tailor the language to their application's unique needs and build upon existing libraries and functions.

The new join package in Flux allows for more complex joins on multiple columns, enabling the performance of various types of joins including inner, full outer, left outer, right outer joins. The `join` function has been replaced by the `join.inner()` function and other specialized functions like `join.time()`, which join tables exclusively on the `_time` column. These new functions provide more flexibility and power for joining data in Flux, making it easier to handle late-arriving data and perform complex joins. The `join` package also allows for custom as functions, enabling users to create their own logic for handling null values and other edge cases. Additionally, the `fill` function with the `usePrevious` parameter ensures that previous values are not dropped when performing joins. The new join package is a significant improvement over the old join functionality in Flux, providing more power and flexibility for developers working with InfluxDB data.

We love to write and ship code to help developers bring their ideas and projects to life, constantly improving our product to meet developer needs and accelerate Time to Awesome. This week, we're highlighting a new feature in InfluxDB Cloud UI that allows users to get more from their graphs through Adaptive Zoom, also known as Dynamic Query Zoom. With this feature, users can trigger an auto-requery of data upon zooming in, revealing more granular information, and the updated window period is displayed at the top right of the graph cell. Users have the option to toggle-off Adaptive Zoom and can statically assign a constant value to the window period via the script editor, with helpful definitions provided for better understanding.

In a world with increasingly complex IoT applications, MQTT is becoming the standard protocol for applications that operate in environments where network connectivity is intermittent or unreliable. Businesses are seeing value from making MQTT part of their tech stack by using it to communicate between consumer IoT devices, such as smart home devices and wearable devices, manufacturing facilities, automobiles, retail businesses, logistics companies, cities, and agriculture businesses. MQTT provides the performance benefits that are ideal for these applications, including reducing bandwidth usage and maximizing battery life, and is being used in industries such as automotive, where it's enabling predictive maintenance to extend the life of vehicles, and smart cities, where it's providing a common communication protocol for sensors and devices made by many different vendors.

InfluxDB is a time series database designed for high write throughput and efficient querying of data based on timestamps, offering improved performance and features compared to traditional relational databases. Time series databases are gaining adoption due to the need for better performance and advanced features in software applications, as well as the ease of selecting specialized tools with microservice architectures. These databases are optimized for specific use cases such as application monitoring, IoT applications, and security, and offer trade-offs between write and query performance, compression, schema flexibility, and maintenance costs.

Telegraf 1.24.1 has been released with several updates, including improved configuration validation, fixes for issues with the Ceph and Modbus input plugins, and enhancements to the Processor plugin, such as the ability to parse tags in the processor parser. Additionally, dependency updates have been made to various libraries, including cloud.google.com/go/pubsub, github.com/urfave/cli/v2, and cloud.google.com/go/bigquery. Users can download the latest release from the InfluxDB website and report any issues or questions through the InfluxDB Community Slack or GitHub Repo.

GitOps is a framework for managing cloud-based infrastructure, taking lessons from DevOps and modern application development methodologies. It uses Git as a single source of truth for configuration, automating processes to match the state described in the repository. This approach provides faster deployments, error resolution, and auditing benefits. Argo is a collection of tools that can be used together or independently to implement GitOps best practices, including Argo Workflows, Argo CD, and Argo Rollouts. These tools provide features such as workflow engines, continuous deployment pipelines, and rollout management for safely moving new deployments into production. Argo Events handles events and takes automated action with Kubernetes, providing flexibility in automating processes with preferred tools. The project has gained over 25,000 GitHub stars and is currently the 6th most active CNCF project.

The latest release of InfluxDB Enterprise, version 1.10, brings several enhancements and bug fixes to the platform. The database now supports CRUD operations for buckets, retention policies, APIs, and Flux transformations, providing more flexibility and control for users. Additionally, features like wildcard support for retention policies, slow query logging, and vectorized arithmetic operators have been added or improved. Bug fixes address issues with backups, influx_inspect, and other areas of the database. The release also includes maintenance updates to Flux and Go, as well as improvements to build processes on macOS.

Telegraf 1.24.0 has been released with several key features, including a change in the default minimum TLS version for security reasons and the addition of new plugins such as AWS CloudWatch Metric Streams Input Plugin, Linux CPU Input Plugin, Supervisor Input Plugin, UPSD Input Plugin, CSV Serializer, PostgreSQL Output Plugin, RedisTimeSeries Plugin, Stomp (ActiveMQ) Plugin, NSDP Input, among others. The release also includes a new serializer, CSV Serializer, and new external plugins such as AWS CloudWatch Metric Streams Input Plugin, Linux CPU Input Plugin, Supervisor Input Plugin, UPSD Input Plugin, NSDP Input.

InfluxDB offers two primary methods for collecting data from IoT devices using MQTT: Native Collector and Telegraf. The Native Collector is a cloud-to-cloud connection feature that allows direct, cloud-to-cloud connections between cloud services and InfluxDB, making it ideal for cloud-first solutions. It involves a simple three-step configuration process and can start collecting data immediately after setup. On the other hand, Telegraf is an open-source plugin-based data collection agent that provides extensive control over processing and transforming MQTT messages as they arrive. With its lightweight design and minimal resource usage, Telegraf offers flexibility in terms of installation and deployment, making it a valuable solution for developers who need more customization options.

MQTT (Machine-to-Machine) is a machine-to-machine communication protocol that allows devices to publish messages to a broker under specific topics, and other devices can subscribe to those topics to receive information. It's popular due to its low bandwidth requirements, making it suitable for IoT devices with limited network connectivity. MQTT brokers connect to client devices, authenticate connections, and store data temporarily, enabling easy scalability as new devices are added to the architecture. Popular MQTT broker options include Mosquitto, EMQX, HiveMQ, AWS IoT Core, and Azure IoT Hub, each with their own pros and cons. Devices need to run client software to connect to a broker, with popular open-source options like MQTT X, Mosquitto's command-line interface, and HiveMQ's Java-based client library available. InfluxDB can integrate with MQTT by using plugins such as Telegraf input plugin or Native Collector, providing efficient data storage and analysis capabilities for IoT sensors collecting time series data.

InfluxDB is a popular choice for handling time series data in the Internet of Things (IoT) due to its ability to handle huge amounts of sensor data. IoT devices, such as smart home thermostats and industrial manufacturing equipment, generate large amounts of time series data that InfluxDB can store and analyze. To incorporate InfluxDB into an IoT system, companies can install it at the edge or use Telegraf on edge devices to collect data for cloud-based computation. InfluxDB supports working with MQTT in two key ways, collecting data from a broker and sending data back to it, allowing for automation and machine-to-machine communication. The platform offers various options for sending data from an MQTT broker to InfluxDB, including Telegraf's plugin and InfluxDB's MQTT Native Collector, which provides a fully managed cloud-based solution.

The Upalgo platform, a SaaS solution for time series data analysis, relies on InfluxDB as its datastore, leveraging its windowing feature and active developer community to handle vast amounts of sensor data. The platform applies machine learning to automate anomaly detection and labeling, iterating on these processes to improve data models. With continuous data ingest and read-intensive learning challenges, the Upalgo platform uses a REST API to consolidate read issues and optimize resource consumption. Anomaly detection requires large amounts of data, and labels are crucial for establishing a baseline truth for the data, which is automatically identified by the platform's anomaly detection feature. The Upalgo platform generates data visualizations using InfluxQL to query data from InfluxDB, accelerating key processes faced by machine learning data scientists.

The situation involved an enterprise IT company hosting a large industry event, requiring real-time monitoring and analysis of its application performance. To address this challenge, they brought in IT Squared and Riverbed to build an application performance monitoring (APM) solution using Riverbed's AppInternals and InfluxDB, which provided senior leaders with top-level views and SREs with granular data, enabling them to quickly identify and resolve issues.

InfluxDB Cloud has introduced Native Collectors, a new feature that allows users to spin up subscribers for well-known industry protocols, enabling the connection, ingestion, and parsing of data into InfluxDB. The first native protocol released is MQTT, and InfluxDB Cloud will be connecting two popular IoT-based platforms, Kepware and The Things Network, using Native Collectors. The plan is to create a table outlining the features that will be covered in a two-part series, including connecting an edge platform to the cloud, sending line protocol within an MQTT payload, authentication, JSON payload parsing, and more. In this first part of the series, Kepware's industrial example will be explored, where a sample Kepware server project is used to demonstrate how to configure Kepware to talk to HiveMQ's public MQTT broker, update the MQTT payload format to support line protocol, and set up an MQTT ingester in InfluxDB Cloud. The second part of the series will cover connecting The Things Network using Native Collectors, with community feedback and ideas for new protocol types welcome.

The InfluxDB native collector's integration with MQTT provides a fast and easy way for users to create cloud-to-cloud data pipelines. This feature is ideal for IoT applications that rely on MQTT, allowing users to directly ingest data from popular MQTT brokers without the need for transformation or additional agents like Telegraf. The Native MQTT collector offers a simple three-step configuration process, making it a no-code solution that enables users to leverage MQTT without adding complexity to their cloud-first solutions. This integration is now available for all paid InfluxDB accounts and provides an alternative option for users who lack the need or resources to install a collection agent.

InfluxDB is a time-series database that can handle large volumes of data at high ingest rates, making it suitable for detecting anomalies or forecasting time-series data. To effectively use InfluxDB for these tasks, users need to query their data using Flux, the native query and scripting language, and then use client libraries such as Python's Pandas to manipulate and analyze the data. Popular Python tools like TensorFlow, Keras, and Prophet can be used in conjunction with InfluxDB to perform more advanced forecasting and anomaly detection tasks. Additionally, Amazon Forecast provides a convenient way to make forecasts without requiring specific training in model selection or deployment. The InfluxDB v2 Python Client Library offers enhancements such as support for Pandas DataFrames and flexible timestamp formats, making it easier to write data to InfluxDB with popular libraries like Pandas. However, InfluxDB has limitations when it comes to tackling time-series data science problems, including a lack of native tooling and limited support for sophisticated forecasting or anomaly detection algorithms using Flux.

InfluxDB has launched Native Data Collectors, enabling developers to easily bring data into InfluxDB without additional software or agents. This feature is particularly useful for IoT applications, allowing for fast data ingestion and transformation. Additionally, the company released new CLI Onboarding tools, making it easier for users to get started with any client language. The platform also introduced Invokable Tasks, which enable developers to invoke scripts and pass variables to them, improving code management and scalability. Furthermore, InfluxDB fixed various bugs to improve the user experience, including issues related to data visualization and navigation.

InfluxDB is a popular time series database that enables efficient storage and querying of large amounts of data collected over time. It offers a NoSQL-like database with good technical documentation, making it easy to install, configure, and get started with. InfluxDB stores data in measurements, tags, and fields, allowing for flexible data modeling and querying. The database provides mechanisms for writing and querying data using the API, CLI, and Telegraf plugins. It supports various programming languages and offers features such as sharding, partitioning, and secondary indexes. When choosing a time series database, factors to consider include data model, query language, reliability, performance, ecosystem, operational management, and support. InfluxDB stands out in the growing market of time series databases due to its overall performance and technical documentation.