April 2022 Summaries

Telegraf, a popular monitoring tool, has released version 1.22.2 with several improvements and bug fixes across its various plugins and output options. The update includes enhancements to the Aerospike Input plugin, where a statistic query bug was fixed, and updates to the Alibaba (Aliyun) CMS Input plugin to improve discovery capabilities. Additionally, the GNMI Input plugin now uses mutexes to avoid concurrent map read/writes, while the MySQL Input plugin correctly parses various numeric forms. The update also includes improvements to the InfuxDB v2 Output plugin, updates to dependencies such as Azure Kusto-go and OpenTelemetry, and fixes for issues with Telegraf's behavior on Windows and with high CPU utilization.

InfluxDB OSS 2.x and InfluxDB Cloud allow users to perform data transformation tasks using Flux scripts, which replace Kapacitor's TICKscripts. Using Flux offers several advantages over Kapacitor, including a more flexible developer experience, easier maintenance of a single language for all time series management, and the ability to write metadata about checks or transformations. However, some features like real-time stream processing and socket-based UDFs are not available in Flux tasks. The transition from Kapacitor workloads to Flux tasks requires careful consideration of cost and benefits, with InfluxDB Cloud and OSS 2.x offering backwards compatibility for Kapacitor workloads.

Kafka offers powerful features like schema design, high availability, data persistence and durability that can be utilized in an IoT architecture. By redesigning a demo to utilize topic design and partitioning, Kafka provides a natural division between devices, achieving similar benefits as MQTT's topic hierarchy structure. Kafka partitions provide consistent ordering of records and parallelism for performance purposes, allowing for topic parallelism and topic replication. Additionally, Kafka Connect enables the use of enterprise-grade connectors like InfluxDB, providing a seamless integration with existing systems. However, Kafka may not be suitable for all IoT solutions due to its requirements for stable networks and lack of key data delivery features.

MQTT and Kafka are two popular messaging protocols used in IoT infrastructure, each with their own strengths and weaknesses. MQTT is a lightweight protocol suitable for low-bandwidth and low-latency applications, while Kafka is a more robust protocol designed for high-throughput and fault-tolerance. Kafka offers features such as schema design, high availability, data persistence and durability, topic design and partitioning, which can be beneficial in an IoT architecture. By utilizing partitions and record keys, Kafka can provide consistent ordering of records and parallelism for performance purposes. Additionally, Kafka Connect provides a catalog of powerful premade connectors for well-known enterprise systems and data storage engines like InfluxDB, allowing for seamless integration with existing ecosystems. However, Kafka may not be the best choice for IoT applications that require key data delivery features such as Keep-Alive and Last Will, which are more commonly found in MQTT.

Kafka and MQTT are two popular protocols used in IoT architecture, but they complement each other rather than competing. Kafka is an event streaming platform that promotes high throughput, availability, and connectors to third-party platforms, making it suitable for IoT applications. However, it requires stable networks with good infrastructure and does not offer key data delivery features like Keep-Alive and Last Will. In comparison, MQTT has a simpler setup but may require more maintenance due to its flexible nature. A demo was created using both protocols to send emergency generator data, showcasing the differences in their implementation, including the establishment of topics, data preparation, and serialization. The author plans to release two more blogs in this series, covering more Kafka features and applying best practices to a real IoT project.

You're invited to the Kafka Summit London 2022, where you can meet InfluxData and the InfluxDB maintainer team at booth G6. In preparation for this event, a variety of resources have been put together, including technical papers, blogs, templates, downloads, and more, to get you started with InfluxDB and Kafka. To celebrate their return, InfluxData is giving away an Arduino IoT kit, and will offer live demos featuring InfluxDB and Kafka, as well as on-site meetings with sales engineers to discuss solutions and provide advice. You can also grab swag, including famous socks and stickers, at the booth. Additionally, registering for the Arduino IoT kit right from the booth will get you started with InfluxDB for free.

Time series data provides critical insights into system interactions and can be used to monitor security threats in real-time, helping companies identify patterns and anomalies that may indicate a security breach. By using this type of data, companies can create deeper security profiles by placing events in context with other events and patterns. InfluxDB is being used to collect and process time series data to build these profiles, allowing for the identification of authentication patterns, elimination of false positives, and tracking of employee security activity across multiple SaaS providers. The solution also includes capturing and visualizing this data, as well as exploring new features such as Bollinger bands to establish normal ranges and detect anomalies.

InfluxDB on the Road is a community event series for developers building IoT, analytics and cloud applications with InfluxDB, a leading time series platform. The one-day workshop events are free to invited attendees and focus on educating and enabling developers to be successful in building applications with time series data. Attendees will learn from InfluxData's executive, product and engineering teams, network with the community, and receive technical training sessions led by the company's engineering team. The workshops cover topics such as popular use cases, product roadmap updates, and delivering faster Time to Awesome for builders. InfluxDB on the Road events are scheduled across the United States and Europe, including locations in San Francisco, Boston, Chicago, Berlin, London, and Amsterdam.

The Industrial Revolution marked a significant shift from manual work to mechanized manufacturing, transforming the way humans work. Today, we are in the midst of another industrial transformation, referred to as Industry 4.0, which involves feeding data into autonomous systems that enhance manufacturing processes. To generate consistent and predictable output, manufacturers have adopted physical instrumentation, such as sensors, to measure different aspects of a process. Critical context for this data is provided by time-series data, which includes a timestamp, allowing industrial operators to make sense of it. Industry 4.0 aims for interconnection, information transparency, technical assistance, and decentralized decisions. Time-series data plays a crucial role in almost every aspect of industrial operations, both physical and virtual. However, legacy data historians are not ideal for an Industry 4.0 system due to cost, vendor lock-in, scalability issues, poor developer experience, and siloed data. Instead, open-source time-series platforms like InfluxDB offer a more suitable replacement, providing flexibility and connectivity necessary to evolve industrial operations. By leveraging InfluxDB, manufacturers can harness data, provide critical information to workers on the factory floor, and add measurable value throughout the manufacturing process.

Telegraf's data collection settings allow users to fine-tune their configuration for optimal performance. The `interval` setting determines how often data is collected, with lower values resulting in more frequent collections. The `round_interval` setting can be used to round the collection interval, ensuring uniform timestamps across all sources. Additionally, the `collection_jitter` setting can be used to add randomness to the collection process, while the `collection_offset` setting allows for precise scheduling of collections. Finally, the `precision` setting enables users to set the granularity of their timestamps, ranging from seconds to nanoseconds.

Network to Code, a vendor-agnostic network automation solution provider, helps enterprises adopt modern DevOps practices by applying tools and techniques used in software engineering to their IT and network infrastructure. To collect telemetry data without reinventing the wheel, Network to Code uses Telegraf, an open source server agent that collects metrics and events from numerous sources, and InfluxDB, a time-series database designed for working with time-series data, which provides query performance, data visualization, reduced storage costs, and automated workflows. By leveraging these tools, Network to Code enables customers to transform the way their networks are deployed, managed, and consumed, reducing operational costs, increasing reliability, and improving speed and agility in IT teams.

WP Engine is a large web hosting platform that hosts over 1.5 million websites in over 150 countries, serving 175,000 customers and processing 5.2 billion requests per day. When their monitoring solution went down during an outage, they needed a scalable solution to handle the scale of their data needs on a global level. WP Engine built a custom observability platform using Telegraf for data collection, Kubernetes for aggregation, Kapacitor and InfluxDB OSS for alerting, Google Pub/Sub for data transfer, and Chronograf and Grafana for visualization. The new system ingests 5 million points every minute, twenty times more metrics than the old system, and stores that data in multiple locations, eliminating a single point of failure issue and reducing human cognitive overhead by half.

InfluxDB OSS has recently received significant improvements with the release of version 2.2.0, which includes features such as OSS to Cloud data replication, allowing for the replication of remote buckets between edge and cloud deployments. Additionally, security enhancements have been added, including private IP validation for Flux and template fetching, which can help prevent SSRF attacks. The Flux language has also seen advancements with new experimental packages, functions, and performance enhancements, such as vectorization and continued bug fixes.

Telegraf 1.22.1 has been released with several improvements and bug fixes, including changes to the agent, input plugins, output plugin, serializer plugin, and new external plugins like PSI input. The update includes fixes for issues such as silent failures, slow tests, inconsistencies in SQL queries, and improved OpenBSD support.

This article discusses time series databases, specifically InfluxDB, which is designed to efficiently store and manage large amounts of time-stamped data. Time series data has trends and captures a point in time, making it essential for various applications such as IoT, real-time analytics, and retail sales forecasting. InfluxDB offers scalability, usability, and increased productivity compared to traditional storage mechanisms. The article provides step-by-step instructions on setting up InfluxDB using R language, connecting to the database, writing data, querying data, and making predictions using time-series models. It highlights the benefits of using a time series database like InfluxDB for real-time applications and provides resources for further learning and optimization.

InfluxData, the creator of the leading time series platform InfluxDB, has been recognized by Comparably as having the best company outlook and best engineering team among small to mid-sized companies. This recognition is based on ratings from current employees, who praised the company's leadership, professional development opportunities, compensation, work-life balance, perks, and benefits. The awards are a testament to InfluxData's focus on giving developers the best platform for time series data, which is seen as having massive potential for growth. The company has also been named one of America's Best Startup Employers by Forbes and one of Built In's Best Places to Work 2022, further solidifying its reputation as a great place to work.