March 2022 Summaries

The Flux language offers a declarative approach to data transformation and analytics, transforming data by pipe-forwarding it between functions where each function transforms the data. The fundamental components of Flux include `from()`, `range()`, `filter()`, `yield()`, and `aggregateWindow()` functions. InfluxDB users can convert common InfluxQL queries into Flux queries to gain a deep understanding of the language, even without studying its basics or annotated CSV output format. Flux provides more power and flexibility than InfluxQL, allowing users to perform joins, manipulate timestamps, filter with conditional logic, conditionally transform data, and write custom functions for advanced analytics. The language requires a shift in thinking about data structure, hierarchy, and shape, and users can leverage the `map()` function to transform each record of the data piped into it. With Flux, users can create alerts and notifications, write median absolute deviation anomaly detection algorithms or Naive Bayes classifiers, and perform math across measurements. The language offers a far improved developer experience compared to TICK scripts, allowing for more flexibility and functionality in data analysis and processing.

Node-RED is an open-source, low-code, visual programming tool that allows users to rapidly glue together different types of hardware and software. It provides a number of pre-built nodes that enable non-developers to create custom integrations and automations using a visual interface, while also allowing developers to write their own custom nodes using JavaScript or external services. The tool has a large community with over 14,000 stars on GitHub and a library of over 3,000 pre-built nodes and 2,000 customizable flows that cover many common use cases. Node-RED consists of core components including nodes, which are triggered by receiving a message object from a previous node or an external event, and flows, which are containers for organizing a sequence of nodes. The tool also provides features such as context, messages, and debug nodes to help users develop and troubleshoot their workflows. With its ease of use and versatility, Node-RED is suitable for a wide range of applications, including IoT projects, smart home automation, event processing, and data visualization and analysis.

Chronograf 1.9.4 has introduced a Flux Builder to help users new to Flux with an easy on-ramp for building their first query, mirroring the experience offered by InfluxDB Cloud and OSS. Additionally, improvements have been made to Kapacitor integration within Chronograf, allowing users to filter scripts and tasks by name, enhancing the overall performance of loading and listing TICKscripts and Flux Tasks.

InfluxDB has been named a winner in the Data Breakthrough Awards for the Best Use of Data for IoT Applications category, recognizing its industry-leading time series platform as vital to the IoT technology stack. The award validates InfluxDB's ability to help developers build and integrate real-time IoT, analytics, and cloud applications with time-stamped data, enabling customers to identify trends over time and ensure mission-critical processes are consistent, efficient, and reliable. This recognition follows recent announcements of accelerating momentum in IoT and Industrial IoT, as well as new InfluxDB features and capabilities for developers working with time series data at the edge.

InfluxDB is gaining popularity among industrial organizations due to its modern approach and better value compared to legacy process historians, which were once considered a "must-have" for critical industrial operations. The shift is driven by the collapse of silos and the emergence of new technologies such as OPC-UA, HTTP, MQTT, and cloud services, which enable secure data capture from industrial assets in real-time. This change is also influenced by the evolving workforce, with younger workers being tech-savvy and obstacle-adverse, and a new mindset emerging that prioritizes technology modernization projects and open-source solutions like InfluxDB. The platform's agility, scalability, and microservices-based approach make it an attractive alternative to legacy process historians.

Mist is an open-source multicloud management platform that provides a unified interface for managing resources across diverse cloud services, hypervisors, containers, and bare metal servers. The platform empowers IT teams to manage their infrastructure by providing a single point of view, enabling easy provisioning, orchestration, monitoring, cost management, and automation. Mist was born out of frustration with the complexity of managing multiple cloud services, as its cofounders were struggling to provide insights to their customers about their resources, access control, and automation processes. The platform connects to infrastructure providers with a native API, allowing users to view performance, spin up VMs, troubleshoot, and destroy them, as well as monitor and compare performance across different clouds. Mist also allows for fine-grained resource permissions, enabling managers to set access based on resources, at the cloud level, globally, and via tags. The platform uses InfluxDB as its core time series database, which provides insights into monitoring metrics, metering data, and automating common processes.

The question of how to get data into a database is fundamental for developers when dealing with local devices, but even more challenging when adding data from edge devices. With the exponential increase in IoT edge devices, companies need proven and reliable ways to collect data from them. Three different approaches to collecting data from edge devices are discussed: crawl, walk, and run. The crawl approach involves storing raw data directly from an edge instance of InfluxDB to a cloud instance, with advantages including quick setup and low edge computing power requirements but disadvantages such as increased unclean data in the cloud instance. The walk approach involves downsampled data being pushed to the cloud where it is written into InfluxDB, offering reduced unclean data and lower edge connection throughput needs but requiring more edge computing power and memory. The run approach involves collecting, processing, and writing data on the edge, significantly reducing unclean raw data sent to the cloud instance and resilient to connectivity outages between cloud and edge, but requiring more edge resources. Each approach has its advantages and disadvantages, and choosing the right one depends on the tech stack and use case.

Telegraf has released a faster and more memory-efficient implementation of the Line Protocol Parser as part of its v1.22.0 update, which is now available for users who make heavy use of line protocol and parse large amounts of data. This new parser can be enabled by setting the `influx_parser_type` option to "upstream" in configuration files, allowing users to opt-in to the improved performance without affecting existing configurations. The default parser will switch to the new parser in a future release, and users are encouraged to try out the updated version of Telegraf to take advantage of its benefits.

Telegraf 1.22.0 has been released, featuring improved Line Protocol parsing, an option to select SNMP translator, and enhancements to the Varnish Plugin. The new release also includes new inputs for Consul, Nomad, Vault, Hugepages, Mock, Redis Sentinel, socketstat, and XtremIO, as well as a Noise Processor for adding noise to numerical field values. Users can enable faster parsing of Line Protocol with a single configuration option or select between gosmi and netsnmp translation. The new collection offset agent configuration setting allows users to define a time offset from the collection interval, while dynamic tagging has been added to the gNMI input plugin.

Kapacitor 1.6.4 has been released with several security-related improvements, including support for SASL-based authentication when integrating with Kafka, deprecation of weak ciphers such as DES and RC4, and removal of support for outdated TLS versions like 1.0 and 1.1. The update also includes fixes and enhancements not explicitly mentioned in the release notes. Users can obtain the latest version from the InfluxDB downloads page and report any issues or questions to the InfluxDB Community Slack channel or GitHub repository.

In the development of InfluxDB's IOx, a new asynchronous Kafka client called RSKafka was created to replace rust-rdkafka. The reasons for this change include complexity, bindings limitations, buffering/modularity issues, expertise and insights concerns, and feasibility considerations. RSKafka is fully-async and uses the Rust standard library for reading and writing integers in big-endian encoding, with a focus on modularity and flexibility. Testing involved serialization roundtrips, serialization snapshots, integration tests, unit tests, and fuzzing to ensure spec compliance and edge case coverage. The client provides features such as batching, custom aggregation algorithms, and extensions through traits, making it suitable for various use cases including IOx. RSKafka is deliberately minimal in its current form but open to community contributions and is available on GitHub.

InfluxDB is increasingly being used in IoT solutions to store data from connected devices. An instance of InfluxDB can now be used on IoT edge gateways as a data historian to analyze, visualize, and eventually transmit aggregated IoT data up to a centralized server. To connect an instance of InfluxDB on your IoT Edge device to another instance of InfluxDB in the cloud, you need a local instance of InfluxDB to use as your edge historian. You can create this by running the official Docker image and completing the setup by going to the Load Data -> API Tokens screen and grabbing the token string that was generated during setup. The token is used to set up your Influx CLI configuration. A bucket is created on this instance to store device data, with a retention policy of 1 hour in this example. To send the data from the edge instance to the cloud instance, you can create a task using Flux script that reads data from the devices bucket, performs downsampling, and sends it to the cloud instance. The downsampling is done by using the aggregateWindow function to calculate the mean value of each field over a rolling 5-minute window. This reduces the volume of data being sent to the cloud while still providing high-level use cases with the original granularity. To make the task more resilient, you can extend the retention policy on the devices bucket and create another task that reads from the downsampled data in the northbound bucket and sends it to the cloud instance every 5 minutes.

InfluxData, creator of the leading time series platform InfluxDB, has been recognized by Forbes as one of America's Best Startup Employers, ranking among the top 25% for employee satisfaction, employer reputation, and hypergrowth. The prestigious list recognizes the top 500 startups in the US that excel in these three criteria. InfluxData was also recently named one of Built In's Best Places to Work 2022 for Bay Area midsized companies, marking its second consecutive year as a top Bay Area employer. The company is committed to keeping InfluxData an exciting and rewarding place to work as it scales its team and enters its next phase of growth.

InfluxDB is an open-source time series database that stores and analyzes data points over time. It's optimized for storing and querying large amounts of time-stamped data, making it particularly useful for applications such as financial trend analysis, IoT sensor monitoring, and sales forecasting. InfluxDB provides a free open-source version that can be run locally, as well as a cloud-based version that supports major cloud services like AWS, GCP, and Azure. The database has a pre-built dashboard for visualizing data and supports asynchronous and synchronous writes. It also includes features such as data point structure, CRUD operations with Python client library, reading data using queries, updating data by performing write operations, deleting data using delete_api, and best practices for writing data.

This article discusses the importance of using time series databases (TSDBs) to efficiently store and query large amounts of time-stamped data. It explains how traditional relational database management systems (RDBMS) can become inefficient when dealing with high volumes of data, particularly in scenarios where data is constantly being added or updated over a long period. In contrast, TSDBs are specifically designed to handle this type of data and provide fast write and query performance, making them ideal for applications that require real-time monitoring and analysis of time-stamped data. The article highlights the benefits of using InfluxDB, an open-source TSDB, which provides features such as high-performance writes, efficient querying, and advanced analytics capabilities. It also demonstrates how to set up and use InfluxDB with a C++ client library, including examples of writing data to the database, querying data, and visualizing results in a graph. The article concludes by emphasizing the importance of choosing the right database solution for specific use cases and highlighting InfluxDB as a suitable option for applications that require efficient time-stamped data storage and analysis.

The Telegraf project has decided to discontinue the creation of release candidates (RC) releases due to low value in generating bug reports and support conversations. Instead, InfluxData will provide Telegraf nightlies for users to test new features and fixes, as well as pull request artifacts that allow users to download and validate fixes through a CI process. These changes aim to improve the overall process of releasing updates to the Telegraf project.

At InfluxData, time series data is a passion project that aims to deliver faster Time to Awesome for users. The company has launched InfluxDB University, a platform offering free live trainings and self-paced courses to help users get up and running with the InfluxDB platform more efficiently. The initiative provides hands-on training in scalable ways to focus on developer skills needed for success using the platform. With courses covering beginner topics to advanced configurations, InfluxDB University is open and available to all, offering digital credentials to learners who complete certain courses through a partnership with Credly. The company plans to continually add new content and iterate on existing courses, seeking feedback from users to make the learning experience better.

InfluxDB University is an online education platform launched by InfluxData to provide developers with technical skills to build applications using time series data, offering on-demand and live instructor-led courses at no cost. The platform aims to deliver faster Time to Awesome for developers building real-time applications with InfluxDB, providing resources and best practices to reduce the time and effort required. The university offers introductory courses covering key elements of the platform, as well as more advanced topics around IoT applications and enterprise features, with plans to add new content in the future.

The tech industry and open source community are predominantly male-dominated, with a significant gender gap in participation and leadership. Women face various barriers to entry, including intimidation from criticism, lack of mentorship, and difficulty finding their place within the community. However, women who contribute to open source projects often share advice on how to overcome these challenges, such as seeking out female role models, building relationships with other women developers, and maintaining a sense of belonging in STEM fields. To increase diversity in open source contributions, it is essential to make it easier for women to get started, provide inclusive communities, and promote connections between women developers.

The types package in Flux allows for more robust data querying by enabling filtering on column types, reducing the risk of runtime errors that can occur when dealing with unknown or mismatched data types. This is achieved through the `isType` function, which can be used to check the type of values within a dataset, ensuring that only valid data is processed and aggregated. By leveraging this functionality, developers can write more reliable and maintainable queries, even in complex data pipelines where data types are uncertain or variable.

Joining InfluxData as a Software Engineer allowed Luis Casillas to grow technically in his career, feeling fortunate to have joined the deployments team where he can share knowledge and help others. He was drawn to the company's core values of humility driving learning, embracing failure, and shared values that promote a better product. The sense of pride in sharing knowledge and helping others resonated with him, as well as the collaborative environment with many groups worldwide. With much to learn and accomplish, Luis is excited to contribute to InfluxData's success.