September 2019 Summaries

In InfluxDB Cloud, the author creates a new bucket for downsampled data and uses Flux, a general-purpose data transformation language, to compress the data. The author writes a query that takes the last 5 minutes of data from the raw bucket, computes a mean, and puts this data into the new "compressed" bucket. A task is created to run this query every five minutes. To create a dashboard, the author selects the compressed data and configures the chart to refresh automatically. The author also adds a gauge to display the current soil moisture level from the uncompressed bucket.

InfluxDB, Release Notes` by Russ Savage (Sep 26, 2019) - A new alpha release of InfluxDB 2.0 is now available, featuring enhancements such as anomaly detection and alerting capabilities, improved user interface, and updates to the Flux library. This release is not intended for production usage but rather for gathering feedback on functionality, user experience, and APIs. The latest iteration can be downloaded from the official website, with users encouraged to report any issues or questions in the InfluxDB Github Repo or Community Site.

A new maintenance release of Telegraf, version 1.12.2, is now available, addressing several improvements and bug fixes to various plugins including Agent, apcupsd Input, JSON Parser, Cisco Telemetry GNMI, and CSV Parser, with the goal of enhancing overall stability and functionality.

InfluxDB Cloud 2.0's monitoring and alerting feature allows users to set up checks, which are conditional queries that assign a status based on specific conditions, such as low battery levels or high CPU usage. Notifications are then configured to receive alerts when these checks meet their conditions, providing users with real-time insights into their data. To set up this system, users can create checks by building queries and assigning statuses, and then configure notifications using notification endpoints and rules. This allows for customizable alerting that can be tailored to specific needs and data sources.

The author, David Flanagan, uses InfluxDB Cloud, Telegraf, and Flux to parse Meetup API data, store it in InfluxDB Cloud, and send Slack notifications. He configures Telegraf to fetch events from the Meetup API every hour, stores them in InfluxDB Cloud, and then uses Flux tasks to filter, pivot, and map the data into a format suitable for sending to Slack. The author also handles authentication with Telegraf, confirms data availability using GUI time selectors, and uses the HTTP package to send notifications directly to the Slack API. Additionally, he leverages the secrets API in InfluxDB Cloud to securely store a Slack webhook endpoint, ensuring that sensitive information is not hardcoded into the Flux task.

InfluxDB's Telegraf Jenkins Plugin provides insights about Jenkins nodes and jobs out of the box, including disk space, memory usage, response time, job duration, result code, and more. The plugin is integrated with InfluxDB Cloud 2.0 and OSS versions, allowing users to monitor Jenkins nodes and execute automation vision. Flux, a lightweight scripting language for querying databases and working with data, facilitates sophisticated analysis of Jenkins events to anticipate problems before they occur. Using the plugin, users can easily identify job failures and create custom dashboards to view queries at a glance. A Flux script can be used to isolate jobs with durations greater than two standard deviations from the mean duration of similar jobs, helping to identify potentially problematic nodes. The InfluxDB Cloud 2.0 free tier allows for good throughput, making it suitable for monitoring Jenkins metrics. Overall, the Telegraf Jenkins Plugin and Flux provide a powerful toolset for DevOps teams to monitor and analyze their Jenkins environments.

InfluxDB Cloud 2.0 has been released by InfluxData, expanding its open source time series database service with new features and capabilities, including a generally available release of the time series database as a service (DBaaS) on September 10. The update introduces Flux data scripting and query language to provide greater insights for users, building upon the existing open source InfluxDB 2.0 project, which is still in alpha. The new release aims to support growth in its cloud-based time series database service, enhancing the overall user experience with improved functionality and features.

The InfluxData team has introduced a new indexing mechanism for InfluxDB called Time Series Index (TSI), designed to optimize RAM usage saturation for larger datasets. This feature aims to support hundreds of millions and eventually billions of data points, expanding the database's capability beyond its current limits. The TSI storage engine allows users to have millions of unique time series without being affected by memory constraints, providing a significant technical advancement in the database since the introduction of the Time Series Merge Tree (TSM) storage engine in 2016. By converting from TSM to TSI indexing, InfluxDB can improve performance and reduce memory problems, making it more suitable for large-scale applications. The conversion process involves five steps, including stopping the container, starting it with a specific environment variable, running the conversion command, exiting and removing the container, and finally starting the container without the entrypoint flag to verify the successful change.

Telegraf's http_response input plugin allows for synthetic monitoring by creating requests that simulate user activity to test externally visible behavior. This complements white-box monitoring which provides insight into internal functioning, and is especially useful as an alerting signal for paging. The plugin can gather many white-box metrics using application-specific plugins, and you can instrument your applications using the InfluxDB client libraries. It allows flexibility in terms of how to monitor applications by specifying URLs to query, request methods, custom requests bodies or headers, and regular expression verification. This provides a lot of flexibility for creating monitoring checks that come from outside the system. By setting up an independent infrastructure, you can run Telegraf on your Mac or another external device to collect data about influxdata.com. The plugin queries the InfluxData homepage and looks for specific strings in the response, and individual plugins can define their own collection interval. This provides end-to-end coverage that's useful in catastrophic scenarios when used in conjunction with white box tools.

InfluxDB Cloud 2.0 is now ready for production use, offering a free tier suitable for prototyping IoT projects. The author of this tutorial created a monitoring system using sensors and Arduino, streaming data to InfluxDB Cloud 2.0 via USB from an Arduino microcontroller. The Python code structures the sensor data into a line protocol format compatible with InfluxDB. A Raspberry Pi is used as an edge gateway to send the data to InfluxDB using its REST API. With minimal effort, the author visualized their sensor data in Data Explorer and set up Slack alerts for critical moisture levels below 100. The tutorial barely scratches the surface of InfluxDB 2.0's capabilities, mentioning potential tools like Telegraf, tags, and Flux but not exploring them in detail.

A new maintenance release of Telegraf is available, version 1.12.1, which includes improvements to various plugins such as Agent, apcupsd Input, File Count Input, HTTP Response Input, icinga2 Input, InfluxDB 2.0 Output, Kafka Consumer Input, and MongoDB Input. These updates address several issues and provide better functionality for some of the plugins. The release also fixes bugs and improves documentation. The updated binaries can be found on Telegraf's downloads page.

InfluxData, a company behind the open source time series database InfluxDB, has been recognized as one of the "Big Data 50" companies by Big Data Quarterly. The list highlights forward-thinking companies that are driving innovation in big data, and InfluxData is featured for its technology designed to handle massive volumes of time-stamped data from various sources such as IoT devices and applications. This recognition aims to identify new and disruptive technologies in the fast-changing big data market.

InfluxData has launched a new time series cloud database platform called InfluxDB Cloud 2.0, which is designed to be serverless and pay-per-use. This move marks the first specialized time series database cloud service to adopt this model, allowing customers to set upper limits on monthly costs through rate limiting. The new platform builds upon the popularity of InfluxDB, which has become arguably the most popular time series database with hundreds of thousands of implementations and several hundred customers since its 2013 release.

InfluxDB is an open source time series database designed to handle high write and query loads, capable of storing metrics, events, logs, traces from various systems and industries. Sensu is a monitoring event pipeline that collects, processes, and routes different event types, including discovery, availability, telemetry, and alerts. Grafana is an open source metric analytics and visualization suite used for visualizing time series data for infrastructure and application analytics. InfluxDB and Sensu can work together to create a complete, scalable monitoring solution by ingesting metrics into InfluxDB through the Sensu event pipeline, allowing for data extraction, storage, instrumenting, and visualizing of metric data. The combination provides a robust monitoring solution encompassing event monitoring, data ingestion, and dashboarding, suitable for dynamic, distributed environments.

InfluxData has announced the general availability of InfluxDB Cloud 2.0, a serverless time series platform designed to meet the specialized requirements of modern time-sensitive workloads, providing organizations with cost-effective solutions for observability and analytics applications, and enabling them to make data-driven decisions at scale. With rapid ingestion rates, advanced analytic capabilities, and a free rate-limited tier, InfluxDB Cloud 2.0 aims to improve the developer experience, making it easier for developers and operators to work with time-stamped data and extract maximum value from their applications. The platform provides a cost-disruptive approach to handling large volumes of high-frequency and high-volume time-stamped data, outperforming non-specialized solutions by up to 100x, and supports a wide range of customer applications including SLA-related monitoring, real-time monitoring, click-stream analysis, and dynamic pricing.

InfluxDB Cloud 2.0 is a unified platform that combines storage, UI and visualization, processing, monitoring, and alerting into one cohesive whole, featuring a common API shared between open source InfluxDB 2.0 and InfluxDB Cloud 2.0. The new platform aims to optimize for developer productivity and happiness when using InfluxDB to create time series applications, covering various use cases such as server monitoring, sensor data, real-time analytics, application performance metrics, network monitoring, financial market data, and more. InfluxDB Cloud 2.0 is a serverless platform with a paid usage-based pricing model and features an easy-to-use application for creating monitoring and alerting rules based on time series data. The new release introduces Flux, a new language for querying, analytics, and data processing, and enables users to leverage these concepts to expand on the basic monitoring and alerting system to create more complex applications. The platform is committed to open source, with InfluxDB 2.0 currently in alpha and available as an unrestricted MIT-licensed gift.

InfluxDB is introducing a new version of its platform, InfluxData 2.0, which aims to improve developer happiness and make data collection more efficient. To start collecting data with Telegraf as a data collector in this new version, users need to configure Telegraf by selecting "Configure a Data Collector" and then choosing the services to monitor, specifying the bucket to write to, naming the configuration, and installing Telegraf, configuring the API token, and starting the Telegraf service. The Telegraf configuration file is generated automatically based on the chosen environment, and users can test their connection to ensure data is being written into their bucket. The output portion of the config file specifies the InfluxDB cluster nodes and authentication token, while the input plugins are enabled by default for popular metrics such as CPU, disk usage, memory, network traffic, processes, swap space, and system information. Users can view and download the actual toml file and restart the Telegraf service if changes are made after collecting data has started.

I am excited to join InfluxData as a new member of their team and contribute my skills and experience to help drive the company's vision forward. I was drawn to InfluxData because it aligns with my personal values, including a passion for open source development and a focus on making great experiences for developers. The company's commitment to its community-driven approach and emphasis on collaboration and caring among team members resonated deeply with me. With my background in solving hard programming problems and my enthusiasm for the Flux language, I am eager to help spread this technology to other projects and communities, while also supporting the team members in their goals and aspirations.

The Maxim DS18*20 series temperature sensors are widely used due to their reliability, accuracy, and ease of use. They communicate over a single contact serial interface called 1-Wire, which makes them easy to integrate into projects. The Linux kernel's built-in w1_therm driver enables these sensors to work on any Linux machine with the appropriate physical interface. Telegraf's temp input plugin can read data about temperature sensors from the filesystem during each collection, but it currently only supports a single sensor due to an issue with the DS18*20 sensors and the gopsutil library used by Telegraf. A workaround is to use a Python script that reads the data and executes it through Telegraf's exec plugin. The author plans to iterate on the built-in temp plugin to make it support multiple sensors, potentially by adding a device name field to the SensorKey struct in the gopsutil library.

InfluxDB Enterprise 1.7.8 has been released with several bug fixes, including removing stray debugging print lines, fixing issues with subquery ordering and series handling, and improving error messages for logging levels and data node connections. The Technical Preview of Flux has also been updated to version 0.36.2. Additionally, specific fixes have been added for InfluxDB Enterprise customers, such as clarifying error messages and removing invalid data from hinted handoff.

A new version of Telegraf, an open-source monitoring system, is now available. The 1.12.0 release includes several key features and enhancements, such as the addition of 13 new plugins that can ingest Docker logs, monitor Logstash, output data to custom scripts, and more. Additionally, users can now load external plugins without compiling them into the main Telegraf binary using a new flag. The release also includes over 50 enhancements to existing plugins across Telegraf, such as support for Elasticsearch 7.x and improved JSON parsing. New inputs have been added, including ones for apcupsd, Docker logs, Fireboard thermometers, Logstash, MarkLogic, OpenNTPD, and uWSGI. Furthermore, new outputs, parsers, and processors have also been introduced, such as the Exec Output Plugin, Form Urlencoded parser, Date Processor Plugin, Pivot Processor, Tag Limit Processor, and Unpivot Processor. The binaries for this release can be found on Telegraf's downloads page.

The concept of high and low cardinality in databases refers to the number of unique values present in a column or set, with high cardinality indicating a large number of distinct values such as ID numbers, user names, or email addresses. Low cardinality, on the other hand, is characterized by repeated values like status flags, Boolean values, or gender. Understanding cardinality is essential for query performance and stability, particularly in databases that deal with complex data relationships. While calculating cardinality can be challenging, it's a crucial step in designing more efficient database systems and understanding how to optimize their performance.