July 2020 Summaries

InfluxDB is being used by Alex Skrivseth, Operations Manager at The Shed App, to monitor gas station tank levels in real-time, providing valuable data to fuel truck drivers. The system uses IoT sensor data from the tanks, which are manufactured by companies like Veeder-Root, and extracts insights such as how quickly stations can run out of fuel if not refilled on time. The app fetches data from every gas tank at the monitored stations, enabling drivers to check current levels and plan their refueling routes more accurately. InfluxDB's ease of use and scalability have allowed Alex to collect metrics from 576 data points per station every ten minutes, providing a wealth of information for analysis and pattern detection. The solution has improved communication between gas stations and truck drivers, reducing the time spent on back-and-forth with dispatchers. With its capabilities, InfluxDB is being considered for expansion into other local gas stations and potentially collaborating with larger oil and gas companies to share this technology.

Playtech is a leading online gaming software vendor that uses InfluxDB to improve its observability and reduce data collection metrics overload. The company collects vast amounts of data from various sources, including sensors and websites, but only uses the most critical data to inform its operations. With over 50 multibranded sites worldwide, Playtech relies on InfluxDB for production system-level monitoring and organizational monitoring, tracking customer experience and tying issues to specific engineering teams. By using a subset of relevant metrics, Playtech has reduced operational costs and improved dashboard usability, avoiding the "data spaghetti" issue that can occur when showing too much data at once. The company's use of InfluxDB has enabled it to make data-driven decisions and improve its overall performance.

InfluxDB is being used to track the position of the International Space Station by Sean Brickley, an intern at InfluxData, as part of a personal side project to work on throughout his internship. The data was obtained from a public API and stored in an InfluxDB instance, but Brickley found it challenging to visualize the coordinates on a world map due to the nature of time-series databases. However, he discovered Flux's experimental geo package, which allows for plotting latitude and longitude measurements as coordinates over time, making it possible to create a dashboard that accomplishes what he wanted with ease. The features are still experimental, but the results are promising, and Brickley is excited to continue working on this project and share updates.

InfluxDB 2.0.0 Beta 15 has been released, featuring several potentially breaking changes, including the removal of the `migrate` command from the `influxd` binary, restrictions on variable names in the UI to avoid conflicts with Flux reserved words, and enhancements such as improved CLI tooling, enhanced UI performance, and an upgraded Flux Query Engine. The release is not intended for production use and invites users to provide feedback through the InfluxDB Community Slack channel.

InfluxDB Cloud is now available on all major cloud platforms: Microsoft Azure, Amazon Web Services (AWS), and Google Cloud Platform (GCP). This allows users to run the leading time series database in whichever cloud provider they prefer. InfluxDB Cloud provides effortless scaling, flexible pricing, and integration with various Azure services. It also offers a range of features for time series analysis, including statistical analysis, forecasting, and anomaly detection. The platform is designed as a serverless database architecture, eliminating the need to provision or pay for unused capacity. Users can access InfluxDB Cloud on Azure through the cloud2.influxdata.com portal, with the option to choose from multiple regions and providers.

A new maintenance release for Kapacitor is now available, addressing security vulnerabilities by updating package validation mechanics to use SHA-256, replacing MD5. This release includes improvements such as support for Discord and Microsoft Teams event handlers, enhanced security features like TLS 1.3 configuration, platform-related bug fixes, and updated binaries for arm64 architecture.

The latest release of Telegraf, version 1.15.1, is now available, featuring several key improvements and new plugins, including a new input plugin for NGINX Stream STS and Redfish, as well as new processors like Defaults Processor Plugin, Execd Processor Plugin, and Filepath Processor Plugin. The logparser input has been deprecated in favor of the tail input with data_format = "grok", while other fields used primarily for debugging have been removed from the splunkmetric serializer. Telegraf's --test mode now runs processors and aggregators before printing metrics, and official packages are built with Go 1.14.5. The release also includes new outputs, such as New Relic Output Plugin and Execd Output Plugin, and updates to existing plugins like SNMP, HTTP, and OPC-UA input plugins.

InfluxData's time series database, InfluxDB, has been embedded in PTC's ThingWorx IoT platform, providing a "time series database for persisting device data" that was previously missing. This move comes shortly after InfluxDB Cloud added support for Microsoft Azure, highlighting the growing importance of cloud-based solutions for IoT applications. The OEM agreement between InfluxData and PTC is expected to benefit both companies by enhancing their respective IoT services with a robust time series database solution.

InfluxDB Cloud, a time series database as a service, is now available on Microsoft Azure, Google Cloud, and Amazon Web Services, expanding its reach to all major cloud platforms. This move aims to accelerate IoT application development with PTC's ThingWorx platform, enabling developers to collect, process, store, and analyze large amounts of data from tens of thousands of internet-connected devices. InfluxDB Cloud offers flexible and transparent usage-based pricing, a free tier, and has been built as a cloud-native application on Kubernetes, making it an ideal environment for customers to develop and run their applications. The deployment of InfluxDB Cloud on Microsoft Azure provides an ideal environment for customers to accelerate the roll-out of applications and benefit from unified billing and support, along with integrations with key Microsoft Azure technologies.

InfluxDB Endpoint Security State Template is a community-created template that helps verify the security state of public endpoints by continuously monitoring their authentication and SSL certificates. The template uses InfluxDB to track the state of web service authentication and SSL certificates, providing insights into service availability, certificate validity, and authentication functionality. It also enables operators to automate certificate renewal and detects potential security issues such as DoS attacks, authentication failures, or expired certificates. The template can be easily set up using the InfluxDB CLI environment and Telegraf configuration, allowing developers to quickly build solutions and improve their security posture.

The new release of Grafana 7.1 extends its built-in InfluxDB datasource to support both Flux language and InfluxQL queries, making it easy to connect Grafana to InfluxDB without a separate plugin. This update allows users to leverage the capabilities of Flux, which enables more powerful queries than InfluxQL, such as joins, math across measurements, and grouping by any column. The new feature also supports advanced analytics and data shaping using Flux's string package for sanitization and normalization, as well as work with geo-temporal data. Additionally, users can create multiple InfluxDB Data Sources in Grafana, one of which uses Flux, and another that uses InfluxQL. The integration is available today with the launch of Grafana 7.1 and supports various variants of Grafana, InfluxDB, and language combinations.

The output format of a Flux query with InfluxDB 2.0 is called Annotated CSV, which contains annotations that describe the data layout, including the datatype, default values, group keys, and field keys. Understanding these annotations is crucial to interpreting the Annotated CSV results, particularly when dealing with multiple filters or group keys. The #group annotation indicates whether a column is part of a group key, which can result in multiple tables being generated based on the filter criteria. InfluxDB users can export data as a CSV with the InfluxDB 2.0 UI, create temporary data with the from.csv() function, or write that data to InfluxDB by following from.csv() with to(). Additionally, users can use regular CSV writing methods such as writing CSV from a file with the CLI or using the Telegraf File Input Plugin.

In this article, Anais Dotis-Georgiou shares a way to generate temporary data with Flux in InfluxDB 2.0, allowing developers to create an initial or default state for one of the input tables needed for a task that requires joining two tables, even if not all data is available. To achieve this, users can navigate to the Data Explorer tab and use the query builder to return similar data from another table, then download it as an Annotated CSV file. The CSV is modified slightly to add new data points, which are then used with the Flux `csv.from()` function to generate a temporary table in InfluxDB 2.0. This workaround provides a solution until a more elegant feature is developed.

A new maintenance release for InfluxDB OSS (Open Source Software) is now available, addressing several fixes and improvements including optional configuration of custom HTTP response headers, performance enhancements such as parallel query planning and batching tombstone writes, and bug fixes for Flux, TSI, Storage, and other components. The binaries for the latest open source release can be found on the InfluxDB downloads page, and users are encouraged to join the InfluxDB Community Slack or post issues in the Github Repo or Community Site for assistance.

This tutorial uses the BIRCH (balanced iterative reducing and clustering using hierarchies) algorithm from scikit-learn with the ADTK (Anomaly Detection Tool Kit) package to detect anomalous CPU behavior in InfluxDB 2.0. The tutorial assumes that the user has InfluxDB and Telegraf installed and configured on their local machine to gather system stats. It uses the InfluxDB 2.0 Python Client to query data, transform it into a Pandas DataFrame, and then applies the ADTK MinClusterDetector function with the BIRCH algorithm to detect anomalies. The tutorial demonstrates how to visualize the detected anomalies using ADTK's plot function and provides insights on applying the ADTK package to continuous functions with InfluxDB.

This maintenance release of Chronograf includes several key fixes for various components of the system, including OAuth configurations, Windows support, dashboarding issues, and TICKscript editor scrolling problems. The update addresses a range of user-reported bugs and improves overall stability and functionality of the platform. For community members, the latest version can be downloaded from the official website.

A new release of InfluxDB 2.0 Beta is available now, with enhancements including improved templates for associating new resources with existing Stacks, an upgraded Flux Query Engine supporting additional services such as Telegram and AWS Athena. The beta build is not intended for production usage but invites users to provide feedback through the InfluxDB Community Slack.

The Median Absolute Deviation (MAD) algorithm is a powerful anomaly detection technique that can be used to identify time series data points that are behaving differently from others. MAD is a widely used algorithm in DevOps Monitoring, where it enables Site Reliability Engineers (SREs) to quickly identify unhealthy containers, VMs, or servers and diagnose infrastructure problems. The algorithm works by calculating the median absolute deviation of each time series at a given timestamp, and then flagging points that have a large deviation from the median as anomalous. MAD is highly effective and efficient, but can be sensitive to anomalies, especially when dealing with large datasets. To mitigate false positives, SREs can use techniques such as adjusting the threshold value or grouping data by region. The MAD algorithm has been implemented in Flux, a powerful open-source query language for InfluxDB, which allows users to write custom anomaly detection algorithms and integrate them into their monitoring pipelines. By using MAD, organizations can improve the efficiency of their root cause analysis efforts, reduce mean time to resolution (MTTR), and honor Service Level Objectives (SLOs).

WayKonect is utilizing InfluxDB Enterprise to improve fleet management by providing a driver-centric platform that enables fleet operators to track vehicles and drivers in real-time, improving efficiency and reducing costs. WayKonect's solution uses telematics data to provide insights on vehicle maintenance, fuel usage, and driver behavior, allowing fleet managers to optimize their operations and reduce their carbon footprint. The company chose InfluxDB Enterprise for its scalability, high availability, and schemaless design, which meet WayKonect's technical requirements for storing and analyzing large amounts of time series data from various sources. With InfluxDB, WayKonect can automate raw telemetry storage, provide a 360-degree view of the fleet, and adhere to data privacy laws such as GDPR and France's Loi Informatique et Libertés.

Supralog built an online incremental machine learning pipeline with InfluxDB OSS for capacity planning, using Kapacitor, Python, and InfluxDB. They used Kapacitor to monitor prediction residuals and trigger alerts, which would retrain the model if it drifted. The pipeline consists of three components: trend, seasonality, and residuals, each modeled by a different machine learning approach (linear regression for trend, LSTM for seasonality, and another LSTM for residuals). The pipeline uses InfluxDB's TICK Stack to store and query data, and Kapacitor to alert on model drift. The authors discuss the advantages of using InfluxDB and Kapacitor for this use case, including their ability to handle large amounts of data and provide real-time insights. They also highlight the importance of online machine learning and incremental training, which allows the model to adapt to changing conditions without requiring a complete retraining.

Geo-temporal Flux is a powerful tool for analyzing time series data with geolocation capabilities. It uses S2 Geometry library to derive geo-temporal powers, allowing users to efficiently analyze data at scale. Geo-temporal analysis requires only latitude and longitude, but tagging data with S2 Cell Identifiers (IDs) helps to improve efficiency. Calculating S2 Cell IDs can be done at query time or as the data is written to InfluxDB, but this takes time and affects cardinality. The earthquake Lambda calculates and tags earthquakes with level 9 S2 Cell IDs at ingest time, striking a balance between granularity and cardinality. Geo-temporal Flux examples include real-time analysis of USGS earthquake data, using boxes, circles, and general polygons to filter rows and count earthquakes within specific regions or distances from events. InfluxDB's emerging geo-temporal capabilities are explored further in Tim Hall's InfluxDays roadmap presentation and can be used in various versions of InfluxDB Cloud 2.0, InfluxDB Enterprise 1.8, and InfluxDB OSS 2.0.