August 2019 Summaries

The Telegraf socket_listener plugin is a powerful tool for sending data from scripts and applications into InfluxDB using UDP or Unix sockets. It allows users to write data in the InfluxDB line protocol format, making it easy to integrate with existing infrastructure components. The plugin can be used for both quick prototyping and long-term solutions, providing a flexible way to collect data from various sources. With its simplicity and ease of use, the socket_listener plugin is an essential component for anyone looking to leverage Telegraf's capabilities in their projects.

InfluxDB Cloud is being used by Tricentis Flood, a cloud-based distributed load testing platform, to collect valuable metrics about the performance of applications being tested. The platform simulates real-world workloads to determine an application's ability to perform under anticipated user loads before it goes live, helping mitigate downtime risk and ensure readiness for production deployments. With InfluxDB Cloud, Tricentis Flood achieves its goals of being a distributed, loosely coupled, shared-nothing, load-test platform that processes data in real-time without delays, allowing teams to test, analyze, and improve application performance seamlessly across the DevOps pipeline.

In this example, a fictional fish tank health vendor uses InfluxDB 2.0 Alpha 15's SQL.to() functionality to write data from Flux results into a Postgres database for storing customer billing data. The vendor leverages Flux to collect and process data on corrective actions taken by automated water systems in their customers' tanks, which are stored in InfluxDB as time series events. However, the aggregated and processed data represents a "finished product" that is more suitable for storage with other customer data, such as billing address and telephone number, in a relational database like Postgres. The vendor creates a simple weekly task to count adjustments for each device and save them to their Postgres database using Flux, demonstrating the power and flexibility of multi-data source Flux.

InfluxDB OSS, Chronograf and Telegraf have released new maintenance updates with various bug fixes and improvements. InfluxDB 1.7.8 addresses issues such as panic when reading Prometheus metrics and incorrect field reappearances after dropping a measurement. Chronograf 1.7.14 introduces the ability to select between local time or UTC for data exploration, while resolving issues in the Flux script editor. Telegraf 1.11.5 includes improvements to its plugins, such as updated MySQL driver, improved error handling and support for parked process state on Linux.

Cardinality in databases refers to the complexity or density of data, with high cardinality indicating a large number of unique values or combinations of values. This concept is considered challenging due to its subjective nature and varying interpretations across different contexts. InfluxData DevRel Katy Farmer discusses how to recognize and address high cardinality in databases, highlighting factors such as data organization and relationships between data aspects that influence its existence. Cardinality serves as a measure of data complexity, enabling the understanding of relationships between different data components and facilitating the design of more stable systems.

The author of the text is David G. Simmons, who decided to calculate the Heat Index after collecting temperature and humidity data. The calculation involves a complex formula by NOAA, which includes several caveats and adjustments depending on the relative humidity and temperature values. The author implemented this calculation in Flux, a programming language developed by InfluxData, using a combination of table joins, aggregations, and mappings to create a single table with the calculated Heat Index. The author plans to add more environmental calculations, such as the Ideal Gas Law and Wind Chill, to a future package in Flux.

InfluxDB is a time-series database that can be used with the Python client library. Improving write performance is crucial for large-scale data import and transfer. Common pitfalls include generating random data during execution, which can lead to slow timing of individual parts of the program. The `time.perf_counter()` function provides high-resolution timing, allowing developers to measure the performance of writing data. Favoring Line Protocol over JSON can also improve write performance by avoiding unnecessary conversions. Additionally, batching points in batches of 5,000-10,000 points is crucial for minimizing network overhead and achieving optimal performance. By following these best practices, Python developers can optimize their InfluxDB write performance and ensure efficient data transfer.

Aquicore provides a cloud-based platform for commercial real estate (CRE) companies to track, analyze and predict building and energy data. The company's clients include major players such as Cushman & Wakefield, Salesforce.com, and The World Bank, and have access to their IoT-driven smart asset platform, which allows them to improve communication between executives, property managers, and building engineers. Aquicore's platform is used to improve 250M square feet of commercial real estate space in 700+ buildings, including the John Hancock Tower and the Salesforce Tower, and has helped its clients reduce energy consumption and optimize performance. The company recognized a need for modern technology to aid in automating billing, drive insights, optimize performance, and inform decision-making for property managers, building engineers, and accounting. They transitioned from using PostgreSQL to InfluxDB Cloud, which is purpose-built for time series data, allowing them to provide real-time visibility into their buildings and improve their competitive advantage.

The Customer Support Podcast features Manny Ruiz, VP success & support at InfluxData, in its 14th episode. Ruiz provides insights into his experience at InfluxData, covering customer growth and the evolution of customer support. He also discusses building his customer support team and using product usage metrics as a measure of success. The podcast offers a unique perspective on B2B customer support strategies and best practices.

The algorithm used in Giraffe, a React-based visualization library for InfluxDB Cloud 2.0, is the Ramer-Douglas-Peucker Algorithm, which reduces complex lines to fewer points while maintaining visual quality. This algorithm is useful for fast rendering of time series data, such as those found in InfluxDB and Chronograf, enabling quick analytics, real-time monitoring, and immediate action by reducing redundant points and computationally exhaustive rendering. The algorithm's effectiveness is demonstrated through examples, including interactive visualizations and beautiful Giraffe visualizations in InfluxDB Cloud 2.0, showcasing its ability to quickly render line plots, scatter plots, and other visualizations without sacrificing accuracy or detail.

Flux, an extensible language used by InfluxDB 2.0, was extended to write data from InfluxDB to MQTT using the `to()` function with an MQTT extension, allowing users to send data to an MQTT broker for IoT applications, with a custom configuration of options such as broker URL, topic, message, and quality of service. A task in the InfluxDB 2.0 UI can be used to create a Flux query that writes to an MQTT broker using this new functionality, which has been submitted as a pull request but is now part of the master branch, awaiting release. The MQTT output allows users to send data from InfluxDB to IoT devices for real-time monitoring and control, with options for message formatting and quality of service.

A well-written Go developer, Mark Rushakoff, shares his experiences on reproducing flaky tests in Go. Flaky tests are nondeterministic tests that fail occasionally due to various reasons such as data order, timing issues, or concurrency problems. To reproduce a flaky test, one must focus on the exact package and test that failed, use -run flags to isolate the failing test, increase the count of iterations with -count=1 to ensure multiple runs, and run the test in a loop until it reproduces within a minute or less. If still unable to reproduce, tools like data race detector, cpulimit for CPU throttling, and adjusting GOMAXPROCS can help identify the root cause of flakiness.

InfluxDB 2 is nearing its release and will ship with a compatibility mode for InfluxDB 1, allowing continued use of the client libraries. However, this mode won't be available yet, so developers can use Telegraf to proxy requests from their applications to InfluxDB 2 without making any code changes. Telegraf provides an input plugin called influxdb_listener that allows it to act as a proxy for InfluxDB 1, and users can configure this plugin to point to their InfluxDB 2 instance. The output plugins are also available, allowing users to write data to InfluxDB 2 without changing any code. Additionally, Telegraf can be used to combi-write to multiple InfluxDB instances, including InfluxDB 1, InfluxDB 2, and InfluxDB Cloud 2, all with no code changes required.

InfluxDB, Release Notes, August 14, 2019` A new alpha release of InfluxDB 2.0 is available, featuring enhancements including dashboard templates for Telegraf plugins, gzip compression in Flux query responses, and improved tasks API. The release also includes a flux library update to version 0.39.0. Users are encouraged to download the latest iteration, provide feedback, and report any issues on the InfluxDB Github Repo or Community Site.

InfluxData uses Kafka as a sophisticated Write-Ahead-Log (WAL) to provide a multi-tenanted, horizontally scalable time series storage in InfluxDB Cloud 2.0, overcoming the limitations of traditional WALs such as scalability and durability issues. The use of Kafka enables InfluxDB Cloud 2.0 to meet several key requirements including multi-tenancy, elasticity, efficiency, robustness, tolerance, and isolation, by providing a distributed and battle-hardened solution that is faster than traditional WALs. The architecture leverages complex data partitioning handled by Kafka, with each partition replicated across multiple storage nodes, ensuring durability and scalability. Additionally, the use of Kafka enables efficient writes, robustness against failures, and isolation between tenants.

Apache Kafka is an open-source stream-processing software platform that allows companies to build sophisticated data pipelines by providing a unified, high-throughput, low-latency platform for handling real-time data feeds. It offers several key benefits, including horizontal scalability, fault-tolerance, and speed. Kafka's architecture enables the decoupling of publishing and consuming messages, allowing it to store in-flight messages, making it a more scalable alternative to traditional publish-subscribe models. The platform also guarantees ordering and parallelism, making it suitable for real-time processing of data streams. Companies such as Hulu and Wayfair have successfully integrated Kafka with InfluxDB to create robust real-time data pipeline solutions that can scale to meet high-throughput metric ingestion requirements.

InfluxDB outperformed Splunk in data ingestion, on-disk storage, and query performance by a significant margin. InfluxDB demonstrated 17x greater average throughput while using nearly 17x less disk space compared to Splunk for the same data set. Additionally, InfluxDB delivered 25x faster response times for tested queries compared to Splunk. These results are valuable for engineers evaluating the suitability of both technologies for time series use cases involving custom monitoring and metrics collection, real-time analytics, IoT, sensor data, and container or virtualization infrastructure metrics. However, InfluxDB is not designed for full-text search or log management use cases, while Splunk was originally developed as a log management solution with metrics being introduced in version 7.0. Developers are encouraged to run the benchmarks themselves to verify the results on their hardware and data sets of choice.

Telegraf 1.11.4 has been released with improvements to several plugins, including Datadog Output, Kubernetes Input, and Socket Listener Input. The new release includes fixes for issues such as NaN or Inf values in Datadog data, a typo in the logsfs_available_bytes field in Kubernetes Input, and a reload panic in Socket Listener Input. The binaries can be found on the Telegraf downloads page.

InfluxData, the creator of time series database InfluxDB, has been recognized by Fast Company as one of its Best Workplaces for Innovators. The company was selected for its commitment to innovation and empowering employees to improve operations and develop new products. InfluxData's culture encourages experimentation, failure, and open discussions to foster creativity and drive innovation. The company hosts regular hackathons and product demos to enable employees to try new things and work with others. This approach has enabled the company to succeed in producing top time series technology for more than three years. Fast Company highlights workplaces that attract and retain top talent by creating environments where employees can put forth bold ideas, engage in radical experiments, and even fail in the name of innovation.