June 2022 Summaries

** InfluxData's time series platform, InfluxDB, has been named a leader in the G2 Grid for Time Series Databases and within the inaugural Momentum Grid Report for Time Series Databases, as both platforms held the top position. This recognition is based on real customer reviews and experiences with InfluxDB, which earned high scores in ease of setup and user satisfaction categories. As an industry-leading time series platform, InfluxDB enables developers to quickly build and integrate time-stamped data applications, and scale massive data workloads within those applications. With features such as high performance, storage compression, and sophisticated query and analytics capabilities, InfluxDB is used by startups to Fortune 500 enterprises building IoT applications with real-time data analysis.

InfluxData has released the second edition of a blog series covering its latest features, including Edge Data Replication, which unifies time series data processing between edge and cloud environments. This solution allows developers to collect, store, and analyze high-precision time series data from the edge and view it in the cloud. InfluxDB Edge Data Replication gives developers a fast and durable way to stream edge data to the cloud with minimal configuration, enabling them to integrate edge and cloud workloads, reduce network costs, transform edge data, enforce edge-cloud consistency, and more. Additionally, new sample apps for Node.js and IoT have been released, as well as improvements in API token management, bug fixes, and other features that enhance developer productivity and user experience.

InfluxDB Cloud and InfluxDB provide businesses with the ability to give their edge devices autonomy to collect, process, and act on data locally, while also maintaining centralized storage and analysis in the cloud. Data replication helps users get the data they need, where they need it, by automatically sending data from the edge to the cloud for real-time processing and analysis. This method is automatic, durable, and reliable, allowing businesses to tailor their data for different user needs, transform data before sending it to the cloud, and reduce storage costs.

The authors of the text have created a hybrid architecture combining MQTT and Kafka with InfluxDB for IoT data processing. The architecture uses an MQTT broker as a bridge between devices and a Kafka cluster, while InfluxDB is used to store and process data. The MQTT proxy acts as a secret MQTT client that connects to the current broker, mapping payload to a Kafka topic. The authors have implemented a Docker-based setup with environment variables for configuration. They have also created a Flux task to aggregate and enrich data before transferring it back into a Kafka topic for further consumption. The architecture demonstrates a decoupled approach to data transformation without modifying other stages of the architecture.

InfluxData's customer VLEEMO is a pioneer in wind energy, leveraging InfluxDB to optimize performance and meet regulatory requirements. Operating 38 turbines across Belgium's harbor generates enough power for over 90,000 households, while ensuring efficient operation within environmental constraints such as shadow flicker and icy conditions. By utilizing time series data and real-time insights, VLEEMO can predict and prevent issues, maximize energy production, and promote sustainable practices, supporting Antwerp's efforts to drive a greener world.

The incident began with a single line of code that was added to a configuration file, which caused ArgoCD to incorrectly apply changes to production, resulting in the loss of a core workload and additional workload. The team responded by reviewing code, creating a recovery plan, and restoring state and data, including etcd, Kafka, and storage engine. They also improved their process for handling public-facing incidents and implemented changes to prevent similar errors in the future, such as using custom annotations to prevent deletion of resources and improving tooling to detect duplicates when generating YAML files. The incident highlighted the importance of having a well-planned disaster recovery strategy and effective monitoring and alerting systems.

This tutorial teaches developers how to efficiently visualize time series data using InfluxDB, a leading high-performance time series database, and Matplotlib, a popular plotting library for Python. The author guides the reader through setting up an InfluxDB Cloud account, creating a bucket, installing required packages, and writing data into the bucket using a Python client. The tutorial then shows how to use Flux language to query the data from the bucket and pivot it into a dataframe for analysis. Finally, the reader learns how to normalize the dataframe to create professional charts and visualize the data with Matplotlib. Throughout the tutorial, the author provides examples of various types of visualizations, including line plots, scatter plots, bar charts, pie charts, and radar plots. The tutorial concludes by analyzing the stock prices over time and discussing options for more in-depth analysis and forecasting methodologies.

The InfluxData team is attending the Data + AI Summit in San Francisco from June 27-30, where they will be showcasing their platform for working with time series data and providing expert advice to attendees. The company has a booth at the conference where experts will answer questions about InfluxDB, and there will also be demonstrations and talks by InfluxData engineers and developers. One of the notable presentations is on "DataFusion and Arrow: Supercharge your Data Analytical Tool with a Rusty Query Engine" by Andrew Lamb and Daniel Heres, which highlights how Rust can power data stacks, while another talk by Zoe Steinkamp will delve into Flux and Edge Data Replication. The InfluxData team looks forward to meeting attendees face-to-face or virtually and providing swag as incentives for stopping by their booth.

InfluxDB has released new improvements, including Flux advancements and bug fixes, as part of its latest general availability release. The open-source version now includes features such as vectorized arithmetic operators, performance enhancements, and added support for OpenTracing spans to track Flux scripts more precisely. Additionally, the release upgrades Go to 1.18.3 and refactors operator profiles in query statistics. Bug fixes have also been addressed, with users encouraged to report issues or ask questions through various channels such as InfluxDB's Community Slack, GitHub Repo, or Community Site.

The OpenTelemetry project aims to simplify the collection and management of telemetry data to enable developers to adopt observability best practices. It provides a standard vendor-agnostic interface, supports metrics, logs, and traces, and offers features such as auto-instrumentation, flexibility, and ease of use. The project has gained support from major tech companies like Microsoft, Google, Amazon, Red Hat, Cisco, and others, and is designed to work with cloud native applications and modern architectures. It allows developers to collect telemetry data without having to modify their codebase in many cases, making it easier to adopt observability best practices and improve the quality of software for everyone.

InfluxData has announced Edge Data Replication, a new feature that combines the deployment strategies of edge and cloud data workloads in one unified solution. This feature allows developers to safely replicate time series data from InfluxDB Open Source (OSS) buckets to InfluxDB Cloud buckets in real-time, enabling businesses to collect, store, and analyze high-precision time series data from the edge and view it in the cloud. The new feature builds upon two key properties of InfluxDB OSS: its ability to run efficiently on resource-constrained systems and its Flux task engine that can analyze and transform time series data. With Edge Data Replication, developers can now integrate edge and cloud workloads, reduce network costs, transform edge data, enforce edge-cloud consistency, and reimagined the TSDB architecture. This announcement is an early step in InfluxData's broader vision to support tomorrow's technologies, applications, and developer challenges with the InfluxDB platform.

InfluxData has announced Edge Data Replication, a new capability that combines the power of cloud with the precision of IoT edge to optimize time series data processing. This new feature enables developers to collect, store and analyze high-precision time series data in InfluxDB at the edge, while replicating all or subsets of this data into InfluxDB Cloud. Edge Data Replication is designed to address the challenges of managing large volumes of time series data across distributed environments, particularly in IoT applications where data is often generated at the edge. By leveraging Edge Data Replication, developers can integrate edge and cloud data operations, reduce cloud ingress and egress costs, transform data before transfer, enforce edge/cloud consistency, and support full data workflow at the edge. The feature is available immediately for InfluxDB users and is expected to simplify the process of building IoT applications with real-time insights and control.

Telegraf 1.23.0 has been released, offering improved features such as the generation of sample configuration files for plugins, enhanced authentication for HTTP output plugin with Google API support, and new inputs including Slab Input Plugin and Fritzbox Input Plugin. The release also includes contributions from multiple open source community members, including Home Depot and @bobuhiro11 and @hdecarne. Users can download the latest binaries on the Telegraf Downloads page and participate in testing bug fixes and features for plugins and systems through a call for testing.

InfluxData's customer, Bboxx, is a high-tech utility company that uses solar-powered smart batteries to provide clean energy to people in developing countries. The company aims to combat energy poverty by powering devices such as lights, radios, and televisions, connecting small business owners with their customers, extending farming capabilities, and empowering children to do schoolwork and read at night. Bboxx co-founder and CEO Mansoor Hamayun says that electricity access is critical to human progress and that his company's mission is to provide clean energy where it's needed. To optimize the performance of its products, Bboxx uses InfluxDB, a platform built for time series data applications, to build its operating system, Bboxx Pulse. The platform helps manage power in developing countries, where patchy internet signals and variable data rates can create challenges. By analyzing and managing time series data, Bboxx can quickly identify usage patterns and anomalies that help improve the product over time. As a result of using InfluxDB, Bboxx has made significant impact, providing clean energy to over 2 million people and replacing 1.93 million kerosene lamps, ultimately contributing to reducing CO2 emissions and empowering economic activities in developing countries.

The U.S. Army Combat Capabilities Development Command (DEVCOM) Army Research Laboratory (ARL) uses InfluxDB to handle network connection uncertainty and develop new technologies in an unpredictable environment. They process data both at the edge and in the cloud, ensuring seamless analysis and efficient use of resources. The flexibility of InfluxDB allows them to replicate their own system tailored to their unique needs, making it easy for teams to work together across different locations. By using InfluxDB, the ARL team is able to improve efficiency, minimize CPU consumption, and quickly create new systems, ultimately supporting U.S. Army modernization in the long term.

InfluxDB is a time series database that specializes in handling time series data, which is indexed by time. It's optimized for reading and writing data with less performance consideration for updating or deleting data. InfluxDB can be used to monitor application performance, store metrics from applications like request processing, memory consumption, and response time. To use InfluxDB with a Ruby on Rails application, Docker needs to be installed and set up, and the latest version of Ruby on Rails must be installed. An InfluxDB client is instantiated using the `influxdb-client` gem, and subscriptions are set up for various events such as process_action.action_controller, render_template.action_view, sql.active_record, and instantiation.active_record. Once instrumentation is set up, metrics can be generated by running the app, and data can be visualized in the InfluxDB dashboard using Flux queries.

Hulu has successfully scaled its internal application and infrastructure monitoring platform by combining two open source tools - InfluxDB, a time series database, and Kafka, an event-streaming platform. This solution enables Hulu to handle massive volumes of data being written in real-time, while also providing the ability to query that data in real-time for analysis. Similarly, companies like CERN and Robinhood are using InfluxDB and Kafka together to solve problems such as storing large amounts of raw data from experiments or aggregating metrics for anomaly detection. The combination of these two tools provides scalability, fault tolerance, and usability features that abstract away complexity, making it easier for developers to implement common time series workloads. Additionally, a number of tools have been created to make integrating InfluxDB and Kafka easier, such as connectors and plugins.

Hulu has successfully scaled its internal application and infrastructure monitoring platform by combining two open source tools: InfluxDB, a time series database optimized for massive volumes of data, and Kafka, an event-streaming platform that provides scalability, fault tolerance, and usability features. This combination allows Hulu to handle over 1 million metrics per second while providing real-time analytics queries with InfluxDB. Additionally, companies like CERN and Robinhood are also using InfluxDB and Kafka together to solve problems such as storing large amounts of data from experiments and powering anomaly detection platforms. The open source nature of these tools has created a strong ecosystem of tools and libraries that enhance their value beyond the core project provides.

A common debate in software development revolves around whether to use pre-existing tools and services (building) for better productivity or stick with lower-level tools or custom-built solutions (buying) for more control, flexibility, and potentially better performance. This decision can impact various aspects of tech industry ideological conflicts, such as cloud vs. on-premises hardware, programming languages, self-hosted vs. managed services, and internal custom tools vs. off-the-shelf software-as-a-service solutions. The choice between using a client library versus direct API access to developer tools is also a point of contention, with each side having valid arguments about trade-offs in terms of productivity, efficiency, flexibility, and control. Ultimately, the decision depends on individual company circumstances, including the importance of the software project to core business value and the company's position in terms of product-market fit.

Telegraf has reached 300+ plugins and is widely used in various use cases. The SNMP Input Plugin is one of the most popular, allowing users to collect metrics from devices running Simple Network Management Protocol (SNMP). To configure the plugin, users need to specify agent addresses, timeouts, SNMP version, community strings, and retries. Best practices include specifying fields directly rather than tables, dividing plugins for each device, and using metadata tags such as system name and location. Processor plugins like Enum and Starlark can be used to process and transform data, such as translating interface states or parsing storage metrics. By following these tips and tricks, users can effectively use the SNMP Input Plugin in Telegraf to collect and analyze network device data.