December 2022 Summaries

Apache Superset is an open-source data exploration and visualization platform that provides a rich visualization library with over 50 types of visualizations, including time series use cases such as bar charts, line charts, scatter plots, histograms, and calendar heatmaps. It supports a wide range of databases through SQLAlchemy and provides direct file ingest for various formats like JSON, CSV, Excel, and columnar files. The platform is based on a Dataset-Centric methodology that promotes the use of datasets with labels, descriptions, metrics, timezone support, and definitions for filtering and aggregation. Apache Superset features a cloud-native design, scalability, high availability, and deployment support for distributed architectures, making it suitable for various use cases. It also allows users to create custom visualizations via its visualization plugins feature and provides a low-code interface for selecting and creating visualizations from datasets.

Red Hat, a global leader in open source enterprise IT solutions, faced the challenge of managing its massive enterprise IT infrastructure, which involves monitoring over 40,000 employees across forty different countries. The company's internal network monitoring team aimed to build a single source of truth for network performance and observability by collecting data from thousands of devices and interfaces worldwide. To achieve this, they adopted InfluxDB as their critical piece in the network monitoring architecture, utilizing Telegraf and gNMI plugins to collect data directly from network devices whenever possible. The system enriches data, detects issues, sends alerts, and stores analyzed data in InfluxDB for further analysis and visualization. This solution relies on Ansible for automation and requires relatively little manual intervention, allowing support engineers to focus on critical issues rather than managing individual devices.

InfluxDB is a key platform for network monitoring, addressing critical aspects such as data collection, storage, and analysis. The platform's capabilities include easy data collection with its open-source agent Telegraf, which has hundreds of plugins to collect data from various sources, and scalable data storage through InfluxDB Cloud. Additionally, InfluxDB provides tools for efficient data analysis, including a visual query builder that enables non-technical users to build dashboards and gain insights. The platform's support for SQL also extends its usability and flexibility, allowing seasoned SQL users to leverage their expertise in querying data. Many companies, such as Cisco and Red Hat, already use InfluxDB to power network monitoring solutions.

InfluxDB is an open-source time series database optimized for fast and highly available data storage, while Grafana is a popular data visualization and monitoring platform used to create dashboards and visualize data from various sources. The combination of InfluxDB as the data store and Grafana for visualization enables users to efficiently query and display their time series data. To set up this combination, users can install Docker Desktop, navigate to a specific directory, run a script, and start the containers using `docker-compose`. After setting up credentials, connecting Grafana to InfluxDB is achieved by adding a data source in the Grafana UI. Users can then create their first Flux query using Grafana Explorer, visualize financial data with more advanced queries, and create dashboards to display their time series data. Additionally, users can import CSV files into InfluxDB, use Flux to perform advanced manipulation and transformation of data, join data from multiple sources, and enrich time series data with metadata from relational databases or other sources.

InfluxDB Cloud is being used by Rune Labs, a software and data analytics company for precision neurology, to collect sensor data from various sources including mobile apps, sensors implanted inside patients, and wearables. The platform collects real-time data that helps patients better understand their condition and improves data-driven healthcare decisions. InfluxDB is used to store the raw data collected from these sources, clean it, normalize it, and provide pre-built visualizations for clinicians to view patient data. Rune Labs takes security and privacy seriously by storing sensitive information in a separate database and using InfluxDB's profiling capabilities to ensure performant queries. The company aims to expand its platform to help patients analyze their own data and identify biomarkers of Multiple Sclerosis lesions in patients with spinal cord stimulators, utilizing InfluxDB's features to make it more scalable and efficient.

InfluxDB Cloud now supports SQL querying for time series data, leveraging the Apache Arrow DataFusion project's query optimizations. This allows users to leverage PostgreSQL-compatible SQL implementation, providing compatibility with various databases and enabling more efficient querying of large datasets. The new feature offers advantages such as improved performance, increased flexibility, and better support for complex queries. Users can migrate existing SQL queries from other PostgreSQL-compatible databases, take advantage of the extremely performant columnar database, query all data without specifying a time range, and use SQL to write basic Flux queries using equivalent SQL queries. The new Script Editor in InfluxDB Cloud enables users to save, open, create, and visualize scripts, making it easier to work with SQL queries. By utilizing the iox.from() function, users can return output that looks similar to SQL queries, allowing for more flexibility in querying data. The inclusion of SQL as a query language excites developers, offering an alternative to Flux and providing improved performance and support for complex queries.

The use of Industrial IoT (IIoT) is heavily reliant on time series data, but challenges such as navigating the divide between physical and digital realms, remote visibility, and transferring domain expertise to new generations of manufacturing professionals hinder its adoption. Legacy industrial process historians have limitations in terms of cost, complexity, and interoperability with other systems, which can limit innovation and optimization opportunities. However, embracing technology like InfluxDB can provide a more proactive and efficient environment by facilitating data collection, transformation, and automation, enabling better results through failure detection, predictive maintenance, and optimization. By combining InfluxDB with other open source software and processes, businesses can unlock significant potential in their IIoT tech stack.

InfluxDB University has made significant progress this year, formalizing its training offerings under one umbrella and expanding into new areas. The platform now provides on-demand courses with a range of interactive content to help users understand InfluxDB, Telegraf, and Flux language. Successful completion of the verified courses earns digital badges that can be shared on professional profiles, demonstrating time series skills and generating career opportunities. The initiative has also returned to in-person learning with an advanced training session in London and plans for future Road Shows in America and Europe. With over 5,000 e-learners registered and 1,000+ badges issued, InfluxDB University is poised for further growth and expansion into new areas such as Spanish language training, use case-specific courses, and the InfluxDB Professional Certification.

InfluxDB is a time series database that handles large volumes of data in near real-time, with an immutable nature that requires careful deletion and editing to maintain accuracy. Time series data updates continuously due to the constant movement of time, making it essential to consider precision levels and intervals for calculations like averages. Downsampling data can be intentional to preserve valuable information but risks losing context if not handled properly; InfluxDB supports downsampling with multiple backup copies and nanosecond precision to maintain context. Additionally, InfluxDB allows joining non-time series data with time series data to produce deeper insights.

InfluxDB, the time series database platform, supports three different query languages: Flux, InfluxQL, and SQL. Each language has its own capabilities and ease of use, with Flux being the most powerful but also having a steeper learning curve. InfluxQL is an extension of SQL for time series data and is available across all platforms, while SQL support is currently only available in the Cloud as part of the IOx data engine. The platform is also introducing tasks, which are scripts that run on a schedule, allowing users to downsample data, perform transformations, or trigger alerts. Additionally, InfluxDB is developing a new script editor for InfluxDB Cloud, offering features such as visual building, script editing, and advanced scripting capabilities.

The new InfluxDB engine is built on top of Rust, Apache Arrow, DataFusion, and Parquet. These technologies help achieve the requirements for the new InfluxDB engine, including no limits on cardinality, best-in-class performance on analytics queries, separate compute from storage, operator control over memory usage, bulk data export and import, and broader ecosystem compatibility. Rust is chosen due to its exceptional performance and reliability, while Apache Arrow provides efficient columnar data exchange and interoperability with emerging standards in the data and analytics ecosystem. DataFusion is an extensible query execution framework that uses Apache Arrow as its in-memory format, providing best-in-class performance on analytics queries. Parquet is a column-oriented durable file format that enables bulk data export and import, and facilitates data storage at the edge. InfluxData's commitment to open source is reflected in their contributions to upstream projects like DataFusion and Arrow, demonstrating a symbiotic relationship between code and community engagement.

Flux is a functional query and scripting language for InfluxDB that provides more functionality than its predecessor, InfluxQL, allowing users to build data pipelines to query, analyze, transform, and act on their data. Flux offers advantages such as processing data server-side, which provides better performance over using a client library, and allows users to perform tasks without requiring a third-party application. It also enables users to manage and process their time series data with packages like the Math package for math applications, Geopackage for geotemporal data, SQL package to query and write data to and from SQL databases, HTTP package for making get and post requests, JSON package for working with and parsing json. In contrast, Python is a popular language that offers over 137,000 libraries for time series tasks like data preparation, cleaning, analysis, forecasting, classification, anomaly detection, but it's not the obvious choice when it comes to performing complex data processing, forecasting, or anomaly detection.

Native Collectors, introduced by InfluxData, enable one-step integration of data from data brokers like HiveMQ MQTT broker into its InfluxDB Cloud time series database, bypassing multiple data hops. This facilitates a simplified IoT architecture, reduces latency, enforces interoperability through standardization, and reduces costs. Native MQTT integration also provides a scalable IoT solution, ensures data integrity, offers consistent data parsing, and improves visibility in monitoring or troubleshooting IoT systems. By eliminating intermediary services, it simplifies the development and maintenance of IoT solutions, allowing developers to focus on building and scaling applications without additional software or new code concerns.

The author of the article runs a blog and other resources on Kubernetes, specifically in a K3s cluster. They use Telegraf and InfluxDB to monitor everything in their cluster. The author uses two components: InfluxDB Cloud for external monitoring and a Helm chart from Telegraf, modified to adapt to K3s. The author configures InfluxDB Cloud by creating an account, creating a bucket, generating an access token, and specifying the Org ID and URL. They then modify the values.yaml file to use InfluxDB Cloud as the output for Telegraf. The author installs the Helm chart using Helm and updates the repo. After installation, they check that everything is running properly by checking the pod status and tailing the logs. Finally, they verify that data is being written to InfluxDB Cloud by navigating to the Explore section.

Prescient Devices has built an industrial Internet of Things (IoT) data management platform powered by InfluxDB, which helps companies build applications to manage edge devices quickly and easily. The platform, Prescient Designer, uses a low-code visual interface based on Node-RED, allowing users to create applications without extensive coding knowledge. It enables companies to handle the life cycle of time series data from real-time analysis to downsampling historical data, and supports custom brokers for compliance requirements such as HIPAA or GDPR. The platform also leverages Grafana dashboards for real-time data visualization, saving companies time by allowing them to build applications quickly and update them rapidly, with changes deployable within hours to seconds.

In this article, Raphael Taylor-Davies and Andrew Lamb explain several advanced techniques for querying data stored in Apache Parquet files quickly. They implemented these techniques in the Apache Arrow Rust Parquet reader, which is one of the fastest implementations for querying Parquet files on local disk or remote object storage. The authors discuss various optimization techniques such as vectorized decoding, streaming decode, dictionary preservation, projection pushdown, predicate pushdown, row group pruning, page pruning, and late materialization. They also explain how to use these techniques to optimize I/O and CPU usage when reading Parquet files from commodity blob storage systems like AWS S3. The authors conclude that while implementing these optimizations requires significant engineering effort, the benefits can be substantial, especially for large-scale data analytics workloads.

At InfluxData, they prioritize building a platform for developers, by developers. They aim to meet the diverse needs of their users, considering factors such as programming languages, time series data comfort levels, and application environments. To achieve this, they provide a range of tools and resources, including client libraries in multiple languages, various approaches for data collection, and support across different cloud providers. The company collects direct feedback from developers to inform product improvements and continues to evolve its platform through trial-and-error processes, leveraging open-source principles to engage with a broader community of users.

The InfluxDB Cloud team has released a new query experience with a revamped interface that aims to improve user comfort and productivity. The new editor includes a schema browser, allowing users to select data directly from the hierarchical view of their schema, and auto-generated Flux code is always displayed to the user. This change enables users to start building queries more easily, without having to toggle between different windows. The new interface also provides an instructional interface to help users learn the InfluxDB data model, a load/save feature for saving and retrieving queries, and improved performance handling large datasets. Additionally, the new editor includes features such as search/filter result table, time range selection, query statistics, and multi-language support, with plans for future additions like SQL support and aggregations.

InfluxDB Cloud is designed to run natively on AWS, reducing data latency for users who rely on AWS and making it easier to get started. InfluxDB has integrations with various AWS services, including CloudWatch, allowing users to track system performance and monitor metrics from multiple sources. Additionally, plugins like the ECS Metrics Plugin and Kinesis Consumer Plugin enable monitoring of EC2 virtual machines, ECS containers, Fargate containers, and Kinesis data, respectively. Other AWS services like EKS, Lambda, and CloudWatch can also be integrated with InfluxDB using various plugins, providing users with a comprehensive view of their system performance and data.

This tutorial describes how to install the Telegraf plugin as a data-collection interface with InfluxDB 1.7 and Docker. It covers preparing Telegraf for InfluxDB and Docker, creating a configuration file for Telegraf and Docker, modifying the Telegraf configuration file, running the Telegraf container on Docker, isolating the Telegraf user ID, and ensuring correct permissions. The tutorial also explains how to use Grafana as a dashboarding tool to visualize metrics collected by Telegraf.