June 2023 Summaries

InfluxDB 3.0 has been released alongside native support for SQL and InfluxQL, a SQL-like query language specifically designed to query time series data. The new version has built upon three open-source projects: Apache Arrow, Apache Parquet, and Apache DataFusion, providing improved performance and compatibility with existing applications and plugins. The v1 API is available for backward compatibility, but requires manual configuration of DBRP mappings in Serverless mode. Client libraries have been updated to support InfluxDB 3.0, including Python, Go, JavaScript, C#, Java, and others, allowing developers to leverage the full performance of Apache Arrow Flight while using InfluxQL-based queries. The author recommends starting with the new v3 client libraries and experimenting with them, as well as pointing existing InfluxQL applications at InfluxDB 3.0 to share experiences in the community.

InfluxDB has introduced a new version rewritten from scratch to improve performance across the board, requiring adjustments to applications to incorporate it. To facilitate internal migration from legacy versions of InfluxDB, Telegraf instances needed a simple way to send data to the new database without interruptions or issues. Adding a new `outputs.influxdb` entry to the Telegraf configuration file, referencing the new InfluxDB Cloud Dedicated cluster, made this possible. The process involves obtaining credentials by contacting InfluxData Support and creating a database and token using `influxctl`. These credentials are then used in the new output plugin configuration, allowing data to flow to both the existing legacy database and the new InfluxDB Cloud Dedicated database after restarting the Telegraf agent.

GitOps is a software development approach that emphasizes the use of Git for managing infrastructure and application configurations, extending the DevOps approach. It originated from Weaveworks' recognition of Git's strengths in building a reliable and scalable software delivery pipeline. The GitOps workflow involves creating an application repository, pushing changes to it using Git, and initiating automatic deployments based on deployment manifests. This approach offers benefits such as version control, consistency, automated deployment, and collaboration. However, it also has limitations like a learning curve, infrastructure restrictions, tooling challenges, and security concerns. GitOps differs from DevOps in its focus on using Git as the single source of truth for infrastructure and software code. Popular GitOps tools include Argo CD, FluxCD, GitLab CI/CD, Jenkins X, and Weave Flux. Organizations are adopting GitOps to manage Kubernetes clusters, microservices architectures, and infrastructure as code, ensuring consistency, collaboration, and scalability. By adopting GitOps, teams can ensure their infrastructure and applications are in a state where they can be easily monitored, tested, and deployed.

Converting timestamps to datetimes in Python is an essential skill for any developer working with time-related data. Timestamps represent specific points in time as numbers, typically counting seconds or milliseconds since a reference point called the epoch (January 1, 1970). In contrast, datetimes are human-readable representations of dates and times that encapsulate year, month, day, and other time elements. Python's datetime module provides functions like fromtimestamp() to convert timestamps to datetimes, which can be used for precise calculations, comparisons, and operations involving time. Understanding the difference between timestamps and datetimes is crucial for mastering temporal data in programming, and this tutorial demonstrates how to perform conversions using Python's built-in modules.

InfluxDB 3.0 is a scalable database that offers high performance for both data loading and querying, focusing on time series use cases. Its system architecture consists of four major components: Data Ingestion, Data Querying, Data Compaction, and Garbage Collection. Each component operates independently, with the Data Ingestion responsible for ingesting data, Data Querying handling queries, Data Compaction compacting small files into larger ones, and Garbage Collection managing data retention and space reclamation. The database uses a catalog to manage metadata and object storage for actual data. InfluxDB 3.0 is designed to be scalable, with the ability to set up multiple dedicated clusters, utilize Kubernetes clusters, and offer infrastructure upgrades through in-place updates and complete Blue/Green rollouts.

MQTT, or Message Queuing Telemetry Transport, is a messaging protocol designed to efficiently transmit data between devices and applications in IoT systems. It operates on the client-server model, where clients communicate with a central server, the MQTT broker, which routes messages based on topic structure. Key concepts include topics, quality of service (QoS) levels, retained messages, last will and testament, and session persistence. Node-RED is an open-source visual programming tool that allows users to create flows by wiring together nodes to perform tasks visually and intuitively. MQTT nodes in Node-RED facilitate communication using the MQTT protocol, enabling interaction with MQTT brokers, publishing messages to topics, and subscribing to topics to receive messages. By combining MQTT with Node-RED, developers can build practical applications in IoT, automation, real-time data processing, home automation, industrial IoT, environmental monitoring, energy management, agriculture, security, and surveillance.

The tutorial explains how to query InfluxDB Cloud with the Apache Arrow Flight SQL interface using C++. The process involves setting up an environment, importing required classes, gathering authentication credentials and a query, defining a main class, connecting to the Arrow client, setting up the Flight SQL Client, executing a query, fetching results, and reading them into an Arrow Table. The tutorial assumes that the user already has a free InfluxDB Cloud account and Docker running on their machine. It provides a code walkthrough and references additional documentation for further learning.

In this example, a reliable data pipeline is configured using InfluxDB and Amazon Web Services (AWS) to collect time-stamped data from industrial equipment. The pipeline uses Telegraf as an open-source data-collection agent to read data from AWS IoT Core's Kinesis stream, which stores the data in InfluxDB. To ensure scalability and flexibility, the setup includes a DynamoDB checkpoint table for storing intermediate results. The configuration also utilizes Docker containers with InfluxDB and Telegraf instances running on an EC2 instance, allowing for easy deployment and management of the pipeline. By leveraging AWS services like IoT Core, Kinesis, and Lambda, and open-source tools like Telegraf and InfluxDB, this setup enables efficient data processing and analysis for industrial operations, providing valuable insights into machinery performance and enabling predictive maintenance.

InfluxDB 3.0 now supports connecting to Tableau for data visualization using the Apache Arrow Flight SQL JDBC driver, enabling faster data transfers, real-time visualizations, and seamless integration between the two tools. The connection allows users to unlock real-time insights, streamline workflows, and create compelling visualizations and reports. InfluxDB Cloud Serverless and Dedicated users can access this feature on Tableau Desktop as of publication, with additional products expected to support the JDBC driver in the future.

Telegraf, an open-source plugin-driven agent for collecting, processing, aggregating, and writing time series data, has introduced a new feature in v1.27 that enables users to filter metrics using the Common Expression Language (CEL). This allows users to reference specific components of a metric, such as its name, tags, fields, or timestamp values, to determine which metrics should be passed on or dropped. The CEL language provides a set of semantics for expression evaluation and supports logical operators, numeric operations, string operations, and time operations, making it easier for users to filter metrics with specific names, tags, fields, or timestamp values. This new feature aims to improve the performance and flexibility of Telegraf's metric filtering capabilities.

Zipkin is an open source distributed tracing system that provides a robust set of features for monitoring and troubleshooting complex systems. It allows developers to trace the path of a request as it passes through a system, making it easier to identify bottlenecks and errors. Zipkin supports multiple languages and platforms, integrates with other monitoring systems, and provides customizable sampling rates, service graph visualization, annotations and tags, easy deployment and configuration, and various use cases such as troubleshooting and debugging, performance monitoring, dependency analysis, and capacity planning. By installing and configuring Zipkin, developers can gain a comprehensive understanding of their system's performance and make data-driven decisions to improve efficiency and reliability.

Home Assistant is a software that allows users to control and automate home devices, such as cameras, thermostats, lights, and sensors. Docker, on the other hand, is a platform that enables users to create, deploy, and run applications like Home Assistant in containers. Docker makes it easy to set up software without issues related to host system incompatibility. To get Home Assistant running on Docker, one can use the host system's CLI by writing code. The process involves downloading the latest stable release of the Home Assistant container image from Docker Hub, installing and starting the container, opening the Docker app to confirm its status, and finally, accessing the Home Assistant interface. This setup provides an isolated environment for Home Assistant, allowing users to manage it securely and efficiently.

Mermaid.js is a JavaScript-based tool for creating visually appealing diagrams and charts in an interactive way, making it easier to explain complex data interactions, project management, and visualization. It offers various types of diagrams, including sequence diagrams, flowcharts, Gantt charts, pie charts, and more, which can be customized with different styles and themes. With Mermaid.js, users can create engaging charts and dashboards for websites and web applications, saving time while rendering interactive graphs, and it is highly adaptable for programmers and non-programmers alike. The tool also allows for styling and theme configurations, making it a valuable resource for data specialists, project managers, and front-end programmers.

InfluxDB 3.0 is a versatile time series database built on top of the Apache ecosystem, offering two cloud-based versions: InfluxDB Cloud Serverless, and InfluxDB Cloud Dedicated. The client libraries for InfluxDB Cloud are being developed to provide unified methods that wrap up the implementation of both SQL and InfluxQL, allowing developers to query with either language. Currently, client libraries exist for C#, Go, Python, Java, and JavaScript, while development is underway for the latter two languages. Additionally, resources such as documentation, tutorials, and community sites are available to help users get started querying and writing data to InfluxDB Cloud.

Telegraf is an open source plugin-driven agent for collecting, processing, aggregating, and writing time series data, allowing users to store sensitive data such as secrets securely using various secret store plugins including Docker, HTTP, JOSE, and OS-specific stores. The new Secret Store feature enables users to reference these secrets in their Telegraf configuration files without exposing them directly, alleviating the need for hard-coded credentials. To get started with Secret Stores, users must first verify plugin support, choose a secret store, add it to their Telegraf config, set up necessary configuration options, save secrets to that store, and update their config to use secrets, often using syntax such as @{secretstore_id:secret_key} to reference the stored secrets.

Node-RED functions are custom JavaScript code nodes in the Node-RED flow-based visual programming tool that enable users to tackle complex data transformations, execute personalized logic, and integrate disparate components within their flows. These functions allow users to write custom JavaScript code to process and manipulate message data, making it possible to perform tasks such as data transformation, filtering, validation, conditional logic, mathematical operations, text processing, date and time manipulation, error handling, dynamic content generation, and multi-step processing. By using Node-RED functions, users can transcend the boundaries of pre-built solutions and infuse their flows with custom logic, data mastery, and creativity, ultimately empowering them to achieve more complex automation and integration projects.

Python is a versatile programming language used widely for various tasks, including manipulating and comparing dates. This comprehensive tutorial guides users through different techniques for comparing dates in Python, covering topics such as comparing dates to today, comparing two dates without time, converting date strings to Python date objects, and more. By understanding how to compare dates, developers can handle various scenarios where they need to determine the relationship between a date and the current date in their applications. The tutorial demonstrates how to use Python's datetime module, including importing the date class, creating sample dates, comparing dates to today, extracting dates from DateTime objects, converting DateTime strings to Python date objects, calculating DateTime differences, comparing timestamps, and more. By mastering these techniques, developers can effectively manipulate and compare dates in their Python projects.

Automating manual processes with new technologies can provide low-risk opportunities for trying out solutions, allowing organizations to get used to them before expanding to other areas. Breaking change in legacy technology is often outside an organization's control, but open source tools' interoperability, extensibility, and flexibility can help mitigate this issue by adding new functionality and capabilities that extend the value of industrial data. Upgrading equipment often involves replacing or updating machines with different sensors or capabilities, which may not be compatible with legacy data historians, making it a high-risk endeavor. However, introducing a system like InfluxDB with broad interoperability and capabilities can help take advantage of new developments with a light lift, reducing the risk of upgrading an IIoT/OT stack. As organizations grow, they have opportunities to implement new technologies while testing and iterating on each aspect of the process, ultimately leading to significant improvements in industrial processes by using solutions like InfluxDB that enable iterative deployment and scaling at the pace that makes sense for their organization.

A new feature-bearing release for Telegraf is now available, which includes several key new features such as Common Expression Metric Filtering, Configuration File Parameter Expansion, and Netflow sFlow decoder. The SQL Input Plugin has also been updated to support FlightSQL, allowing users to query against the new InfluxDB v3. Additionally, new plugins have been added, including ctrlX Data Layer Input, Intel Baseband Accelerator Input, Clarify Output, Nebius Cloud Monitoring Output, and others. Important changes include improved time parsing with abbreviated time zones and deprecation removals for settings and plugins deprecated before v1.0. The latest Telegraf release binaries can be found on the InfluxDB Downloads page.

The modern factory's relationship with data is undergoing a significant change, shifting from a focus on the past to a future-driven approach. This requires an update in technology, particularly in legacy data historians that lack the necessary tools and interoperability for deriving meaningful insights. Successful predictive maintenance strategies and impactful data-driven decisions rely on real-time data processing at nanosecond precision, enabling advanced statistical analysis, machine learning models, and AI-derived insight. Amazon Web Services (AWS) and InfluxDB offer a suite of tools to modernize the Industrial IoT (IIoT) environment, including Telegraf for collecting industrial sensor data and InfluxDB for storing, visualizing, and analyzing large datasets. The integration of AWS Greengrass with InfluxDB enables real-time data processing and analytics, while Edge Data Replication (EDR) allows operators to collect, visualize, manage, and store data on their own terms. This technology enables users to connect to virtually any data source, process, analyze that data, and create visualizations to derive better decision-making and improved outcomes.

InfluxDB Cloud Dedicated is a hosted and managed InfluxDB Cloud cluster dedicated to a single tenant, designed to handle high write and query loads. It uses the Java Flight SQL client, part of Apache Arrow Flight, for efficient data transmission over gRPC. The tutorial assumes an existing InfluxDB Cloud Dedicated account and provides step-by-step instructions on reading data from the cloud using the Java Flight SQL client, including creating a source database, target database, source token, target token, and downsampling query. The client injects header metadata in every request and sets up authentication using a BearerCredentialWriter. After successful authentication, it executes a query, retrieves the stream data of the query from the endpoint, creates an InfluxDB WriteApi function, and writes the prepared point into the target database. The tutorial also covers visualizing the data with Grafana and querying the data with SQL using the Flight SQL protocol.

InfluxDB Cloud 3.0 is a versatile time series database built on top of the Apache ecosystem, allowing users to query it with the Apache Arrow Flight SQL interface for efficient working with time series data. A tutorial guides users through querying InfluxDB Cloud using the Go Flight SQL Client, which provides high-performance data services and efficiently transmits large datasets over gRPC. The tutorial assumes a free InfluxDB Cloud account and Docker setup, requiring authentication environment variables such as a token and organization to be set up before executing queries on the server. With this guide, users can explore querying InfluxDB Cloud with Flight SQL for their own projects, taking advantage of features like downsampling and data processing with tools of their choice.

The concept of observability focuses on collecting data from all parts of the system to provide a unified view of software at large. Modern software systems rely on fault tolerance, no single point of failure and redundancy, but errors still occur due to complexity in distributed architectures. Traditional monitoring, which focuses on individual areas of the system, is limited in its ability to track issues across a distributed system. Observability, on the other hand, aims to provide a unified view by collecting data from all parts of the system. It consists of three pillars: metrics, logs and traces. Metrics paint the overall picture of a software system, while logs record events and traces track end-to-end behavior of requests as they move through the distributed system. OpenTelemetry is an open-source framework that simplifies the collection of telemetry data, allowing developers to access a wider set of options for analyzing their logs, metrics and traces. InfluxDB 3.0 is a purpose-built time series database that stores metrics, logs and traces in a single database, providing real-time analytics, unlimited cardinality and fast querying capabilities.

Home Assistant is an open-source home automation platform that allows users to control and automate smart home devices from a single interface. It provides a powerful and flexible solution to manage smart home devices and services, with over 2,447 built-in integrations covering various devices and services. Home Assistant enables users to create custom automation depending on their needs, integrate third-party services, and use community-developed add-ons to extend functionalities. The platform can be installed on different devices and operating systems, including Raspberry Pi, macOS, Windows, and Linux. It also offers a range of integrations with popular smart home devices and platforms, such as Z-Wave, HomeKit, Google Assistant, MQTT, Philips Hue, Sonos, Google Nest, and Tasmota, allowing users to control and monitor their devices from a single interface. By using Home Assistant and InfluxDB, users can collect, persist, and visualize data from their smart home devices over time, helping them optimize their home automation setup and gain more control and visibility into their smart home devices and services.

In this article, an IoT app is developed to monitor a house plant, utilizing sensors and advanced software systems to track the plant's health metrics over time. The project uses InfluxDB, a time series platform that specializes in storing sequential data as it appears over time, allowing for comparisons, alerts, and monitoring of both physical and virtual worlds. A Particle Boron microcontroller is used to collect data from sensors, which are then sent to InfluxDB via an API. The data is visualized using Plotly's graphing library, providing a dashboard for the user to monitor their plant's health. The project also demonstrates how to query data from InfluxDB using Python and Pandas DataFrames, enabling users to retrieve specific data points and create custom graphs.

MAJiK Systems has developed software that utilizes data from manufacturing machines to provide AI-driven insights for improving factory operations. The company's Visual Factory tools use machine data to power predictive analytics, but they faced challenges with securely transferring data from industrial computers (PLCs) to the cloud-based predictive modeling solutions. MAJiK turned to InfluxDB to manage the large volume of time series data required for these advanced analytics models. With InfluxDB and its edge-to-cloud gateway solution, MAJiK can now securely transfer data, manage observability, and train machine learning models in the cloud. The integration has enabled customers to see significant reductions in downtime and waste, with an average of 35-45% reduction in downtime and 7-10% reduction in waste.