May 2023 Summaries

InfluxDB has developed an open solution for collecting, storing, and analyzing machine data, which is crucial for modern factories. The solution involves Kepware for data collection and InfluxDB for storage and analytics. Kepware is a scalable PLC, IIoT connectivity solution that connects to devices with different protocols, providing a single source for industrial data. InfluxDB is purpose-built to handle time series machine data at scale, allowing companies to achieve higher data resolutions in milliseconds or even nanoseconds. The solution also leverages edge and cloud computing, enabling faster reaction times and easier analysis of larger data sets. By using this open solution, companies can build data-driven insights that work for their business, such as monitoring vibration metrics on CNC machines and creating digital twins from highly granular machine data.

InfluxDB is an open-source time series database designed for storing and querying time series data, Grafana is a versatile analytics and monitoring platform offering powerful data visualization capabilities, Home Assistant is an open-source home automation platform that enables control and monitoring of various devices using a web interface or mobile app. InfluxDB integrates with Grafana to visualize and analyze Home Assistant data, allowing users to control and monitor their smart home devices in real-time while visualizing the data. To get started with this setup, users need to install and configure InfluxDB, install the InfluxDB integration in Home Assistant, send data from Home Assistant to InfluxDB, and then install Grafana using Docker or on a local machine, configuring it to connect to the InfluxDB database, creating a Grafana dashboard to visualize the data.

OpenTelemetry is an open-source tool used for adding observability to applications, providing logs, metrics, and traces about system behavior. It's technology-agnostic, allowing integration with any language, framework, or technology. To use OpenTelemetry with JavaScript, developers can follow a step-by-step process, including installing required dependencies, configuring the SDK, and exporting telemetry data to the console. The tool supports both push and pull models for collecting and processing telemetry data, providing flexibility for organizations to choose their strategy. With its large community of contributors, terrific documentation, and support from the Cloud Native Computing Foundation, OpenTelemetry is ready for production use, but requires rigorous testing and validation to ensure accurate data collection and processing.

InfluxDB 3.0 has improved storage compression and performance, supports unlimited cardinality data, and delivers fast SQL queries compared to previous versions, thanks to its new database engine built on top of Apache Arrow. The Flight SQL interface provides high-performance SQL access to InfluxDB databases using the Arrow Flight RPC framework, allowing for faster data access and lower latencies without converting data to Arrow format. To use InfluxDB with Flight SQL, a data source name (DSN) must be configured, which can be done using the usql CLI or programmatically, and requires installing Go 1.17+ and the FlightSQL Library. The DSN configuration includes parameters such as token, timeout, and TLS settings, which can be customized to fit specific connection requirements. Real-world examples of connecting to InfluxDB locally and in the cloud are provided, and the driver is available on the Apache Arrow GitHub page for more information.

Edge computing enables devices to function reliably even when network connectivity is unreliable, making it a crucial technology for critical applications that require high reliability and low latency. Various hardware options are available to power edge computing projects, including Nvidia Jetson chips designed for AI tasks on the edge, Particle devices with cellular connectivity, Google Coral chips powered by custom TPUs for machine learning workloads, Raspberry Pi boards known for their usability and community support, Orange Pi devices offering better performance at a lower price point, Odroid single-board computers with expandable RAM, and Intel NUC mini-PCs that consume relatively little power. Each of these options has its tradeoffs, and the choice depends on the specific project requirements, cost optimization, convenience, and development speed considerations.

Exponential smoothing is a time series forecasting method that uses an exponentially weighted average of past observations to predict future values, assigning more weight to recent observations and less to older ones. It's a widely used technique that can adapt to changing trends in the data, providing accurate predictions by giving different weights to different time periods based on their importance. Exponential smoothing is useful for short to medium-term forecasting of business metrics such as sales, revenue, and customer traffic, inventory management, finance, and marketing. The method has several types, including simple exponential smoothing, Holt's linear exponential smoothing, and Holt-Winters' exponential smoothing, each with its own formulae and assumptions. Python libraries such as Pandas, Statsmodels, and Prophet can be used for implementing different types of exponential smoothing methods.

The SQL BETWEEN operator is used to select rows that are within a specific range, inclusive of the start and end values. It can be used with numbers, text, and dates, and is commonly used in scenarios where data needs to be filtered by a specific time range or numeric value. The operator is usually placed after the WHERE clause and can be used to filter data outside a specified range using the NOT BETWEEN operator. Additionally, the BETWEEN operator can be combined with other operators like IN to achieve more complex filtering conditions.

OpenTelemetry is an open-source observability framework that provides visibility into applications and infrastructure, enabling quick issue diagnosis and resolution. It's a powerful tool for organizations building complex software applications, offering standardized approaches to collecting telemetry data from metrics, traces, and logs. By using OpenTelemetry in Python, developers can automate the process of generating telemetry data, reducing manual configuration and code changes required for instrumentation. This allows for more consistent and standardized collection of telemetry data across different applications and frameworks. With its support for automatic and manual instrumentation, multiple languages, and telemetry data types, OpenTelemetry provides a flexible observability solution that helps organizations improve the reliability and performance of their distributed systems.

The C# date classes provide various options for handling and manipulating date data in applications. The DateTime class is the most commonly used, representing a specific date and time with properties and methods for operations like formatting and adding/subtracting time intervals. The DateTimeOffset class includes information about the time zone offset, making it more useful when working across different time zones. The TimeSpan class represents a duration of time rather than a specific date and time, allowing calculations involving dates and times. These classes provide various methods for getting the current date and time, adding or subtracting time intervals from a date, converting time zones, handling daylight saving time, and parsing dates from strings. Understanding the different date classes, their formats, parameters, and methods is essential to create powerful and reliable date and time functionality in C# applications.

The dateutil module is a Python library that simplifies the process of working with dates and time stamps, providing functions such as parsing dates from strings in various formats, calculating the difference between two dates, adding or subtracting time from a given date, and more. It's an extension of the datetime module and can be easily installed using pip. The module provides a parse function for parsing dates into desired string formats, a relativedelta function for calculating the difference between two dates, and functions for adding or subtracting time from a given date. These functions make it easier to work with dates and times in various ways, such as handling time series data.

Time series data has distinct characteristics, storage requirements, and workloads that differ from relational data, making it unsuitable for traditional relational databases like Postgres or MySQL. Relational databases are designed to maintain accurate records of objects and relationships, whereas time series data tells the story of what's happening now. Time series databases like InfluxDB are purpose-built to handle the unique demands of time series data, offering low latency, flexibility, and advanced analytics capabilities. By recognizing these differences and using a time series database, developers can avoid common pitfalls such as slow write speeds, schema rigidity, and high storage requirements, making it easier to work with time series data effectively.

The InfluxData team is attending Microsoft Build 2023, with live and virtual components, including a booth at the event in Seattle from May 23-25. The team will be presenting various demos, including an integration of InfluxDB with AI deployed at the edge. Experts from InfluxData will be available for one-on-one sessions to discuss how InfluxDB can bring value to applications and provide feedback on existing solutions. Additionally, attendees can expect prizes, including InfluxDB socks and other giveaways, and a chance to win by visiting the booth.

InfluxDB Cloud 3.0 is a versatile time series database built on top of the Apache ecosystem, allowing users to query it with the Apache Arrow Flight SQL interface for working with time series data. To query InfluxDB Cloud with this interface using Java, developers need to set up a connection with gRPC and TLS encryption, authenticate with an authentication token, and create a custom interceptor to inject database headers on every request. The code walkthrough demonstrates how to use the Java Flight SQL Client to execute queries on the server, fetching data from the result and printing it as a string. This tutorial is part of Apache Arrow Flight, a framework for building high-performance data services, providing efficient transmission of large datasets over gRPC.

InfluxDB 3.0 measures up against Microsoft Azure Data Explorer (ADX) in terms of its flexibility and ease of use for managing time series data, offering both cloud and on-prem solutions, as well as a range of features such as API access, schema-on-write, and real-time processing capabilities. ADX, on the other hand, is a relational database that supports time series workloads but lacks the flexibility of InfluxDB, being only available as a managed cloud solution on Azure and not supporting on-prem or edge deployment. Both databases offer features like OLAP workloads, batch processing, and data visualization capabilities, but InfluxDB's open-source Apache Arrow ecosystem and interoperability with other ecosystems give it an advantage in terms of extensibility and scalability.

This article discusses planning and executing IoT projects, highlighting key considerations such as hardware choices, networking protocols, integration tools, IoT platforms, and data storage options. It also showcases six practical IoT projects with tutorials and source code links, ranging from home automation to AI-powered security cameras and self-driving RC cars. Understanding the right hardware, tools, and frameworks can make or break an IoT project, especially when it comes to iteration costs.

InfluxDB is working to consolidate metrics, traces and log data into a single solution due to their unique characteristics and differing query patterns. Time series data has distinct features that set it apart from other types of data, with metrics, traces and logs requiring different workloads and data structures. Logs pose the greatest challenge in terms of data model design, parsing, querying and performance, mainly due to their high cardinality and variable attribute naming conventions. In contrast, traces tend to have more structured output and consistent tag keys, while metrics exhibit both bounded and unbounded tag values. Efforts are underway to develop a unified solution that can handle all three types of time series data, with InfluxDB IOx aiming to manage high-cardinality tracing data alongside metric and raw event data in a single database core.

InfluxData recently announced InfluxDB 3.0, which expands the number of use cases feasible with InfluxDB by storing traces, metrics, events, and logs in a single database. To demonstrate the capabilities of OpenTelemetry, an open standard for observability, InfluxData provides a working example that runs on top of InfluxDB 3.0. The demo uses Hot R.O.D. to simulate time series data, collects it with OpenTelemetry Collector, and writes it to InfluxDB 3.0. The demo is accessible through a KillerCoda environment or by running a local installation from the GitHub repository. Once set up, users can visualize their data in Grafana using Jaeger UI, which provides a bridging interface for metrics, logs, and traces between InfluxDB 3.0 and Grafana visualizations. The demo showcases how OpenTelemetry and InfluxDB 3.0 can be used to create observability solutions, and users are encouraged to play with the demo, fork the repository, and apply its components to their own use cases.

InfluxDB's `date_bin_gapfill` function is used to fill missing rows in time series data by calculating the time interval based on user-defined parameters. If no rows exist within a certain time interval, it inserts a new row with a null value. The function can also be combined with interpolation or last observation carried forward (LOCF) methods to replace null values with linearly interpolated or last observed values, respectively. These functions are necessary for advanced analytical tools and machine learning models that require fixed time intervals in their data inputs. InfluxDB's `date_bin_gapfill` function addresses the challenge of filling missing rows and values in time series data, making it easier to perform basic analysis and cleaning tasks.

IoT data is becoming increasingly prevalent, with an estimated 30.9 billion connected devices worldwide by 2025, generating vast amounts of time series data that require efficient storage and management to maintain bottom-line performance. Selecting the right tools for IoT data analysis from the outset can significantly ease data processing and reduce costs. Time series databases like InfluxDB are well-suited for this type of data due to their ability to query based on time, making it easier to analyze and visualize large datasets. High-volume ingestion tools such as Telegraf help manage the constant influx of data, while cleaning and downsampling can optimize storage and improve performance. Real-time monitoring is crucial for quick decision-making, and historical aggregation and cold storage are essential for long-term data analysis and preservation. By choosing the right tools and following best practices, organizations can efficiently process IoT data and make informed decisions.