March 2023 Summaries

InfluxDB's Flux is a data-processing and querying language used for time series data analytics, offering capabilities similar to SQL but also built-in analysis and data science features. It can be used through the browser, Visual Studio, or command line, with no extra installation required. Flux queries allow for data cleansing and normalization, standardizing search tags, and calculating metrics such as successful writes and percentage of "ok" points. Larger ranges can be queried using aggregateWindow(), enabling faster results and easier readability. Downsampling can reduce raw high-precision data to lower precision aggregates, making analysis on smaller datasets possible. Alerting capabilities allow for filtering down specific fields, determining quantiles, and setting alert types and levels, enabling real-time reactions to critical data points.

This tutorial guides users on how to query InfluxDB Cloud using Flight SQL, Pandas, and Jupyter Notebooks to explore and analyze time series data. The tutorial assumes Python 3.6 as the programming language and provides step-by-step instructions on importing dependencies, executing a SQL query, creating a reader object, reading all data into a pyarrow Table, converting the data to a Pandas DataFrame, downsampling the data with Pandas, creating a visualization of the data with matplotlib and seaborn, and writing the downsampled data back to InfluxDB Cloud. By following these steps, users can gain insights and make informed decisions based on their time series data using InfluxDB Cloud, Flight SQL, Pandas, and Jupyter Notebooks.

The third wave of the Industrial Revolution, also known as Industry 4.0, has brought about significant advancements in manufacturing and industrial processes, with a focus on instrumentation, machine learning (ML), and artificial intelligence (AI) to enhance industrial processes. However, legacy data historians, which were introduced during the second wave, have limitations that hinder their ability to keep up with modern industrial and manufacturing processes. These limitations include lack of interoperability, vendor lock-in, poor developer experience, scalability issues, and high cost of ownership. To overcome these challenges, organizations need to move beyond legacy data historians and adopt time series databases like InfluxDB, which can manage large datasets, run real-time analytics, leverage AI/ML processes, deploy predictive maintenance procedures, generate digital twins, and more. By adopting InfluxDB, industrial organizations can unlock the full potential of Industry 4.0 and advance their manufacturing practices.

The compactor is a hidden engine that handles post-ingestion and pre-query workloads in the background, enabling low latency for data ingestion and high performance for queries. The tasks of post-ingestion and pre-query include data file merging, delete application, and data deduplication, which are performed on a separate server to avoid sharing resources with servers handling data loading and reading. Data compaction is a critical process that reorganizes data into smaller files, reduces I/O operations, and improves query performance. The system uses a combination of techniques such as data overlapping, compaction levels, and isolated compactors to optimize resource utilization and improve efficiency. By separating the tasks of ingestion, querying, and compacting workloads into independent servers, the system can maximize resource utilization and minimize out-of-memory incidents.

** The Python time module is a powerful tool that allows developers to easily manipulate dates, format strings, calculate durations, and more. It provides various functions that enable tasks such as scheduling tasks, handling financial transactions, validating user input, logging data, and creating games with precise timing mechanisms. The module represents time values in seconds since the Unix epoch and offers methods to get the current time, parse date/time strings, measure time intervals, compare time values, and format time values into strings. With its vast capabilities, the Python time module is essential for building applications that require accurate handling of dates and times.

InfluxDB provides edge monitoring tools that help industrial IoT practitioners manage data at the edge in a distributed system. The Edge Data Replication (EDR) feature enables organizations to automatically send data from edge devices to InfluxDB Cloud for centralized storage and analysis, creating a holistic view of operations. To set up EDR, users need to create a connection between the local source (edge device) and the remote destination (InfluxDB Cloud), specify the data source and destination buckets, and configure downsampling to reduce data transfer costs. Once activated, InfluxDB Cloud can provide real-time analytics and visualization capabilities for the data, enabling proactive maintenance and optimization of industrial operations.

InfluxData has received the ISO/IEC 27001:2013 certification for its Information Security Management System (ISMS), which supports the company's InfluxDB database cloud and enterprise offerings, as well as ensuring compliance with ISO/IEC 27018 for protecting personally identifiable information. The certification demonstrates InfluxData's global commitment to comprehensive organizational security and risk management, following an extensive audit process by Coalfire, a certified body through ANSI National Accreditation Board (ANAB) and the United Kingdom Accreditation Service (UKAS). This achievement validates InfluxData's industry-recognized IT security and privacy protection standards, with Peter Albert, CISO, stating that achieving ISO 27001 certification is further validation of the company's commitment to this standard.

SQL databases provide functions to reduce complexity when working with dates and times, allowing users to extract, convert, and format date-time values efficiently. The available functions vary depending on the SQL distribution used, such as MySQL or SQL Server. For example, in MySQL, the NOW() function returns the current date and time in a specific format, while in SQL Server, the GETDATE() function achieves the same result. These functions can be used to query data from a specific date, add or subtract from a specific time value to find data related to a date and time before or after a specific time, and convert the date-time format. Practical examples demonstrate how to use these functions in MySQL and MS SQL Server to retrieve current dates and times, such as getting all records from today or fetching stories posted at the current hour. Additionally, the concept of time series databases is introduced, which are designed for storing data collected over time and provide a more efficient way to query specific points in time.

IoT sensors have become increasingly significant in various industries due to their ability to collect data from the environment or equipment and communicate with systems for real-time monitoring and automation. These smart sensors typically consist of a sensor that captures data, a microprocessor that analyzes the output via an algorithm, and communication resources that enable action. The most commonly used types of IoT sensors include motion sensors, proximity sensors, accelerometers, pressure sensors, infrared sensors, temperature sensors, and humidity sensors. These sensors have various applications in home automation, healthcare, industrial manufacturing, environmental monitoring, and agriculture, enabling better control, monitoring, and automation. As technology advances, more use cases will emerge, leading to a greater impact on industries and creating new opportunities for innovation.

Telegraf 1.26.0 has been released, featuring improved static builds for Linux, default configuration updates, and the addition of new inputs, parsers, and processors such as OpenSearch Query, P4 Runtime, Radius, Windows Management Instrumentation, Apache Avro, and Lookup. State persistence is now available in select plugins, allowing data to be collected where it was left off without changing or losing its state. The release also includes a call for testing and invites users to join the InfluxDB Community Slack to report issues or have questions.

InfluxDB's IOx database engine solves the cardinality problem for time series data by storing each tag and field as a column, significantly reducing the total number of columns and improving performance. Cardinality issues arise when unbounded data creates unique combinations of measurement and tags, resulting in high cardinality. InfluxDB's schema-on-write approach allows flexibility in data shape, but also requires consideration to avoid cardinality issues. The IOx engine introduces a limit of 200 columns per measurement to prevent wide schemas from impacting performance. Additionally, sparse schemas with many null values can be optimized by splitting data into multiple measurements or using more efficient storage formats. With the new IOx engine, InfluxDB can now ingest and analyze large workloads in real-time, providing consistent performance across various time series use cases.

eBPF (Extended Berkeley Packet Filter) is a technology that creates a sandbox environment allowing bytecode programs to run in the core of an OS kernel without modifying or changing the default behavior of the kernel. eBPF was created to prevent needless packet transfers from the kernel space to the user space, and it works by accessing a program written in some kind of language, compiling it into bytecode, attaching it to a specified hook, and loading it into the kernel. The verifier checks the bytecode for security and assurance compliance before execution, and the Just-In-Time (JIT) compiler compiles the bytecode into native machine code for stronger execution speed. eBPF can be used for resource management, networking, security, observability, and debugging, providing a safe environment for programs to execute without affecting the kernel. However, it may not be suitable for use cases requiring more flexibility and control over program execution, and alternatives like Iptables, SystemTap, and LTTng exist for specific tasks. eBPF is a powerful tool for extracting metrics from the kernel without modifying or breaking the kernel itself, making it an attractive option for certain use cases.

Deduplication can be an effective alternative to transactions for eventually consistent use cases of a distributed database. It allows data to be redundant as long as it can be managed effectively, and by identifying the redundant data and eliminating that data at read time, the expected result can be produced. In contrast, a transactional system always produces consistent results but is complicated to build and maintain due to the need for guaranteed consistency. Deduplication in practice involves organizing data properly and implementing the right deduplication algorithms, such as sorting data inserts on their keys and using a merge algorithm to find duplicates and deduplicate them. By performing deduplication during read time or as a background task, it is possible to improve query performance while avoiding sharing CPU and memory resources with data loading and reading.

InfluxDB Cloud, powered by IOx, is a significant release that aims to be fast for large time series workloads, with features such as fast queries for leading-edge data, fast ingest of massive amounts of data, and smallest possible on-disk footprint. The engine leverages the Apache Arrow project to enable users to write time-series-specific queries using a familiar language (SQL) and their tool(s) of choice. This approach to open source truly enables the community by providing components to assemble high-performance databases. InfluxDB IOx also embraces emerging standards, such as SQL query language, Parquet, and Flight SQL, which allows cooperation with other developers and companies to satisfy all types of users. Additionally, it supports Apache Superset, a full-featured BI/dashboarding tool that is part of the Apache foundation. The release aims to simplify data querying, making it easier for users to write time-series-specific queries without worrying about cardinality concerns. InfluxDB IOx is designed to process large time series workloads quickly and efficiently, with support for popular libraries such as Pandas, Plotly Express, and Neural Prophet. Overall, the release offers significant benefits, including better performance on bigger workloads, support for all team's favorite tools, and a deeper commitment to open source.

A time series database (TSDB) offers a convenient perspective on modeling tasks involving time-varying entities. Unlike traditional entities, which model an instantaneous view, time-varying entities describe the variations of all their time-varying attributes over time. TSDBs have a more sophisticated relationship modeling than entities, distinguishing between relationships between static entities, time-varying entity and static ones, and two time-varying entities. The relationship cardinality is orthogonal to the discriminatory aspect of whether time-varying attributes characterize the relationship. Modeling these complex relationships requires advanced techniques, including windowing and pivoting, as demonstrated in the provided InfluxDB query example. Effective data modeling involves mapping time-varying entities into TSDBs while preserving identifying nature of attribute and using tags for performance optimization.

Data modeling is a critical step in developing any information system, as it defines the structure of data and its relationships. A data model can be divided into three levels: conceptual, logical, and physical. The conceptual level defines what the system contains, while the logical level defines how the database management system should be implemented. The physical level describes how to implement the information system using a specific technology. Data modeling is used in various methodologies, including relational modeling and dimensional modeling, which focus on different aspects of data storage and analysis. These methodologies differ in their approach to handling time series data, with relational modeling focusing on temporal validity and dimensional modeling considering time as an analytical dimension. Understanding these concepts is essential for effective data modeling and developing a well-structured information system.

InfluxData's time series database technology has been resonating with Peter Albert since he first encountered it in the early 2000s at his previous company, iPass. He recalls using MRTG and RRDTool to monitor network traffic and CPU load, which were essential tools for building a global network operations center. Since then, Peter has been involved with various projects that utilized time series data, including cloud security management platforms and fraud monitoring companies. He now serves as the CISO of InfluxData, where he appreciates the company's strong engineering team and its commitment to innovation, particularly in areas like IoT and cloud security breach detection. Under the leadership of CEO Evan Kaplan and CTO Paul Dix, InfluxData continues to develop its industry-leading time series database platform, including its latest third-generation database, IOx.

InfluxDB's new IOx database engine enables the handling of various unique time series workloads, including metrics, raw events, tracing, and logs. Metrics are polled at fixed intervals to gain visibility into systems or data sources, while raw events allow for real-time analysis without polling. Tracing is enabled with the InfluxDB IOx engine, which solves the cardinality problem that affected previous versions. Logs can also be handled by InfluxDB IOx, although it's not yet optimized for log workloads. The new engine maximizes the number and type of time series workloads available in a single database, unlocking previously difficult use cases and making them performant.

InfluxDB Cloud, powered by IOx, is now available in two AWS regions, Frankfurt, Germany, and Virginia, USA. This columnar database, built on Apache Arrow with cloud environments in mind, supports SQL natively and includes native InfluxQL support, enabling unlimited cardinality. The underlying design and architecture of IOx removes the cardinality problem, allowing developers to scale compute layers dynamically without manual work. Rust is used for its fine-grain control over memory, safety, and concurrent applications. Apache Arrow Flight SQL is seen as the new standard for database systems, with InfluxDB Cloud leveraging it for data transfer between clients and servers. The database uses object storage for persistence and Apache Parquet as the persistent format. A demo was provided by Balaji Palani, VP of Product Marketing, showcasing the new Data Explorer and Apache Superset visualizations, while Q&A session addressed various benefits and features of InfluxDB Cloud, powered by IOx.