April 2023 Summaries

The complexity of modern applications continues to increase, driving demand for comprehensive observability solutions. InfluxDB 3.0 is a time series database designed to handle large-scale monitoring and analytics workloads, offering unlimited cardinality, native SQL support, high-performance data ingestion, seamless integration with observability tools, scalability, and flexibility. By combining InfluxDB 3.0 with other observability tools, organizations can gain greater insights into system performance, monitor, analyze, and visualize their infrastructure, making it a powerful tool for building the next generation of observability platforms.

The new InfluxDB 3.0 Essentials Course is a comprehensive introduction to the latest version of InfluxDB, focusing on its core concepts and features, including InfluxDB IOx, data ingestion, querying, visualization, and best practices for schema design. The course aims to equip developers with the knowledge and skills necessary to get started with the new version, covering topics such as setting up the database, ingesting and visualizing data, and using SQL to analyze and query data. By taking this course, learners can gain a deeper understanding of InfluxDB's capabilities and improve their ability to work with the platform effectively.

InfluxDB 3.0 is a significant evolution of the InfluxDB product suite, with a new designation for all products powered by the InfluxDB IOx engine, including InfluxDB Cloud Dedicated. This cloud-based offering provides a fully managed solution for users with large data sets or customization needs, and offers features like private connectivity, not available in multi-tenant offerings. With a capacity-based pricing model, InfluxDB Cloud Dedicated is suitable for enterprise-grade solutions without infrastructure management, and leverages the Apache Arrow ecosystem to support a wider range of time series data types and use cases. The new product suite adds capabilities such as SQL support through DataFusion, lowering barriers for entry for SQL users, and allows users to utilize more value from their data by reading Parquet files natively. InfluxDB Cloud Dedicated is now available in all AWS regions except China, with plans to expand to Azure and Google Cloud in the second half of 2023. To get started, customers can contact sales representatives to work up a proof of concept and ensure they have the necessary capacity for their time series workloads.

InfluxDB 3.0 is the next-generation time series database powering all InfluxDB products, offering significant gains in performance, high-volume ingestion and compression, real-time querying, and unlimited scale to the database. Developed as an open source project, it now serves as the foundation for all InfluxDB products, delivering unprecedented scale and performance with modern programming language Rust, built-in Apache Arrow data structure, and columnar engine designed for scale and performance. InfluxDB 3.0 is available in cloud products such as InfluxDB Cloud Dedicated, a new product for developers that delivers the performance, power, and flexibility of InfluxDB with the security of a fully managed service, and will be released later this year in self-managed database options like Clustered and Edge. The upgrade to InfluxDB 3.0 enables customers to experience real-time query response, unlimited cardinality and high throughput, low-cost object store, SQL language support, and more, with minimal key changes from existing workloads.

InfluxDB 3.0 is now available, serving as the foundation for all InfluxDB products, bringing high performance, unlimited cardinality, SQL support, and low-cost object store to the platform. It introduces support for a single datastore to power use cases in observability, real-time analytics, and IoT/IIoT that rely on high-cardinality time series data. The new release is available in InfluxData's cloud products, including InfluxDB Cloud Serverless and InfluxDB Cloud Dedicated, with plans for two self-managed products coming later this year. It offers performance gains of 100x faster queries, 10x ingest performance, and 10x greater data compression, making it an ideal solution for customers working with large data sets who require reassurance and security of data isolation in a dedicated, single tenant cluster. InfluxDB 3.0 is also optimized for analytics databases with real-time capabilities for large volumes of data, allowing developers to ingest and query full fidelity time series data of all types, in real-time at scale, with no compromises.

Grafana has released a new community plugin that allows users to communicate with Flight-SQL-compatible databases. This plugin utilizes the Flight RPC framework and the Apache Arrow in-memory columnar format, enabling users to query data from InfluxDB databases using SQL. The plugin is available for free and can be easily set up by following a simple configuration process. With this new feature, users can now take advantage of the open source Apache Arrow project and advance its development.

A distributed database architecture is a design that comprises several nodes working together to store, manage, and access data. This approach offers improved scalability, availability, performance, and fault tolerance compared to traditional single-server databases. Distributed databases can scale vertically by adding more nodes, ensuring high availability even if one node goes down. They also provide reduced network traffic by storing data closer to where it will be used. However, designing and implementing a distributed database architecture is more complex and costly than a single database instance, with higher costs for hardware, software, and skilled personnel. The choice of architecture depends on the application's specific needs, including data partitioning, replication, and consistency models.

Time series data is a type of data where each observation is recorded at a specific point in time and collected at regular intervals, allowing for analysis of changes or patterns over time. Regression analysis with time series data can be used to gain valuable insights into the behavior of complex systems, identify trends and patterns, and make informed decisions based on predictions. Time series regression helps understand the relationship between variables over time and forecast future values of the dependent variable. The process involves collecting and preparing the data, specifying a model, estimating it using statistical software, conducting diagnostic tests, interpreting the results, and potentially forecasting future values. This technique is valuable for analyzing temporal relationships in various industries, including finance, economics, engineering, and social sciences.

Telegraf is an open-source tool used for data collection, leveraging its plugin-based architecture to connect with various data sources. InfluxDB provides scalable storage and querying capabilities for time series data, offering advanced analytics and real-time insights. Grafana enables visualization of data through customizable dashboards, allowing users to query and analyze data in real-time, and receive alerts when necessary. The TIG stack offers a comprehensive solution for predictive maintenance by integrating Telegraf, InfluxDB, and Grafana to provide a flexible and scalable platform for industrial operators to gain deeper insights into their machinery and equipment. By leveraging the capabilities of this open-source alternative to legacy data historians, users can save money on maintenance costs and optimize production with real-time data-driven decision making.

Formatting dates in SQL queries using various date formatting functions and languages can be challenging due to the numerous date formats and SQL engines available. This post provides a comprehensive guide on how to format dates in MySQL, PostgreSQL, and Microsoft SQL Server, including examples of code snippets and syntax for each function. It also introduces the concept of time series databases and their relevance to dates in SQL. The post emphasizes the importance of formatting dates accurately and consistently, highlighting best practices such as ensuring correct time zones, handling localization requirements, and using precision levels accordingly. Additionally, it discusses the benefits of using time series databases for managing and analyzing timestamped data, providing an overview of popular options like InfluxDB. By following the techniques and tips outlined in this post, developers can streamline their data management and analysis for modern software applications.

SQL provides various date functions to manipulate dates more efficiently. These functions can be used to extract specific parts of a date, format dates in different ways, and perform calculations with dates. The CURDATE() function returns the current date, while the EXTRACT(), DAY(), MONTH(), and YEAR() functions allow for extracting specific parts of a date. The DATE_ADD() and DATE_SUB() functions add or subtract intervals from a date, respectively, while the DATEDIFF() function calculates the difference between two dates. The DATE_FORMAT() function formats a date according to a specified format string, and the GETDATE() function returns the current date and time in SQL Server. Understanding these date functions can help developers work more efficiently with dates in their SQL applications.

Predictive maintenance is a critical strategy in today's business landscape, focusing on predicting equipment failures and other potential issues before they happen. It involves collecting and analyzing data from various sources such as sensors and historical records to optimize maintenance schedules, reduce unplanned downtime, and improve operational efficiency. Advanced analytical techniques like machine learning, artificial intelligence, and statistical modeling are used to process the collected data and generate estimates about the remaining useful life of equipment or the likelihood of failure within a specific time frame. Predictive maintenance plays a valuable role in managing assets and devices in IoT environments, where sensors continuously monitor data points like temperature, vibration, and pressure while transmitting that data for analysis. By shifting from reactive to proactive maintenance, organizations can achieve cost savings, increased asset longevity, enhanced productivity, and other benefits. Various tools are required to take advantage of predictive maintenance, including data collection tools, connectivity protocols, data storage solutions, data processing platforms, analytics and forecasting tools, and visualization libraries or frameworks. Some popular platforms for IoT and predictive maintenance include PTC ThingWorx, IBM Maximo, Azure IoT Hub, AWS IoT Core, and InfluxDB.

The InfluxDB add-on for Ockam Orchestrator allows customers to automatically grant uniquely identifiable, least privilege, time-limited credentials for any client connecting to InfluxDB Cloud. This improves security posture by reducing the risk of exposing sensitive secrets and simplifying remediation in case of a credential breach. The add-on enables organizations to define standardized policies that define credentials suitable for their use cases, with each client requesting its own unique credential when needed. The time-limited nature of these credentials automatically revokes them upon expiration, eliminating operational complexity and reducing the risk associated with managing access at scale.

The InfluxDB add-on for Ockam Orchestrator is a new solution designed to improve security posture for customers using InfluxDB Cloud by automatically granting uniquely identifiable, least privilege, time-limited credentials for any client that needs to connect. This feature addresses the challenge of managing access at scale and helps reduce risk associated with sharing credentials across multiple clients. The add-on allows organizations to define a standardized policy for their use case and enables automatic revocation and rotation of credentials when necessary.

The author has built an aquarium monitoring and alerting system using a Raspberry Pi, InfluxDB, Telegraf, and Grafana. The system tracks key metrics such as filter running status, temperature within acceptable bounds, scheduled tasks, and water flow rate. To collect readings from sensors, the author used scripts that communicate with the sensors via their respective protocols (1-wire for temperature and pulse-based for water flow). These scripts are then written to InfluxDB using Telegraf's InfluxDB_Listener Input Plugin, providing a buffer against downstream failures. The system also includes alerting capabilities in Grafana, which notify the user of anomalies such as extreme temperatures or low water flow rates. The project is relatively inexpensive, with most components being under £10, and can be expanded to include additional probes for monitoring other water quality parameters. However, finding reliable and affordable sensor hardware remains a challenge.

The InfluxData team is attending KubeCon + Cloud Native Con EU 2023 in Amsterdam, where they will have an on-site booth and virtual presence. The team will be showcasing the benefits of using InfluxDB with Kubernetes through live demos, expert sessions, and giveaways. Attendees can learn about improved support for tracing in InfluxDB Cloud, powered by IOx, as well as schedule 1:1 sessions with experts to discuss their use cases and get personalized insights. The team will also be offering prizes, including famous socks, and encouraging attendees to sign up for a free InfluxDB Cloud account ahead of time.

InfluxDB Cloud is a powerful platform for storing, processing, and analyzing time series data, and when combined with Tableau, it provides a robust toolset for data analysis. By uploading Parquet files from InfluxDB Cloud to Tableau, users can create interactive visualizations and dashboards, derive insights, and make data-driven decisions faster. The combination of Tableau and InfluxDB Cloud enables organizations to quickly create appealing and insightful visualizations, allowing them to quickly analyze their data in real-time. Additionally, the IOx storage engine provides near-unlimited cardinality, high-performance storage and querying capabilities, making it an ideal solution for large datasets. With the help of Parquet files, users can easily ingest and process their data, and Tableau's forecasting capabilities enable the automatic selection of algorithms, allowing users to focus on taking action on forecasted data rather than generating forecasts.

In this article, a tutorial is provided on how to connect Grafana with InfluxDB IOx by using two different methods. The first method involves installing custom Flight SQL plugins to utilize the SQL query language in Grafana, while the second method uses the Flux query language to access data from InfluxDB. To achieve these configurations, users need to have their InfluxDB connection details handy and follow specific steps such as updating configuration files, allowing unsigned plugins, and setting environment variables. The tutorial concludes by highlighting the two ways of configuring, connecting, and querying time series data from Grafana using InfluxDB IOx.

The current state of the database ecosystem includes relational databases still being the most widely used type, but with NoSQL databases gaining adoption. Various types of databases are emerging, including document databases, graph databases, time-series databases, columnar databases, in-memory databases, key-value databases, and search engine databases. The performance of a database is affected by factors such as on-disk storage format, primary index data structure, data compression, hot and cold storage, durability/disaster recovery, and others. When choosing a database for an application, factors to consider include data access patterns, latency requirements, write frequency, schema flexibility, consistency guarantees, and the team's existing skill set. It is also essential to weigh the additional complexity of managing a specialized database against its potential benefits.