August 2023 Summaries

Combining Grafana and InfluxDB Cloud provides a powerful tool for real-time data monitoring and alerting, enabling users to effectively monitor their data and respond quickly to potential discrepancies or anomalies. The platforms offer an efficient, scalable, and customizable solution that allows users to visualize their data and set up alerts using the Flight SQL plugin. With Telegraf installed on a machine, users can write CPU metrics to InfluxDB Cloud and query and visualize data with Grafana. To set up an alert in Grafana, users must first configure the Flight SQL plugin, add a new rule, select a condition for the alert, and create a threshold. The alert evaluation behavior and contact points can also be configured, allowing users to test their alerts and verify that they are receiving notifications. Overall, this combination of platforms offers a robust solution for real-time data monitoring and alerting.

The blog discusses the basics of infrastructure monitoring using Telegraf, InfluxDB, and Grafana. It explains the difference between monitoring and observability, highlighting that they are complementary approaches to system health and performance. The authors categorize these concepts into several distinct fields, including network monitoring, server monitoring, application performance monitoring (APM), and cloud infrastructure monitoring. They present a problem to solve, where they aim to create a solution for each of these fields using the TIG stack. Telegraf is used for data collection, while InfluxDB 3.0 provides a purpose-built time series database for handling metrics, traces, and logs at scale. Grafana is used for data visualization and tracking of metrics across various data sources. The authors also touch on OpenTelemetry, which aims to provide the ability to visualize and inspect traces alongside metrics using a single pane of glass through Grafana.

Splunk is a comprehensive monitoring platform that provides service mapping, distributed tracing, code-level visibility, intelligent analytics, and advanced features such as real-time monitoring and observability. It offers a 14-day trial period and flexible pricing plans depending on workload, data, activity, or number of hosts. Datadog provides end-to-end distributed tracing, automated deployment tracking, synthetic monitoring, real user monitoring, service dependency visualization, and scalable pricing with a 14-day trial. New Relic offers infrastructure monitoring, distributed tracing, log management, synthetic monitoring, and pay-as-you-go pricing at $0.30/gigabyte above the free 100 GB data. Dynatrace provides deep transaction tracing capabilities, end-to-end visibility into application transactions, PurePath technology, and Davis AI engine for automation and intelligent analytics. AppDynamics offers comprehensive visibility, deep diagnostics, intelligent analytics to optimize application performance, and integration with Cisco's networking expertise. Sentry is a widely-used APM tool providing error monitoring, crash reporting, performance monitoring, transaction tracing, release tracking, and flexible pricing tiers including a free developer tier. Application Insights offers live metrics, distributed tracing, intelligent analytics, alerting, and monitoring, with Azure pricing tiers based on data volume and features. Paessler PRTG provides customizable reporting, distributed monitoring, intelligent network discovery, and scalable pricing based on the number of sensors used. Elastic APM leverages Elasticsearch to collect transaction logs, system metrics, user behavior data, and offers real-time monitoring, RUM, transaction sampling, logging integration, and cross-cluster search. InfluxDB is a high-performance time series database for storing, querying, and analyzing time-stamped data with features such as data collection, time series data models, data visualization, and query languages. SigNoz is an open-source APM tool providing distributed tracing, advanced filtering, custom aggregation, performance analytics, and flexible pricing plans including teams and enterprise tiers. SolarWinds AppOptics offers transaction tracing, live code profiling, exception tracking, root cause summaries, and seamless integration with other SolarWinds products. When choosing an APM tool, consider factors such as ease of use, scalability, pricing, features, and capabilities, and may choose to use multiple tools depending on specific needs.

The author of the article successfully modified Grafana's source code to enable a specific feature, specifically the "Logs for this span" button in the Trace visualization. The modification involved adding two new data sources, InfluxDB and FlightSQL, to the list of supported log input sources. To achieve this, the author added these data sources to the `TraceToLogsSettings.tsx` file and created a custom query function for each data source. The author also modified the `createSpanLink.tsx` file to include the new queries. After making the changes, the author built a Docker image of Grafana with the modifications and tested the results, which included enabling the "Logs for this span" button and adding a log link next to each span. The article aims to inspire developers to explore and adapt open-source projects to their needs, rather than simply using existing features without modification.

This tutorial discusses the use of TensorFlow Lite, a lightweight version of TensorFlow, for deploying machine learning models on edge devices such as smartphones and microcontrollers. It provides steps for installing TensorFlow Lite on various platforms, including Android, iOS, and Linux, as well as methods for getting and building a model, including using pretrained models, training custom models with TensorFlow, and converting existing TensorFlow models to TensorFlow Lite. The tutorial also covers how to run inference on the model using the TensorFlow Lite interpreter and provides an overview of the differences between TensorFlow and TensorFlow Lite. Ultimately, it aims to help developers get up and running with TensorFlow Lite for deploying machine learning models on edge devices.

The Telegraf metric aggregation tool is widely used due to its diverse plugins that interface with various systems without requiring complex software logic. However, managing credentials in other subsystems can be challenging. To address this, the article discusses strategies for deploying a Telegraf Docker container using Docker Compose v2, including the use of secrets and secret stores, which provide a safer way to pass credentials to plugins. The article also explores splitting Telegraf configuration files into individual files to make maintenance easier and demonstrates how to mount these files to the Telegraf container using Docker volumes or Configs.

The InfluxDB 3.0 client libraries available are v1, v2, and v3 Lightweight Client Library, each with its own strengths and weaknesses depending on the version of InfluxDB used, query language required, and administrative APIs needed. The v3 Lightweight Client Libraries are recommended for new customers or those using InfluxDB 3.0, as they provide a simple and efficient way to write to and query from InfluxDB without relying on administrative APIs. Apache Arrow Flight is another option for querying InfluxDB 3.0, offering high performance and support for SQL, InfluxQL, and other languages. Existing customers using v1 or v2 client libraries can continue to use them with InfluxDB 3.0, but may need to consider upgrading to the v3 Lightweight Client Library or Apache Arrow Flight for future applications.

Teréga, a French gas storage and transportation company, is accelerating its energy transition by creating a mixed-energy business model that incorporates biomethane and hydrogen into traditional energy sources. The company's legacy IT infrastructure was outdated, making it challenging to onboard new customers and projects, maintain, and innovate. After researching cloud-native data historians, Teréga built its own proprietary historian, IO-Base, using InfluxDB as its foundation, which provides a high-performing, scalable, and resilient solution for handling massive time series data. The company's new historian, IO-Base, is cloud-native, secure, and user-friendly, with features such as real-time metrics creation, customizable user interfaces, and seamless integration with AWS infrastructure.

In today's data-driven world, collecting and querying data from multiple sources has become crucial for organizations. To maximize their datastores' potential, implementing best practices such as optimizing data ingestion pipelines, designing appropriate schema structures, and utilizing advanced querying techniques is essential. InfluxDB, a powerful time series database, provides a robust solution for managing and analyzing time-stamped data, allowing organizations to make informed decisions based on real-time insights. Understanding different data sources and selecting the right data ingestion methods is vital, as well as ensuring data integrity and consistency throughout the collection process. Effective data modeling involves understanding the structure and relationships of data entities and designing schemas that facilitate efficient data storage, retrieval, and analysis. Combining data sets from multiple sources requires tools such as SQL and Pandas for flexible and efficient data processing. By embracing a multi-data-source approach, organizations can unlock their data's full potential, gain actionable insights, and drive business growth and innovation.

Node-RED is a programming tool built on top of Node.js that provides a way to create automation with minimal code, wiring different nodes together to carry out automation tasks. To get started with the Node-RED dashboard, users need to install the dashboard module in their existing Node-RED setup and ensure it reads node-red-dashboard. The dashboard allows users to create a graphical user interface (GUI) for their automation using various nodes on the Node-RED dashboard. Users can arrange their UI using tabs and groups, displaying different widgets such as buttons, texts, charts, etc. They can also display audio nodes, sliders, gauges, and charts to create interactive dashboards. The dashboard offers a light theme by default but allows users to switch to dark themes or customize their own themes. Additionally, users can edit settings on the main page of Node-RED to personalize their dashboard's appearance.

Time series decomposition is a statistical process used by businesses to break down time series datasets into individual components such as trend, seasonality, and noise. This technique helps analysts discover patterns and variations within the data, making it easier to model and forecast future data points, identify anomalies, and make accurate data-driven decisions. Time series decomposition is applicable in various fields like economics, retail, healthcare, manufacturing, and logistics. Two common methods for decomposition are the additive model and the multiplicative model. Python, R, Julia, and Rust are some programming languages used for time series decomposition.

InfluxDB 3.0 has been compared to InfluxDB Open Source (OSS) 1.8 through recent benchmarks, showcasing its superior performance in various areas. InfluxDB 3.0 provides 45x better write throughput and 4.5x better storage compression compared to InfluxDB OSS, making it more efficient for given hardware resources and capable of reducing storage costs by over 90%. Additionally, InfluxDB 3.0's queries are significantly faster across various query types, with performance improvements ranging from 2.5-45x, depending on the type of query and data range. The newer version of InfluxDB also outperforms OSS in terms of data ingestion and storage size on disk, making it a more scalable and cost-effective solution for time-series data management.

This post explores the use of webhooks in Home Assistant for automation purposes, allowing users to send information after an event trigger, create custom automation scenarios, and integrate with third-party services. By creating a webhook, users can automate tasks such as sending notifications or executing actions based on conditions like location data or sensor readings, providing flexibility and control over their home's smart devices. The use of webhooks also enables data analytics, allowing users to analyze behavior and gain insights before creating automation scenarios. With its adaptability in smart home automation, the possibilities for using webhooks with Home Assistant are endless.

Data models are critical for complex software applications as they have a profound effect on how the software is written and how we think about the problem being solved. Proper data modeling takes time, but its benefits include improved performance, reduced errors, and easier data analysis, ultimately leading to better business insights and cost savings. To achieve effective data modeling, it's essential to keep business objectives in mind, properly document the data model, design it to be adjustable over time, and choose the right data modeling technique for the type of data being worked with. Additionally, considering data governance and security, avoiding premature optimization, and selecting the right database based on data access patterns are also crucial aspects of data modeling. By following these best practices and understanding how your data will be accessed, you can make informed decisions to optimize your data storage and performance.

The Pandas pivot function is used to reshape data by transforming rows into columns. It's particularly useful when you want to rearrange data from a "long" format to a "wide" format, making it easier to compare attributes across different entities. The pivot function offers three main parameters: index, columns, and values. It is crucial for multiple reasons such as data reshaping, improving data analysis, enhancing data visualization, maintaining data integrity, and time efficiency. Pivoting helps maintain data integrity by avoiding unnecessary duplication. In Pandas, using the pivot function is often more time-efficient than manually reorganizing data or using more complex transformation functions. The pivot function can be used in various scenarios such as project management, website traffic analysis, and time series analysis.

The benefits of routine maintenance in industrial settings are often overlooked, as even well-planned programs can lead to more downtime than necessary. To improve equipment effectiveness, organizations use measures like Overall Equipment Effectiveness (OEE), which takes into account factors such as performance speed and availability. A shift from preventive to predictive maintenance has proven valuable, utilizing machine and product data to determine when maintenance is required before a problem or breakdown occurs. This approach is made possible by advanced analytics and modeling platforms that rely on time series data, such as InfluxDB, which can be used to analyze machine sensors and predict potential failures. By leveraging these tools, organizations like a German welding shop and an engine head manufacturing plant have seen significant improvements in their quality assurance processes, reduced production costs, and increased efficiency.

Apache Arrow DataFusion's new parallel aggregation capability has improved performance by 2-3x for queries with a large number of groups, reaching near-DuckDB-speeds querying Parquet data. This improvement is significant for developers building products and projects with DataFusion, allowing them to spend more time on value-added domain-specific features. The new optimization reduces allocations, uses contiguous native accumulator states, and vectorized state updates, leading to improved performance for high cardinality groups. DataFusion's community effort has led to the development of this improvement, which is part of the next generation "Deconstructed Database" architectures using fast, modular components.