October 2018 Summaries

InfluxData's Head of Product Marketing Navdeep Sidhu discusses the growing adoption of Time Series Platforms across industries, driven by recent technology trends and the need for organizations to leverage AI/ML-driven smart apps. The growth is spurred by four key use cases: DevOps, IoT, Microservices, and Real-Time Analytics, which require handling large amounts of metrics and events. These platforms are becoming increasingly important as they enable organizations to turn to AI/ML-driven smart apps and continue to grow with the requirements of predictive analytics and newer app architectures.

Telegraf 1.8.3 has been released, offering improvements to several plugins including AMQP Output, IPMI Sensor Input, Jolokia2 Input, and PostgreSQL Input, with fixes for connection leaks, panics, and version checks, among other enhancements.

Monitoring database health and behavior can be overwhelming due to numerous available metrics. To simplify this process, it is recommended to focus on foundational, database-agnostic metrics that provide a solid foundation for understanding database performance. These metrics include throughput, execution time, concurrency, and utilization, which together create a comprehensive picture of a database's overall health and behavior. By tracking these key indicators, developers can gain insights into their database's performance and make informed decisions to optimize its operation.

Three new maintenance releases have been made available for Telegraf 1.8.2, InfluxDB 1.6.4 and InfluxDB Enterprise 1.6.4. The InfluxDB Enterprise release includes a critical security fix that requires the use of an internal-shared-secret to address a potential security issue. This change addresses a weakness in JWT authentication used for intra-node communication when authentication is enabled without setting an internal-shared-secret. Additionally, Telegraf has received improvements to its plugins, including support for unsigned integers and fixes for issues with Docker, Filecount, Logparser, MongoDB, MQTT Consumer, Ping, VMware vSphere, and x509 Certificate inputs.

As IT services continue to be consumed and delivered in a cloud-based infrastructure, DevOps processes are increasingly instrumenting this environment, while microservices-driven apps are replacing traditional monolithic architectures. This shift is driving the need for better data monitoring and analysis than ever before, as organizations seek to improve their ability to navigate these changes. When evaluating a Data Monitoring solution, it's essential to consider key factors such as scalability, security, and integration with existing tools and processes. To develop an effective monitoring strategy, businesses should ask themselves questions like what metrics are most important for their use case, how data will be visualized and presented, and what level of alerting and notification is required. By considering these factors, organizations can choose a Data Monitoring solution that meets their specific needs and helps them stay ahead in the rapidly evolving IT landscape.

InfluxDB allows for storing raw events as irregular time series, which presents unique challenges when performing common operations like calculating the mean of values. To address this, InfluxDB enables converting an irregular time series to a regular one on the fly by aggregating individual values within arbitrary windows of time. This approach requires imposing some kind of regularity on the data, such as breaking up the time range into discreet units using InfluxQL's GROUP BY clause and then aggregating the values within those windows. The choice of window size and interpolation method can significantly impact the final results, highlighting the need for understanding the data well enough to make educated decisions about how to work with it. By utilizing tools like fill() options in GROUP BY clauses, users can effectively handle irregular time series and derive meaningful insights from their data.

The fourth InfluxDays industry event is being held on November 7 at The Village in San Francisco, focusing on the impact of time series data for real-time decision making. This event attracts attendees from a wide range of industries and roles, including CTOs, developers, architects, and engineers. Companies such as PayPal, RingCentral, FuseMail, MuleSoft, and Coupa Software will share their experiences with leveraging time series databases to inform real-time decisions. The event aims to educate attendees on the use of metrics and event data from various sources, covering topics on time series data for monitoring, analytics, and IoT applications. InfluxData's modern Open Source Platform has gained over 420 customers across industries, including manufacturing, financial services, energy, and telecommunications, by enabling businesses to derive better insights and take digital transformation initiatives.

Rust is a programming language that offers several strengths, including performance, low cost integration with C and C++ libraries, and the ability to create libraries that can be linked to other languages via FFI (Foreign Function Interface). The author of the text has been drawn to Rust's potential due to its relevance to their work on InfluxDB, a time series database. To learn Rust, the author has used a project-based approach, starting with implementing a lexer, parser, and tree walking interpreter, as described in Thorsten Ball's book "Writing an Interpreter in Go". The author has found Rust's documentation to be excellent, with built-in documentation for the standard library and third-party libraries, making it easy to learn and use. Despite some initial challenges, particularly with implementing a parser, the author has made significant progress in learning Rust and is excited about its potential for building secure systems software. The author plans to continue learning Rust and applying its principles to their work on InfluxDB, including creating an embeddable implementation of Flux, a new scripting language.

I've summarized the text for you: Configuring Docker Telegraf Input Plugin is crucial to monitor containers with InfluxDB. The author started by creating two images, one for a static HTML page and another for an nginx reverse proxy image that forwards requests to all of the static app containers in Round Robin with a built-in DNS resolver. They configured a docker-compose.yml file and created a config file using telegraf --input-filter docker --output-filter influxdb config > docker_telegraf.conf. The author used Telegraf and InfluxDB to monitor their containers locally, sending metrics to the default "telegraf" database. After running several tests, they were able to see that the proxy was successfully working and Telegraf was gathering metrics from their containers. However, this exercise has left them with many unanswered questions about container monitoring, DevOps, and InfluxDB.

Writing logs directly to InfluxDB is a viable alternative to using syslog or the Telegraf plugin, as explained by Noah Crowley in an article published on DZone. The article delves into how syslog works and its role when used with Telegraf, which converts messages to line protocol for writing to InfluxDB. Crowley demonstrates that writing data directly to InfluxDB is possible, following a specific schema. To optimize performance, it's recommended to batch large numbers of points before sending them to the database.

DB-Engines' latest industry results reveal that InfluxData's Time Series Database management system, InfluxDB, has taken a leading position in the rapidly growing Time Series Database category, with its popularity increasing by 49% over the past 12 months and 144% over the past two years. According to DB-Engines' October results, InfluxDB is nearly three times more popular than the nearest competitor, scoring the biggest increases in user popularity over the past month and year. The growth in this database segment is attributed to the increasing requirement for analyzing time-stamped data efficiently, which is best served by a Time Series database. InfluxData's unique features enable customers to quickly build monitoring, IoT applications, and real-time analytics applications, supporting their DevOps initiatives and deriving better business insights through data-driven real-time actions. More than 420 customers have selected InfluxData as their modern data platform for metrics and events.

This article, published by DZone, provides a comprehensive guide on using Google Core IoT with InfluxData. InfluxData has collaborated with Cloud Service Providers to ensure InfluxDB remains the preferred Time Series Database for IoT. To complete the tutorial, users will need specific components and tools. The tutorial covers setting up IoT Core, an overview of InfluxData components, and a step-by-step walkthrough of building and configuring Telegraf, a new plugin designed specifically for Google Core IoT.

The author discusses the ability to write logs directly to InfluxDB without using the syslog protocol or Telegraf plugin. This allows users to ingest and view logs in Chronograf, providing detailed insight into system issues. By following the syslog protocol, users can create log entries from any application that makes an HTTP connection to InfluxDB, leveraging fields and tags for troubleshooting purposes. The schema for logging data is well-defined, allowing users to write points directly to the database and land them in the `syslog` database, where they appear in the Log Viewer. This approach enables users to create logs entries from any application that can make an HTTP connection to InfluxDB.

InfluxData has partnered with Google Cloud Platform to provide a one-click access to its Time Series Platform on the Google Cloud Marketplace, allowing users to easily adopt and utilize the platform's features such as real-time data collection and analysis for performance optimization. The partnership is part of InfluxData's strategy to support all leading cloud vendors and provides customers with flexibility without lock-in, facilitating cloud adoption and seamless application management. With the new listing, users can access the Time Series Platform through a simple one-click configuration, making it easier to get started with the platform and start processing events and metrics in minutes.

InfluxData has integrated with Google Cloud IoT Core to enhance users' IoT environments by expanding data collection and analysis capabilities, allowing for real-time decision making and control through InfluxDB and the Time Series Platform. The integration features a Telegraf agent that provides open-source insights into IoT environments, while also offering one-click access to InfluxData's platform through Google Cloud Platform's Marketplace listing.

InfluxData has announced an integration with Google Cloud IoT Core, enabling users to gain better insights and analytics from their IoT environments through a new Telegraf agent. The integration aims to provide a more comprehensive view of IoT data, allowing users to make more informed decisions. Sharjeel Noor, Senior Director at InfluxData, emphasized the significance of this announcement for the IoT industry, highlighting the potential benefits of improved data analysis and management. The Telegraf agent is an open-source tool that facilitates seamless communication between devices and the cloud, making it easier for users to collect, process, and analyze their IoT data.

The tutorial provides a comprehensive guide on how to use Google Core IoT with InfluxData, a Time Series Database for IoT deployments. The tutorial assumes an existing instance of InfluxDB running and accessible. A Telegraf plugin is built for Google Core IoT, and the process of building and installing it is detailed. The tutorial covers setting up Google Cloud Platform account, installing the Google Cloud Command Line Tool, generating Public/Private key pairs, creating a device registry and device, and configuring Telegraf to support Google-specific functionality. The client application is written in Go and can be run on various devices that support Golang. The tutorial concludes with running the client app, verifying the results, and launching Chronograf to see data streaming into InfluxDB.

Google Cloud IoT Core has integrated with InfluxData's time series platform to provide users with expanded data collection and analysis capabilities, enabling real-time decision making and control from IoT sensor data. The integration allows users to collect and analyze metrics and events data in real-time, improving operational efficiency and optimizing businesses with automated action and control. InfluxData's Telegraf agent is now available for Google Cloud IoT Core, providing a powerful time series platform that ensures data is collected and analyzed in real-time. This integration fits with InfluxData's focus on being the data services platform of choice, offering benefits as part of larger ecosystems like Google Cloud. The partnership enables users to maximize ROI from their IoT applications by capturing, understanding, and acting on data collected from IoT devices and sensors in real-time.

Telegraf is a plugin-driven server agent that collects and reports metrics and data, and its integration with Google Cloud IoT enables customers to easily gather time series data from IoT devices and sensors and store it in InfluxData for alerts and monitoring. This integration accelerates Time to Awesome for Google Cloud IoT customers by improving their IoT environments with expanded data collection and analysis capabilities. The Telegraf plugin provides organizations with the ability to monitor and track sensor data, use historical data to gain insights, and deliver automated action and control with no human interaction. InfluxData's Data Services layer supports this integration, allowing users to collect time series data for instrumenting, observing, learning and automating any system. The integration also addresses the critical industry need of handling the vast amount of IoT data expected to be generated by 2020.

K-Means is an unsupervised learning technique used for clustering, and it can be applied to time series data, which are sequences of measurements taken at regular intervals over time. Anais Dotis-Georgiou explains that K-Means clustering is particularly useful for time series data because it allows for the identification of patterns or groups in the data without requiring any prior knowledge of the underlying structure. The technique can be used to reduce the dimensionality of high-dimensional time series data, making it easier to analyze and visualize. By grouping similar values together, K-Means clustering can help to highlight trends and anomalies in the data, providing valuable insights for time series analysis and forecasting.

There is no need for database indexes in most cases, as the database can simply use a linear search to find the desired information. However, when dealing with large datasets or frequently accessed data, database indexes can significantly improve performance by allowing the database to quickly locate and retrieve specific records. Indexes are created on columns or fields that are used most frequently in queries, reducing the number of writes required to update the index. The main difference between relational and NoSQL databases lies in how indexing is implemented, with relational databases relying on column-based indexes and NoSQL databases often using more flexible and customizable indexing methods.

This post walks through how to visualize dynamically updating time series data stored in InfluxDB using the JavaScript graphing library Dygraphs. To begin, sample data from InfluxDB is queried periodically on the frontend, and then fetched from InfluxDB using the HTTP API. The fetched data is then used to construct a graph using the Dygraphs constructor function, which includes options such as title, labels, and stroke width. Additionally, an interval is set to fetch new data every five minutes, allowing for dynamic updates of the graph over time.

Telegraf 1.8.1 has been released, featuring improvements in performance and functionality, including updates to its Basic Stats aggregator, Cloudwatch output, GROK parser, HTTP Listener input, OpenTSDB output, SQLServer input, vSphere input, x509 Cert input, with bug fixes for issues such as adding tags with empty values, resolving panics during network errors, and fixing metrics discrepancies due to clock inconsistencies.

K-Means is used for anomaly detection in time series data by first windowing the data into segments, then clustering these segments using K-Means. The centroids of the clusters represent different shapes or polynomials that the data takes. By analyzing the shape of each cluster and its position in the 32-dimensional space, it's possible to detect anomalies in the data. However, K-Means has limitations, such as only converging on local minima, which can lead to poor clustering and predictions if initial centroids are placed poorly. Additionally, using the Euclidean distance as a similarity measure can be misleading, especially when dealing with non-uniform time-steps or sensor data.

K-Means clustering is an unsupervised learning technique used for organizing data points into groups based on their similarity, maximizing data similarity within clusters and minimizing it across clusters. It's particularly useful for time series data analysis as it can help detect anomalies such as one-off spikes, tightly packed data with controlled systems, or normal distributions. The technique can also be applied to contextual anomaly detection, where the system is trained to recognize patterns in healthy, normal signals, allowing it to predict and reconstruct new data points and measure error to determine if an anomaly is present.

The Drive to Autonomy Looks Promising, Despite Evil Robots Aside` In various industries, including transportation and manufacturing, there is a growing focus on implementing autonomous systems. The end goal of these human-designed systems is autonomy, which has the potential to revolutionize numerous sectors. This shift towards autonomy is often referred to as the "Automation Economy," following the era of the "Information Economy." To achieve autonomy, four key steps must be taken, and data considerations are crucial in designing such systems. The development of autonomous systems is an exciting area that holds promise for significant positive change.