January 2018 Summaries

The second annual Influx/Days event is being held on February 13th at the Maiden Lane Conference Center in New York City, focusing on the impact of time series data and its applications in various industries, including DevOps, IoT, and analytics. The event features presentations from Tesla and Comcast, as well as educational sessions and networking opportunities for attendees from diverse roles and industries. InfluxData, a modern Open Source Platform built specifically for metrics and events data, is hosting the event to empower developers to build next-generation monitoring, analytics, and IoT applications. The platform provides tools and services for accumulating, analyzing, and acting on time series data, enabling customers to derive better business insights and drive digital transformation initiatives.

InfluxDB 1.3.9 and InfluxDB Enterprise 1.3.9 maintenance releases are now available, featuring improved performance and bug fixes for both open source and enterprise versions of the database software. For users of InfluxDB Enterprise, it is recommended to follow specific upgrade steps if using the TSI preview, as detailed on the official website. The latest version binaries can be obtained from the downloads page or directly through the InfluxDB Enterprise portal for enterprise customers.` ` `The maintenance releases are part of a larger effort to support ongoing development and improvement of the software, with the latest open source release being 1.4.2, which is available on the downloads page.

Time series data refers to a sequence of data points indexed in chronological order, and it has unique properties that make it valuable for various applications. Unlike other types of data, time series data inherently has a natural time order, which is essential for certain industries such as finance, meteorology, and machine learning. Time series data is used for forecasting, signal detection, anomaly identification, and managing large datasets from IoT devices, among other use cases. Its primary advantage lies in providing concrete metrics that help users make informed decisions and monitor changes over time. Time series data has become increasingly important in various fields, including engineering, DevOps, and scientific research.

Telegraf is a lightweight, plugin-driven collection agent written in Go that collects data about systems and applications, allowing users to easily contribute functionality without knowing the codebase. It has over 150 plugins, with most contributed by the community, giving it the ability to collect metrics from various inputs and write them to different outputs. Telegraf's input plugins need to satisfy three interfaces: SampleConfig, Description, and Gather, which are used to generate configuration files using TOML. The Trig plugin is an example of a Telegraf plugin that generates sine and cosine waves for demonstration purposes, demonstrating how to create a new plugin by fulfilling the interfaces and importing it into the main Telegraf package. To contribute a new plugin, users need to write tests, a README.md file, a LICENSE file, and sample input/output formats, and follow the excellent guide on GitHub.

Comcast's largest cable TV and internet service provider in the US also offers home security, cellular services, and a commercial broadcast television network. Providing excellent service is crucial, especially under the X1 platform and Xfinity home security services, where dedicated engineers support the company's infrastructure. Rob Frohnapfel from Comcast's Cloud Product Engineering team aims to help his team focus on writing code and delivering products faster, by managing their cloud infrastructure. His team has developed an IaaS and SDS platform using Cloud Foundry PaaS and Kubernetes, hosting thousands of unique applications, virtual machines, and containers with massive storage needs. At InfluxDays NYC 2018, Rob will share how his team uses InfluxDB to make sense of these metrics, identify opportunities, partner with the business, and drive outcomes such as reliable cloud performance and cost savings. With early registration still available at $199, attendees can hear Rob speak in person and learn about the company's success with their multi-tenant cloud metrics solution.

Tom Wilkie, the founder of Kausal, a company hosting Prometheus, will be speaking at InfluxDays NYC on February 13, 2018, about how to instrument services using the RED method, which aims to help developers focus on their code's core functionality and avoid unnecessary instrumentation efforts.

The latest version of Flux (formerly IFQL) has been released with several new features focused on improving the language itself. The release includes support for defining and calling functions within Flux, as well as several new built-in functions such as map, shift, integral, derivative, and difference. This allows users to create small reusable components in their queries and apply logic to the records of the data using well-scoped functions. Functions are a powerful tool that can help build libraries of useful snippets over time. The release also introduces support for multiple values, with the covariance function being one of the first to utilize this feature. The new features aim to bring more power to Flux, and users are encouraged to provide feedback or suggest additional functions through open issues or pull requests on the Flux repo.

Sensu is a popular monitoring solution for both applications and infrastructure, designed to address the needs of a modern cloud computing environment. The Sensu framework is composed of client and server applications that communicate via RabbitMQ by default, but other transports can be used. Configuration is entirely done using JSON files, making it easy to integrate with automation tools like Ansible or Chef. Sensu checks follow the same format as Nagios Plugins, which lets developers take advantage of a vast number of plugins in the Nagios ecosystem as well as those provided by the Sensu community. Checks can be any program or script that writes data to STDOUT or STDERR and returns an error code that corresponds with a given status. Once the check results have been pushed to the message bus, one or more Sensu servers pulls the events from the bus and handles them, processing the results, triggering alerts, or forwarding metrics to a long-term store. Out of the box, Sensu doesn’t do anything with data that might be collected during a check, but provides the ability to configure handlers which will process and forward the data to an external store in our case, InfluxDB. By storing the metrics data, development teams can use the data for analysis at a later date; looking at performance data to drive the engineering roadmap, or as part of the incident response or postmortem process. Additionally, by querying metrics in a time series database, Sensu can perform checks against multiple datapoints, reducing noise and flapping alerts. The InfluxDB instance is not using any kind of authentication or SSL, so we’re using an extremely simple configuration. For this example we’ll be using the CPU Percentage Check from the Sensu CPU Checks Plugin to gather metrics about processor usage. We have two hosts configured running Ubuntu 16.04: the Sensu server itself, which is running Sensu, Redis, RabbitMQ, and Uchiwa, and a second server for InfluxDB. The Sensu server is configured with a legacy-servers subscription in addition to the dev and ubuntu-servers subscription common to both hosts. We’ll use the legacy-servers subscription to ensure that we only run our metrics collection checks on servers without Telegraf installed. We’ll be using the InfluxDB Sensu Plugin, which provides a number of integrations between InfluxDB and Sensu: check-influxdb.rb A monitoring check for InfluxDB via the /pingendpoint; metrics-influxdb.rb A metrics check that uses an InfluxDB query; mutator-influxdb-line-protocol.rb A handler that sends check results to InfluxDB; check-influxdb-query.rb A mutator that transforms check output into InfluxDB line protocol. We can install the InfluxDB plugin to Sensu’s embedded Ruby environment, /opt/sensu/embedded/, using the following command: $ sudo sensu-install influxdb. We’re already running a metrics check on our “legacy” hosts, so we want to set up the the InfluxDB handler to deal with events as they’re received by the Sensu server. First, we’ll enable the handler by adding it to /etc/sensu/conf.d/handlers.json: { "handlers": { "influx-tcp": { "type": "pipe", "command": "/opt/sensu/embedded/bin/metrics-influxdb.rb" } } }. We add a configuration at /etc/sensu/conf.d/influx.json: { "influxdb": { "host" : "192.168.227.134", "port" : "8086", "database" : "sensumetrics" } }. The InfluxDB instance in this example isn’t using any kind of authentication or SSL, so we’re using an extremely simple configuration. We can verify that data is being received by InfluxDB using the InfluxDB CLI. Log into the InfluxDB host and type influx at the prompt: $ influx Connected to http://localhost:8086 version 1.4.2 InfluxDB shell version: 1.4.2 > Use the SHOW MEASUREMENTS command to verify that all metrics have been created: >SHOW MEASUREMENTS name name ---- ---- cpu sensu CPU Percentage cpu_guest sensu CPU Percentage ... We can query one of the measurements to see individual data points: SELECT * from cpu_idle WHERE time < now() - 1m name: cpu_idle time host metric value ---- ----- ------ ----- 1515534170000000000 sensu cpu_percentage 99.5 1515534270000000000 sensu cpu_percentage 100 1515534370000000000 sensu cpu_percentage 100 ... We can use a single template to parse the metrics, as each metric has the same format: host.measurement.field. The InfluxDB instance in this example isn’t using any kind of authentication or SSL, so we’re using an extremely simple configuration. For this example we’ll install Telegraf alongside Sensu and ship Graphite plaintext to it via a socket. We can find installation instructions for Telegraf on various platforms here: Next, we’re going to disable the collection of CPU statistics that Telegraf performs by default, because they are redundant to Sensu’s Metrics Collection Checks. Make sure to comment out the [[inputs.cpu]], [[inputs.disk]], [[inputs.mem]], and [[inputs.network]] sections from the default configuration. We’ll need to configure an input for Sensu to send metrics to as well. We’ll define a socket_listener, give it a port and a data format, and also specify our template for parsing the Graphite plaintext. Add this section to the Telegraf config at /etc/telegraf/telegraf.conf: [[inputs.socket_listener]] service_address = "udp://:8094" data_format = "graphite" templates = [ "host.measurement.field" ] Restart Telegraf to pick up the new configuration using sudo systemctl restart telegraf. We can configure a UDP handler in Sensu with the only_check_output mutator by adding this to your /etc/sensu/conf.d/handlers.json: { "handlers": { "telegraf-graphite-handler": { "type": "udp", "socket": { "host": "127.0.0.1", "port": 8094 } } } }. Update the check so that it uses the new handler by editing /etc/sensu/conf.d/cpu_percentage.json: { "checks": { "cpu_metrics": { "type": "metric", "command": "metrics-cpu-pcnt-usage.rb", "subscribers": [ "legacy-hosts" ], "interval": 10, "handlers": [ "debug", "telegraf-graphite-handler" ] } } }. And finally, restart the Sensu services: $ sudo systemctl restart sensu-server sensu-api sensu-client. Our metrics are coming out of Sensu in this format: sensu.cpu.user 0.50 1515534170 and running through this template host.measurement.field and resulting in one measurement cpu with multiple fields. Let’s check that’s what we’re seeing in the database. Open up the InfluxDB CLI, select the sensumetrics database, show the measurements and select the last minute of data, like we did before: > USE sensumetrics Using database sensumetrics > SHOW MEASUREMENTS name name ---- ---- cpu sensu CPU Percentage ... We can query one of the measurements to see individual data points: SELECT * FROM cpu WHERE time < now() - 1m name: cpu time host metric value ---- ----- ------ ----- 2018-01-09T20:32:09Z 0 sensu 98.51 0 0 0 0 0 0.5 1 ... We can use Telegraf to collect and ship metrics directly to InfluxDB from our applications, which might be a more efficient solution than transforming the metrics in Sensu. Next week we’ll continue exploring the integration of Sensu and InfluxDB by creating a Metrics Check based on the data we’ve captured.

The 2018 DEVIES Award from DeveloperWeek recognizes InfluxData's InfluxDB Enterprise time series platform for its innovation and popularity among developers, as a leader in handling IoT data and metrics, and for attracting press attention. The platform has been selected from hundreds of nominations based on three criteria: developer technologies well-regarded by the community, innovativeness, and press/attention. This award highlights InfluxData's success in empowering developers to build next-generation monitoring, analytics, and IoT applications with its modern Open Source Platform.

Chronograf 1.4.0 and Telegraf 1.5.1 have been released with several key features, including support for multiple organizations and users, customizable Gauge visualization type, and Kapacitor logs accessible within the TICKscript editor. The new release also introduces a time shift feature that allows for week over week or month over month comparisons. Additionally, maintenance releases are available for Telegraf, and binaries can be found on the downloads page.

InfluxData has announced a partnership with ThingWorx, integrating its time series platform to enable developers to store, analyze, and act on IoT data in real-time, and for ThingWorx customers to build monitoring and alerting applications more quickly. This integration will support the analysis of millions of events per second and real-time analytics applications focused on streaming data and anomaly detection. The partnership aims to provide value to both developers and ThingWorx customers by enabling faster and more efficient IoT data management.

Rickshaw is a visualization library built on top of d3.js, used to pull time series data from InfluxDB and visualize it in a graph. Margo Schaedel guides readers through the process of querying InfluxDB, installing Rickshaw, and adding the functionality to paint InfluxDB time series data in a graph. The article provides tips for effective data visualization and references the TICK Stack’s Getting Started guide for further information.

InfluxData has partnered with PTC to integrate its time series platform with Thingworx, a platform designed for industrial innovation, aiming to simplify IoT developer experience and enable faster application delivery. This partnership will make InfluxData discoverable through the ThingWorx Marketplace, allowing customers to use it for building monitoring and alerting applications, as well as real-time analytics solutions. By joining the PTC Partner Network, InfluxData is improving its accessibility and expanding its capabilities, ultimately benefiting IoT developers who can now utilize a more streamlined developer experience to create their applications.

The author, David G. Simmons, was unable to find a Telegraf plugin for monitoring wireless interfaces on his Pine-64 box, so he wrote one himself. The original output from `/proc/net/wireless` had a two-line header that made it difficult to parse, but the author's custom plugin transformed the data into InfluxDB-friendly values. To make the data accessible to Mac users, the author found a similar way to retrieve wireless information using the `airport` command on Mac OS, which resulted in more readable output. The author created two custom plugins for Telegraf and Chronograf to monitor wireless interfaces, one for Linux and another for Mac, allowing them to track the health and activity of wireless interfaces, particularly important for IoT gateway devices that rely on wireless connections.

InfluxData has integrated its time series platform with PTC's ThingWorx to simplify the IoT developer experience, enabling developers to build next-generation applications faster and more efficiently. The integration allows InfluxData to be discoverable and accessible via the ThingWorx Marketplace, making it easier for developers to use InfluxData's time series data platform in their IoT applications. With this partnership, InfluxData aims to solve the time series data challenge for IoT developers, allowing them to focus on building applications rather than dealing with data management. The integration is expected to enable developers to build monitoring, alerting and notification applications, as well as real-time analytics applications that support millions of events per second. This partnership is a significant step forward for InfluxData, which has rapidly built its developer and customer base across various industries, including manufacturing, financial services, energy, and telecommunications.

In a talk by Colin Breck of Tesla, a seasoned expert in time series infrastructures, Time Series Databases are highlighted as a critical operational technology for the enterprise due to their efficient storage, aggregation, and query capabilities. This is particularly relevant for large-scale industrial applications where real-time data analysis is crucial. InfluxDB, a popular Time Series Database, was used by Tesla's team for monitoring, aggregating, and controlling distributed renewable-energy assets, demonstrating its value in operational intelligence. Colin Breck emphasizes the importance of considering Time Series Databases as critical infrastructure beyond just storing data, making them an essential component of enterprise operations.

Time series data refers to a type of data that is collected over a period of time and has a continuous value or measurement associated with it. This category of data exists on its own because it serves a distinct purpose in analytics, often helping organizations make predictions and identify patterns in their operations. Time series data can be used for various applications such as forecasting, anomaly detection, and monitoring, which are crucial for businesses to optimize their performance and decision-making processes. It is essential to distinguish between metrics and events, as they serve different purposes in time series analysis. By understanding the significance of time series data, organizations can unlock valuable insights that inform their strategic decisions.

The author of the text is monitoring MySQL throughput to gain insight into traffic and identify potential problems before they occur. They are using Chronograf, a data visualization tool, to monitor their MySQL database and understand how it's performing. The author starts by visualizing questions and queries in Chronograf, which helps them see trends and anomalies in their traffic. They then investigate the cause of an anomaly in their queries, using InfluxQL to query their data and gain insights into reads per second. Despite facing several personal failures along the way, the author is able to use throughput monitoring to gain valuable insights into their MySQL database's performance.

The company InfluxData has strengthened its position in the DevOps monitoring and IoT market by increasing the depth of its sales leadership team, with former Apcera and VMware sales executive Jay Stephenson joining as VP of Worldwide Sales and Field Operations. This move is expected to accelerate the expansion of InfluxData's product offerings for DevOps and IoT applications worldwide, driven by rapidly growing global demand. With over 15 years of sales experience in Monitoring, DevOps, and Systems Management, Mr. Stephenson brings a wealth of knowledge and expertise to InfluxData, with the goal of driving sustained growth in this next phase of the company.

Nathaniel Cook, lead developer of Kapacitor, will be presenting at InfluxDays NYC 2018, where he will discuss how to use Kapacitor for data processing before storing it in InfluxDB and cover best practices for anomaly detection and machine learning. He has a background as an operations engineer with experience in data science and machine learning, which he applies to his work on detecting anomalies in time series data. Nathaniel's talk will also touch on configuring the clustered version of Kapacitor Enterprise. Early registration is still available at $199 until January ends.

To maintain its leadership position in the fastest growing time series database category, InfluxData's InfluxDB has seen a significant increase in popularity, with a 57% growth over the past 12 months and a 14% quarter-over-quarter growth. According to DB-Engines' latest results, InfluxDB is nearly three times more popular than its closest competitor, outpacing the competition with its user base. This growth is attributed to the increasing demand for analyzing complex environments, as metrics, events, and other time-based data are generated at an exponential rate. The InfluxData Platform provides a comprehensive set of tools and services to accumulate, analyze, and act on time series data, enabling customers to build next-generation monitoring, analytics, and IoT applications. With over 350 customers across various industries, including manufacturing, financial services, energy, and telecommunications, InfluxData is pioneering the shift to time series in modern metrics and events platforms, making it possible for customers to become data-driven and take on digital transformation initiatives.

In order to gain a detailed view of IT operations, organizations are using time-series monitoring tools, which are based on time-series databases optimized for collecting and analyzing time-stamped data at regular or irregular intervals. These tools provide high-resolution views of IT performance, enabling organizations to identify issues before they become major problems. By leveraging these tools, companies such as Houghton Mifflin Harcourt can monitor various types of metrics, gain insights into their systems, and optimize their operations for improved efficiency and reliability.

Katy Farmer, a new DevRel member at InfluxData, will be presenting two workshops at InfluxDays NYC 2018. Her first session focuses on installing and running the TICK Stack, while her second session covers writing queries using InfluxQL. Both sessions aim to help attendees download and use the TICK Stack, as well as write efficient queries. Katy will also share best practices for query optimization and may demonstrate some advanced query techniques. Additionally, she will be sharing snacks, including sour patch kids, with the audience. Early registration before January 31st can secure a reduced price of $199, and attendees can also access nearby hotels and transportation options to make the event more convenient.

The article by Paul Dix from DZone discusses the concept of the "Decomposable Monolith," a design/architecture pattern that allows for easy transition to services-based development, even when starting with a monolithic architecture. The Decomposable Monolith is proposed as a way to structure new projects in a modular and serviceable manner, enabling easier scalability and maintainability. By using this approach, developers can create applications that can be broken down into smaller, independent services, making it simpler to adopt a services-based development model.

Ryan Betts, Director of Engineering at InfluxData, will be speaking at InfluxDays NYC 2018. He has successfully implemented his mission to make InfluxDB simple and powerful for developers to use, with a series of blog posts detailing the internals of the database platform. His talk will cover similar topics, which have been found informative and useful by previous attendees. Early registration before January's end can secure an early registration price of $199, with additional resources provided for attendees, including nearby hotels and transportation options.

This article discusses visualizing time series data from InfluxDB using the Rickshaw library, which is built on top of d3.js. To get started, one needs to ensure they have an instance of InfluxDB running locally and set up a server file using Node/Express to query InfluxDB for data. The server file uses the Influx client library to fetch data from the database, which is then parsed and formatted according to Rickshaw's requirements. The article provides a detailed example of how to create a custom graph with Rickshaw, including the creation of x and y axes, legends, and toggler functionality to switch between different graph formats. The final product showcases a beautifully rendered time series graph that can be customized further using CSS flair.

Sean Porter, creator of Sensu, will be speaking at InfluxDays 2018 about "Data Collection & Prometheus Scraping with Sensu 2.0". He believes that monitoring is a core competency for companies and will discuss how Sensu 2.0 can collect data from heterogeneous environments and store it in InfluxDB. Sean's talk will cover various patterns of data collection, including scraping Prometheus metrics, and enable self-service data collection for service owners.

The New Stack | The Technical Architecture Behind TICK, a Time Series Analysis Platform Using InfluxDB` This article by The New Stack features an interview with Paul Dix, Co-founder and CTO of InfluxData, discussing the technical architecture behind the TICK Stack. The TICK Stack is a platform built from ground up to handle metrics and events, consisting of Telegraf, InfluxDB, Chronograf, and Kapacitor. Decoupling is key to scaling the stack, and time series data is well-suited for event-based architectures. Paul Dix explains how querying the TICK Stack in Telegraf works with time series metrics, as well as use cases for InfluxDB and the TICK Stack. The technical architecture of the stack allows for efficient handling of large amounts of time series data.

CRN has published a list of 10 Data Analytics Vendors To Watch In 2018, which includes InfluxData, selected for its stack and value proposition in IoT, DevOps monitoring, and real-time analytics. The article highlights the shift from traditional static reporting tools to software that can analyze streaming data in real time, positioning these vendors as leaders in big data and business analytics.

InfluxData supports Prometheus monitoring by integrating with Prometheus service discovery and scraping into Kapacitor, enabling real-time analytics capabilities. This integration allows for metric read/write application programming interface interoperability with InfluxDB, facilitating faster issue resolution through access to time-stamped data. The use of InfluxData's solution is particularly beneficial in scenarios where DevOps teams need to quickly identify the source of problems, reducing the time and effort required to resolve issues.