September 2017 Summaries

The author of the article is revisiting their previous blog post on building a Telegraf plugin for beginners and has made significant improvements by expanding its functionality to parse multiple values from the JSON data sent by a Particle device, reducing code size by about 80% due to using built-in calls to the Particle Cloud. The revised plugin now handles more complicated JSON structures by creating a "Dummy" structure that can be unmarshaled into and then further processed into a new structure with the desired fields and values. The plugin also enables seamless transition of devices from one database to another without requiring manual data migration, making it suitable for various Particle projects.

InfluxDB Enterprise supports multiple data center replication patterns, including replication on ingest, using Telegraf as a data pipeline, Kafka for durable writes, subscriptions for cross-cluster replication, Kapacitor for aggregations and roll-ups, backup and restore, and the "nuclear option" of storing raw data in long-term storage. These patterns offer varying degrees of ease of setup, flexibility, and consistency, allowing organizations to choose the best approach based on their specific needs and infrastructure.

The development of a successful SaaS Pricing Engine is critical for a company's success and requires careful consideration of several key factors. To build a system that provides nanosecond pricing, companies need to use a time series database to store utilization data, design their architecture for cloud scale to handle the volume of metrics required, and prioritize 99.999% uptime to ensure customer satisfaction. By meeting these requirements, companies can provide granular pricing that meets customers' demands and achieve maximum efficiency for both the customer and the company itself.

grade` is a tool for Go developers to track their Go benchmarks in InfluxDB, allowing them to compare and analyze their benchmark results over time. The tool takes output from a benchmark run and uses it to generate line protocol data that can be stored in InfluxDB. This allows developers to easily visualize and compare their benchmark results, making it easier to identify performance improvements or regressions. With `grade`, users can also make decisions about how often to run benchmarks, what hardware to use, and how to analyze the data they collect.

The author wants to write a Telegraf plugin to interact directly with the Particle Cloud's Webhooks architecture, which allows devices to send data to InfluxDB. The author starts by familiarizing themselves with existing Webhooks plugins and then modifies an existing Rollbar plugin to create their own Particle.io Webhooks Telegraf Plugin. They define a `ParticleData` struct to map the incoming JSON data from Particle to the desired format for InfluxDB, and implement event handling logic in the `particle_webhooks.go` file. The author also adjusts the Particle JSON output to publish temperature data and defines a webhook with the correct URL and settings to send the data to their Telegraf instance. With some help, the plugin was completed in about a day, and the author plans to expand it to parse other fields in the JSON data and put them into the database based on other fields.

In a Docker container, InfluxDB's database needs to be stopped before it can be restored from backup, which creates a catch-22 situation. To address this challenge, an environment was set up using a Docker container with InfluxDB, where the database is backed up when the container shuts down, and then restored when the container restarts. The steps involved capturing the container ID and ephemeral port, backing up the database, dropping the database, stopping the container, running a restore command in an ephemeral container, starting the container again, and querying the restored database to show that the records have been successfully recovered. This approach allows for the restoration of InfluxDB databases from backup while avoiding the need to stop the container, which is essential for maintaining the integrity of the isolated environment provided by Docker containers.

Influx/Days 2017 Speaker: Christine Yen` Christine Yen, CTO and Co-founder of Honeycomb, will speak at Influx/Days about Observability, a topic that has been met with skepticism due to its perceived similarity to monitoring. However, she believes Observability is distinct from traditional monitoring solutions, focusing on empowering users to ask real questions of their data in real-time. With experience building complex systems and a background in analytics and software development, Yen aims to share her insights on the evolving nature of Observability and its potential to address tough debugging challenges in distributed systems.

AWS is sponsoring the Influx/Days reception and has become an Advanced Tier Technology Partner in the AWS Partner Network, highlighting their commitment to helping developers deliver real business value. The company offers a comprehensive cloud platform with over 90 fully featured services, trusted by millions of customers worldwide, including startups, enterprises, and government agencies. This partnership is part of InfluxData's long-standing support for AWS, which has become the world's most adopted cloud platform after 11 years.

InfluxData has recently improved its Docker support for ARM architectures, allowing users to run InfluxDB and other InfluxData products on platforms like the Raspberry Pi with ease. This is made possible by a new manifest format that enables multiple images of different architectures to share the same tag, as well as a method to build multiple architectures from a single Dockerfile. With these improvements, users can now set up a cheap and easy computer for monitoring at home using InfluxDB on a Raspberry Pi. The process involves installing Docker on a laptop or other compatible device, creating a shared network for communication between containers, launching the InfluxDB and Chronograf images, and configuring Telegraf to report metrics to the local InfluxDB instance. While there are some caveats and limitations to consider, especially with older ARM architectures without native float support, users can now take advantage of Docker's improved ARM support to deploy InfluxData products on a variety of devices.

InfluxDB is a high performance Time Series Database designed to handle metrics and events with high write throughput and query rates. The TICK Stack, consisting of InfluxDB, Telegraf, Chronograf, and Kapacitor, provides an open source platform for collection, storage, graphing, and alerting on time series data. This platform specializes in processing timestamped data, offering optimizations such as compression and query functions to address the fundamental issues of high write throughput and query rates. The TICK Stack is designed to be interoperable yet allows each component to provide significant value as a standalone installation. It can be easily set up on macOS using Homebrew package manager, with most configuration values not requiring changes out of the box.

Using InfluxDB and NODE-RED, developers can create a system that brings IoT data to life by visualizing it and making it actionable. The author sets up a NODE-RED flow with an MQTT broker and creates alerts based on temperature readings from sensors. They use Kapacitor to define alert rules and write custom TICK scripts to refine the alerts and make them more precise. The system is refined further by creating an HTTP endpoint that sends color changes to a GlowOrb, which is controlled by the temperature readings. This project demonstrates how IoT data can be transformed into actionable insights and physical effects using InfluxDB and NODE-RED.

InfluxDays is a technical conference that marks its inaugural event, with Grafana joining as a key sponsor and speaker. The event will provide technical content on building applications around time-series data, aligning with Grafana's capabilities for DevOps and IoT analytics monitoring. Joint customers of InfluxData and Grafana use both tools to query, visualize, alert on, and understand their time series data stored in InfluxDB, enabling better solutions for their customers. Grafana's Director, Dan Cech, will speak on a topic related to data visualization and alerting with the company, helping attendees understand how to effectively use Grafana with InfluxDB. The conference covers various software tools and technologies, including IoT applications, real-time analytics, and control systems, offering discounted tickets until September 30th.

The author attended the PromCon conference in Munich where they gave a talk on integrating InfluxDB with Prometheus. They were inspired by the interest in more integration between the two systems and decided to bring native support for the Prometheus remote read and write API into InfluxDB. This new feature allows users to store scraped data from Prometheus in InfluxDB, mirroring the direction of their new data model and query language. The implementation differs slightly from the example remote storage adapter and ignores measurement and field names, instead putting everything into tags. The functionality is currently a beta and requires working with the nightly InfluxDB build and setting up Prometheus configs to use the new endpoints. Once rolled into InfluxDB Cloud, customers will be able to use it as a long-term store for their on-site Prometheus servers or for InfluxDB Enterprise customers, they'll have a clustered, highly available long-term data store for their Prometheus data.

Mesosphere and InfluxData have collaborated to create a new DC/OS Plug-in for Telegraf, enabling container monitoring solutions to collect time series data. The plug-in allows Mesos developers to easily pull metrics and events from any application or service deployed using DC/OS. This integration simplifies the experience for developers using both DC/OS and InfluxData, aiming to capture metrics and events as time series data. The partnership is part of a larger effort to educate and collaborate with customers across different products and technologies.

InfluxData has added support for tracing and Zipkin to its Telegraf and InfluxDB platforms, providing a way to visualize production systems at the microservice level to understand how and why they misbehave. Tracing allows developers to detect the whole path of how services interact, simplifying troubleshooting in microservice architectures. The new plugin collects spans sent by tracing clients and converts them to Telegraf's internal data format, writing the collected data to an InfluxDB database for visualization using Jaeger's UI. The plugin accepts spans in JSON or Thrift formats, uses Annotations tags and fields to track data from spans, and allows developers to run queries against the stored data to retrieve specific information. This addition addresses the need for a contained unit of code that runs a Zipkin HTTP server in the background, collecting spans at regular intervals and writing them to InfluxDB for visualization.

The author, David G. Simmons, built an IoT demo using the InfluxData stack to visualize temperature, atmospheric pressure, humidity, and light sensor readings from a remote environment. The demo uses Particle.io devices with Photon sensors and Chronograf for dashboard visualization. The data is collected by the InfluxData Cloud and analyzed through trend analysis, which helps identify "Aha!" moments such as detecting when a central air conditioning unit cycles on and off based on temperature and humidity changes. This highlights the importance of visualizing IoT data to discover hidden gems and make informed business decisions.