November 2017 Summaries

The deployment of IoT for clean energy is expected to increase in the coming years due to the growing demand for smart homes, according to Michal Knizek from tado, a company focused on home climate control. Tado chose InfluxData as its time series data processing and storage solution, which enabled it to store and process IoT sensor data at scale in a cost-effective way. The use of InfluxDB Cloud also reduced maintenance and administration efforts for tado, allowing the company to focus on its core business. By leveraging IoT technology, tado is fulfilling its mission of providing smart energy management without sacrificing comfort.

This Jax article provides helpful tips on keeping Docker running securely in production, presented by Gianlucca Arbezzano during DevOpsCon 2017. Security is a core issue when using the Docker ecosystem, particularly with Docker Swarm, and Arbezzano covers this topic through setting up a production environment, immutability, and related security concepts.

InfluxData was featured in CRN's article "The 10 Coolest Big Data Startups Of 2017" as one of the top 10 startups developing leading-edge software and systems for big data. The article highlights the growing need for automated, real-time, and mission-critical big data systems, and InfluxData CEO Evan Kaplan was mentioned in the profile. The selected startups were chosen for their innovative solutions to meet the demands of modern big data systems.

InfluxData's CMO Mark Herring discusses the need for companies to equip development teams with advanced software solutions built to foster innovation and developer happiness, highlighting the evolution of instrumentation that has created the need for purpose-built monitoring platforms to monitor, analyze and act on metrics, events, and other time series data. As workloads increase, choosing a suitable monitoring platform is crucial, with new monitoring requirements emerging in today's landscape. Herring emphasizes the importance of equipping development teams with advanced solutions to drive innovation and happiness.

The Linux Foundation has announced 22 new Silver Members, including InfluxData, a company that delivers an open source Time Series Data Platform built from the ground up for metrics and events. The Linux Foundation is a nonprofit organization enabling mass innovation through open source, where members help support development of shared technology resources while accelerating their own innovation through open source leadership and participation.

Building a custom service input plugin for Telegraf to monitor keywords from a Twitter live stream and send them into InfluxDB. The first step is to set up the TICK Stack, including Telegraf, InfluxDB, Chronograf, and Kapacitor. Developer environment setup involves installing Go and configuring it for development. A new directory is created in the plugins folder with a file named twitter.go, where the code for the plugin will be written. The plugin configuration and Twitter API keys are set up, including Consumer Key, Consumer Secret, Access Token, Access Token Secret, and a list of keywords to track. The Start and Stop functions are added to turn the plugin into a service input plugin. The fetchTweets function is used to continuously listen for new tweets from the Streaming API and send them to InfluxDB. After completing the coding, the build instructions are followed to create a config file and run Telegraf with the new plugin. Finally, the results are explored in Chronograf to verify that everything is working as expected.

InfluxDB is queried using a SQL dialect called influxql, which focuses on the query engine's internal workflow rather than the language semantics itself. The database supports two main querying patterns: windowing and searching for specific points, both of which filter data by criteria applied to dimensions stored as tagsets. InfluxDB maintains an index to make filtering by tagsets efficient, storing mappings of measurement names to field keys, series IDs, and tags. The current default index is stored in-memory, but a new index structure called Time Series Index (TSI) is being developed to store the index on SSD, allowing for higher cardinality datasets. The query engine parses, plans, and executes queries by determining the type of query, separating time range and condition expressions, and validating semantic correctness. InfluxDB uses flexible schema-on-write tagsets instead of pre-defined dimension tables in a star-schema, making it different from traditional SQL columnar OLAP databases. The database supports retention policies to enforce time-to-live policies against data, but also allows for DELETE and DROP statements to remove unwanted points, which can be expensive due to the need to undo work on disk. InfluxDB does not support an UPDATE statement, but re-inserting a fully qualified series key at an existing timestamp will replace the old point's field value with the new one.

We had a great time at our inaugural InfluxDays 2017 event in San Francisco, featuring industry-based content and networking opportunities, with speakers from top companies including Honeycomb.io and Grafana. Our users learned about the new query language, Influx Functional Query Language (IFQL), and contributed to its development during a workshop. We also introduced several new features in InfluxDB 1.4 and Telegraf 1.4.4, including performance improvements and bug fixes. The event was successful, with great feedback from users and sponsors, and we're now looking forward to InfluxDays 2018 NYC, scheduled for February 13-14, 2018, with a call for papers open and registration available at a discounted rate.

Today, InfluxData has joined the Cloud Native Computing Foundation (CNCF), a non-profit organization that sustains and integrates open source technologies such as Kubernetes and Prometheus, to contribute its expertise in time series analytics of metrics and events. Through its membership, InfluxData aims to work with other CNCF members to define and support a standardized metric format, which will give the industry a consistent way to track and define metrics in a vendor-neutral manner. This standardization is expected to reduce vendor lock-in and make it easier for developers to build tools for observability. InfluxData's commitment to open source and driving adoption of standards around metrics will enable developers to solve real business problems faster.

Monitoring is crucial for effective continuous delivery (CD) models as it enables organizations to adopt a proactive approach and meet their service level agreement (SLA) requirements. To achieve this, it's essential to implement the right monitoring strategy that balances efficiency, scalability, and measurability. InfluxData CMO Mark Herring outlines five best practices for DevOps monitoring that facilitate informed business decision-making, allowing organizations to make data-driven choices and stay ahead in their industry. By adopting a proactive approach to monitoring, organizations can avoid an interrupt-driven approach and ensure seamless delivery of services.

The InfluxDB team has released an early alpha of a new query language and engine called Flux, which is designed to work with InfluxDB 1.4 and is compatible with the latest version. Flux is based on JavaScript and is intended to be more natural for time series data than SQL. It features a functional paradigm, named parameters, and a special expression language for specifying predicates. The goal of Flux is to decouple the query engine from the storage tier, allowing for future work on workload isolation and data science tasks. Flux is open-sourced under an AGPL V3 license or commercial license, and the team plans to continuously add features and release them regularly.

InfluxDB 1.4 introduces several significant enhancements, including improved InfluxQL capabilities, support for Prometheus read and write endpoints, and performance improvements to compaction. The new version also includes features such as SHOW CARDINALITY queries, EXPLAIN and EXPLAIN ANALYZE commands, and support for unsigned 64-bit integers. Additionally, the InfluxDB CLI now supports Ctrl+C to cancel a running query, and message pack formats can be used for responses by setting `application/x-msgpack` in the Accept header. The release also includes experimental features such as TSI progress, IFQL prototype interfaces, and other changelog tidbits.

@ThingsExpo Journal | Accelerating IoT Deployments to Achieve Business Goals Faster` This article by Mark Herring discusses the potential of IoT to measure and analyze behavioral statistics, highlighting the need for real-time insights to facilitate automation. A Time Series Database (TSDB) is identified as a key component in enabling developers to work with the diverse data requirements often found in IoT deployments. By leveraging an IoT data platform, developers can gain actionable insights from their data, ultimately driving business goals forward faster.

Telegraf 1.4.4 has been released, marking a maintenance update that addresses various bug fixes and improvements to the popular monitoring tool.` `The new version is now available for download, bringing enhanced stability and reliability to users.` `This release is considered a minor update, focusing on resolving issues and fine-tuning performance without introducing any major changes or breaking existing functionality.` `Telegraf 1.4.4 can be downloaded immediately from the official website, allowing users to take advantage of these improvements and maintain their monitoring setup.

Percona Live Dublin recap` I was a speaker at Percona Live in Dublin on September 26, 2017, where I presented about the InfluxDB Internals. The event featured over 140 speakers covering topics such as MySQL, MongoDB, Elasticsearch, and use cases for large databases from various companies. I expected to hear more opinions about using traditional engines for time series data, but instead, attendees were curious about the advantages of an engine designed specifically for this type of data. My talk focused on compression algorithms, retention policy, and sharding, which received positive feedback. The conference provided opportunities to speak with database administrators working with traditional databases, gathering valuable stories and scenarios that can help improve InfluxDB's performance and maintainability.

Chronograf is a powerful tool for creating graphs and dashboards that can be easily shared with others. The Chronograf REST API allows users to download their dashboard and share it with colleagues, who can then upload it to their own instance of Chronograf. To get started, users need to find the ID of their dashboard by listing all available dashboards and identifying the desired one. Once the ID is obtained, the JSON file for that dashboard can be downloaded and shared via URL or command line. The recipient can then upload the dashboard to their own Chronograf instance using a similar command line process. This allows users to share their custom dashboards with others while maintaining control over the data and configuration.

InfluxData has been named as the leader in the fastest growing time series database category, according to DB-Engines, with its popularity increasing by 67 percent over the past year, nearly three times more than its closest competitor. The company's Time Series Database, InfluxDB, is widely used for metrics, events, and IoT sensor data, and has a large customer base across various industries. According to Mark Herring, InfluxData CMO, the growth of time series databases is driven by the exponential increase in data generated from physical sensors, virtualization, and microservices, necessitating real-time results that business requires. InfluxData's platform provides tools and services for accumulating, analyzing, and acting on time-based data, enabling customers to build monitoring, IoT, and analytics applications. With over 350 customers, including major companies like Cisco Systems and Tesla, InfluxData is pioneering the shift to time series in a modern metrics and events platform, making it possible for customers to become data-driven and take on digital transformation initiatives.

The author, David G. Simmons, wanted to measure and monitor community engagement on Discourse with InfluxDB. He found that Discourse provides some rudimentary monitoring, but using their APIs to get data in a new way was more effective. After reverse-engineering the API and parsing JSON objects, he wrote Node.js code to fetch the data and insert it into InfluxDB. The author successfully automated the collection of statistics, such as user visits, topics with no response, and cumulative totals, which can now be visualized on a dashboard for management to see. This project has freed up time for the author to focus on other tasks, and they invite readers to join their community and start participating.

Vagrant is a tool for building and managing virtual machine environments in a single workflow, created by HashiCorp. To spin up a Vagrant TICK Stack, you need a Vagrantfile, a bootstrap script, and any desired files. The Vagrantfile defines parameters such as memory, mount points, and network settings, while the bootstrap script installs the TICK Stack components, including Telegraf, InfluxDB, Chronograf, and Kapacitor. You can customize the versions of the TICK Stack installed by modifying the version numbers in the bootstrap script. Once the VM is started, you can use it to test and develop your TICK Stack projects using Vagrant's wrapper scripts or simply running `vagrant up`.

Grafana, an open-source visualization tool for time series data, can be effectively used with InfluxDB, a time series database built specifically for storing time series data. The integration between Grafana and InfluxDB allows for a seamless setup, where Grafana calls the InfluxDB API to query data. To set up Grafana monitoring with InfluxDB, users need to download both applications and configure them accordingly, including setting up a collection agent such as Telegraf. It's recommended to use the latest browsers for best performance. Users can customize their graphs by selecting fields, aggregators, and transformations, and optimize time series monitoring by using collapsible rows, reducing queries, and caching data. Additionally, users should monitor Grafana security settings, including enabling anonymous access and password resets, and configure logging options to prevent errors and improve performance.

Kapacitor is an integral piece of the InfluxData platform, and it's evolving to handle more data processing workloads, including alerts and downsampling or aggregations. The most common type of workload for Kapacitor is alerts, which can be created using Chronograf to visually design them, allowing users to easily copy and edit the TICKscript. For downsampling, Kapacitor recommends running aggregation scripts as batch scripts, with the period() function determining the interval between runs, while the every() function determines how often data should be collected. Downsampling can be done at various intervals, such as from 30 seconds to 5 minutes or 1 hour. To effectively develop TICKscripts, it's recommended to set up a GitHub repo and/or Wiki for sharing templates and guides.