June 2018 Summaries

The TICK Stack is a set of tools used for data collection, storage and visualization. Docker is a widely-used toolchain for Linux containers that provides several benefits over running software directly on a host machine. The author runs Docker on their desktop and Raspberry Pi, using it to deploy and manage the TICK Stack. To run the TICK Stack using Docker, one needs Docker and Docker Compose installed on their local machine. The author uses official images from the Docker Hub for Telegraf, InfluxDB, Chronograf, and Kapacitor. They create a new Docker network, expose ports, mount volumes, and start containers using Docker Run commands. Docker Compose is used to define and deploy multi-container applications, including the TICK Stack. The author provides an example docker-compose.yml file that defines two services: influxdb and chronograf. They demonstrate how to deploy these services using Docker Compose, access logs, execute commands, and tear everything down. The author also mentions the InfluxData Sandbox, a distribution of the TICK stack with a docker-compose.yml file and helper scripts for quick setup.

The article addresses common stumbling blocks faced by new InfluxDB users, providing guidance to avoid pitfalls and navigate the platform effectively. It covers topics such as retention policies, memory usage, timestamp formatting, backup and restoration, and handling write/query loads, offering considerations, resources, and questions to inspire further learning from the documentation. The article acknowledges that the InfluxData team has already provided FAQs and recommends exploring additional TL;DR tech tips for a smoother experience with InfluxDB.

Here's a neutral and interesting summary of the text in one paragraph: To get Telegraf pulling syslog data into InfluxDB, users need to follow steps for their operating system - Mac OS or Linux. The process involves installing HomeBrew on Mac OS and rsyslog on both systems, configuring the rsyslog configuration file with necessary settings such as temporary directories, asynchronous processing, and disk mode, and then restarting the service. On Linux, users install rsyslog via package managers like apt-get, configure the rsyslog.conf file similarly, and restart the service. Once configured, Telegraf is used to accept syslog messages from the rsyslog server, and the data is stored in InfluxDB for visualization using tools like Chronograf.

This DZone article by Margo Schaedel provides an overview of InfluxDB's retention policy best practices, explaining how to create a suitable retention policy for specific use cases. The author discusses the role of retention policies in determining data duration and offers guidelines for setting up optimal data retention durations and shard group durations. By following these best practices, readers can simplify their InfluxDB setup and ensure efficient data management.

The New Stack | InfluxDB Moves to Cloud-Native Architecture` InfluxData has transitioned to a cloud-native architecture for its TICK Stack components, including Telegraf, InfluxDB, Chronograf, and Kapacitor. This move is aimed at improving scalability and flexibility. According to Paul Dix, founder and CTO of InfluxData, decoupling the components in the architecture allows for better performance and reliability. The TICK Stack is designed to handle both regular and irregular time series data, including metrics and events. InfluxData has also introduced a new query language called Flux, which enables users to write more complex queries and analyze their data more efficiently.

The real challenge with IoT lies not in the devices themselves but in managing the vast amounts of data they generate, which is expected to be a "tidal wave" that requires highly distributed data collection, processing, and analysis. The purpose behind IoT devices is to collect and provide valuable insights through data, highlighting the need for a robust data layer to support its deployment. Identifying this data layer is crucial for successful IoT implementation.

The latest maintenance releases of InfluxDB 1.5.4 and InfluxDB Enterprise 1.5.4 are now available, containing important defect fixes that InfluxData recommends applying as soon as possible to ensure stability. The binaries can be found on the downloads page for open source users, while InfluxDB Enterprise customers can download them from the enterprise portal. Additionally, users who have been using TSI preview with earlier versions need to follow specific upgrade steps to take advantage of this feature in production use.

GDPR has led to a scramble among SaaS vendors to get their data practices in order, with some considering offering on-prem versions of their solutions to comply with the regulation's requirements. The costs of implementing GDPR controls can be significant, and building an on-prem version of a SaaS solution is traditionally difficult and cost-prohibitive. However, Gravitational has developed a solution that helps SaaS providers deploy and manage their applications across multiple environments, including on-prem, private cloud, and public cloud, providing multiregional deployments and management for complex applications. This solution uses Kubernetes to deliver availability, portability, and security to its customers, making it easier for them to adhere to regulatory requirements and keep user data private and secure.

Creating an effective retention policy for InfluxDB can be challenging, especially when dealing with time series data that accumulates rapidly. Understanding what a retention policy is and how it affects data storage duration, replication factor, and shard group duration is crucial. When setting up a retention policy, considering the length of use case requirements and designating an optimal shard group duration is essential to maximize performance. The ideal shard group duration should be twice the longest typical query's time range, with each shard group containing at least 100,000 points per group and 1,000 points per series. Tweaking these settings may require trial and error, but finding the right balance can significantly save time in the long run.

I joined InfluxData as a Developer Advocate because I discovered my love for data through a meandering path that led me to Biotech labs, where I worked with robots and realized I wanted to work with data. After attending a bootcamp for Data Analytics and Visualization, I created a dream job checklist that included exploring data, learning about various things, being surrounded by kind nerds, and improving my communication skills. When I learned about InfluxData, it checked all of those boxes, offering a wide range of use cases applicable in almost every industry, allowing me to analyze time series data creatively and share my analysis with others.

InfluxData has launched a new "metrics first" approach to log analysis, which aims to speed up the visualization and analysis of structured application and system events captured via log files. This new capability expands InfluxData's platform functionality with high-speed parsing and ingestion using the syslog protocol, custom log parsing, and pre-built log visualization components. The release also provides additional benefits for developers, such as improved scalability and flexibility in analyzing log data. With this new approach, InfluxData is positioning itself to handle the increasing volume and complexity of log data generated by modern applications.

DevOps is a complex intersection of processes, tools, and philosophy that can be challenging to understand and adopt, especially for smaller companies with limited resources. Adopting DevOps doesn't start with adopting tools, but rather by embracing the underlying values of the DevOps community, which include collaboration, communication, and insight. Effective communication is crucial in DevOps, as it enables teams to share data, collaborate, and build their culture without creating barriers. Smaller companies can adopt successful strategies from larger organizations, but they must adapt them to suit their unique needs and context. By prioritizing collaboration, communication, and insight, smaller companies can successfully implement DevOps practices that improve their code, reduce resources, and enhance their overall culture.

Introducing Flux Editor in Chronograf` InfluxData has introduced a new language, Flux, to query and manipulate time series data, built from scratch to address shortcomings in InfluxQL. The Flux Editor in Chronograf provides an intuitive interface for users to explore their data, visualize queries, and easily make changes. With features like yield functions, temporary yields, and visual diagramming, Flux aims to simplify complex queries and provide a more powerful toolset for time series data analysis. By leveraging Flux, developers can focus on insights and business value rather than logic for querying and combining data. The editor is available now via the nightly build of Chronograf, allowing users to start exploring this new query language immediately.

This DZone article by Margo Schaedel simplifies understanding of InfluxDB, covering retention policies, shards, and shard groups, providing introductory resources for context on time series data, and offering guidance on how to make sense of these concepts in open source versus Enterprise versions.

InfluxData has announced the expansion of its global partner ecosystem, which now includes over 35 certified partners offering consulting, OEM, and technology partnerships. These partners align their products with InfluxData's platform, a data platform specifically designed for time series data. The article highlights the benefits of being an InfluxData partner, such as direct collaboration with the company to streamline integration and operation with its open source and commercial products.

InfluxData has released new capabilities to speed up the visualization and analysis of structured application and system events captured via log files, providing a "metrics first" approach that enables developers to easily ingest, correlate, and visualize all time series data. The platform now supports high-speed parsing and ingestion using the syslog protocol, custom log parsing, and pre-built log visualization components, allowing users to rapidly understand where and how to interrogate large volumes of event data contained within logs without performing expensive search queries. This release is expected to enable faster problem investigation and root-cause analysis capabilities, reducing guesswork and prior knowledge required to sift through log data. The new solution provides improved workflows for log visualization and analysis, allowing developers to easily analyze captured log events for a specific time interval and narrow down data by important metadata elements.

InfluxData has introduced a "Metrics First" approach to log analysis, enhancing observability and enabling quicker problem resolution by speeding up visualization and analysis of structured application and system events captured via log files. The new capabilities are targeted at expanding the functionality of the InfluxData platform and delivering advantages to developers. This approach is designed to improve the overall experience for users and help them resolve issues more efficiently.

The "Metrics First" approach to log analysis emphasizes the importance of metrics in guiding monitoring and management decisions. InfluxData's Telegraf and Chronograf tools are designed to work together to provide a unified view of system health, allowing users to focus on metrics first and then use log data for contextual information. The latest release of Telegraf includes a high-performance syslog plug-in, enabling log collection and storage with any output source that Telegraf supports. Chronograf 1.5 introduces the ability to visualize non-metric data in tabular format, making it easier to quickly identify relevant log data. InfluxData plans to continue developing the Log Viewer tool, adding configuration options and visualization capabilities, while also addressing anti-patterns such as using InfluxDB for long-term log storage. The goal is to improve observability and enable faster problem resolution through a unified metrics-first approach.

InfluxData has expanded its global partner ecosystem to meet growing worldwide demand for its time series database platform, now featuring over 35 certified partners across technology, consulting, and OEM partnerships.` `The company's partner ecosystem is built around its industry-leading data platform designed specifically for time series data, providing a range of benefits for partner companies that align their products with InfluxData's solution.` `InfluxData offers three distinct types of partnerships, allowing partner companies to tailor their collaborations to suit their specific needs and goals.

The Age of Instrumentation` refers to a period where technology is used to collect and analyze relevant data at the right time to inform intelligent decision-making, leading to better business outcomes. The benefits of instrumentation are significant, but only if companies start investing in the right tools now. This shift requires addressing technical imperatives such as using platforms purpose-built for time series metrics and events to drive real bottom-line results. By leveraging these tools, businesses can make informed decisions that lead to improved performance and growth. The key to success lies in starting this journey of instrumentation early on, as late adoption may result in missed opportunities for significant gains.

InfluxData is expanding its global partner ecosystem to meet growing demand for its time series database platform, which enables real-time decision making by monitoring and analyzing large volumes of data streams. The company's partner ecosystem includes over 35 certified partners across technology, consulting, and OEM categories, who provide integration with the platform and leverage it in their unique environments. Partners can draw on InfluxData's expertise to optimize the developer experience, access a large open source community, and accelerate go-to-market efforts. With more than 420 customers worldwide, including major companies like Cisco Systems and Tesla, InfluxData is pioneering the shift to time series data in modern metrics and events platforms, enabling developers to build next-generation monitoring, analytics, and IoT applications faster, easier, and to scale.

Telegraf 1.7.0 has been released, featuring a new feature-bearing release with improvements and additions to various plugins, including the `sysloginput`, `cassandrainput`, `jolokiainput`, `tls_ca`, `tls_cert`, and `tls_key` plugins, as well as 12 new input plugins such as Aurora Input, Burrow Input, Fibaro Input, JTI OpenConfig Telemetry Input, Mcrouter Input, nvidia-smi Input, and Syslog Input. The release also includes new processor plugins like Converter Processor, Regular Expression Processor, and TopK Processor, and new output plugins like HTTP Output and Application Insights Output, which can be used to ingest logs from syslog or rsyslog servers, query the `nvidia-smibinary` for GPU stats, and write Telegraf metrics to Azure Application Insights. The release was made possible by contributions from over 60 open source community members.

The General Data Protection Regulation (GDPR) may pressure SaaS vendors to consider offering on-premise versions of their solutions in order to comply with the regulation's ambiguous terms, which could increase costs for these providers. The challenges involved in building an on-prem version of a SaaS solution are significant, but companies like Gravitational have developed innovative solutions to address these issues, such as Telekube, which can help alleviate some of the burdens associated with GDPR compliance. As a result, some SaaS vendors may be forced to consider alternative delivery models that offer more control and flexibility for their customers, potentially leading to a shift away from traditional cloud-based services.

The information technology industry has developed numerous tools to serve specific needs, but the new Age of Instrumentation requires specialized platforms designed specifically for data instrumentation and analysis. As Mark Herring notes, "Anything that can be instrumented will be instrumented" as companies seek to capture and work with sensor-generated data. The need for purpose-built platforms arises because general-purpose tools cannot adequately address the specific requirements of this activity. These new platforms are now ready to leverage the Age of Instrumentation, enabling companies to effectively manage and analyze their sensor-generated data.

Telegraf 1.6.4 has been released, featuring updates for several plugins including Aerospike Input and Cloudwatch Output. The release includes fixes for known issues such as a memory leak in the Aerospike library and a duplication of values as fields and tags. Additionally, support for unsigned integers has been added to Cloudwatch Output. The update is now available on the downloads page.

InfluxDB is a time series database that uses shards, shard groups, and retention policies to manage its data. A shard group is a container for shards, which store the actual time series data. Retention policies determine how long data is stored in InfluxDB, with policies specifying an expiration date after which data is automatically dropped from the database. Shard groups are closely related to retention policies, as every data point must be associated with a database and retention policy. The open-source version of InfluxDB has a single node instance, making shard groups redundant, whereas the Enterprise version supports clustering and replication, allowing for multiple nodes and shards within shard groups. The replication factor determines how many copies of data are made, ensuring that data is evenly distributed across nodes in a cluster. Understanding these concepts can help simplify InfluxDB management and improve performance.

InfluxDB, The United Nations, and Radiation` The author recently attended a workshop at the Abdus Salam International Center for Theoretical Physics in Trieste, Italy, where they represented InfluxData and EclipseIoT. The workshop aimed to teach participants from diverse backgrounds how to use IoT sensors over LoRaWAN to monitor radiation, with a focus on using InfluxDB and its TICK Stack for data collection, management, analysis, and visualization. The author worked with participants to help them build and deploy applications for remote radiation monitoring, including setting up Telegraf to pull data from an MQTT broker into InfluxDB. The workshop was run jointly with the U.N.'s International Atomic Energy Agency, which monitors compliance with nuclear non-proliferation treaties and leakage from nuclear reactors. The event brought together a diverse group of participants who were working on exciting projects related to IoT and radiation monitoring.

InfluxData's Director of Product Marketing Chris Churilo profiles the use case of PipelineFX, a company helping organizations manage digital media applications and programs by providing a time series solution to solve issues with high-frequency data collection. PipelineFX customers relied on Qube! to make critical business decisions, but needed an alternative to its traditional relational database. By deploying a time series solution, PipelineFX maximized efficiency and optimized existing and future infrastructure, saving time and money for their customers.

The Age of Instrumentation has created unprecedented business opportunities for companies that can capture and analyze new data streams, enabling informed decision-making and competitive advantage. By leveraging Time Series Databases, organizations like BBOXX are able to store, visualize, and analyze large amounts of sensor readings in real-time, driving business value from diverse data sources. The use case of BBOXX demonstrates the potential for optimized metrics and events databases to deliver significant benefits in various industries.