December 2018 Summaries

In this article published by Towards Data Science, Vibhor Nigam demonstrates how to utilize open-source InfluxDB with Spark's structured streaming to process, store, and visualize time series data in real-time. He provides a step-by-step walkthrough for setting up a single node instance of InfluxDB and extending the ForeachWriter of Spark to use it for writing to InfluxDB. The author highlights the efficiency of InfluxDB in data storage and its ease of use, showcasing significant compression capabilities that can reduce large datasets by nearly half.

InfluxDB: How to Do Joins, Math Across Measurements' by Anais Dotis-Georgiou discusses two use cases for performing math across measurements in InfluxDB using the new query language and engine Flux. The first example calculates the batch size of lines written to the database per request, while the second monitors the efficiency of a heat exchanger over time. The article provides detailed walkthroughs and tips for working with Flux, highlighting its capabilities and potential applications.

The text discusses the recent trend of companies adopting community licenses, such as Confluent's switch from Apache2 to their own Community License, and argues that these moves are not about promoting open source principles but rather about protecting business interests. The author criticizes copyleft licenses like AGPL and SSPL, which they believe represent an "evolutionary dead end" for software development, in favor of liberal licenses like MIT and Apache2, which allow for greater flexibility and community engagement. The author also suggests that permissive OSS licenses paired with closed commercial products is a more ideal approach for companies to commercialize open source projects, as it allows for the widest possible adoption and largest possible community.

StreamSets has integrated InfluxDB as its default data platform for handling time series data within its Dataflow Performance Manager (DPM) platform, providing advanced data management capabilities such as Data SLAs and historical performance tracking. This partnership enables StreamSets customers to efficiently manage complex dataflows with optimized handling of time series data, which is a key component in cloud-native application infrastructure workloads. The integration aims to simplify the process of provisioning, testing, and scaling dataflows, allowing companies to remain agile in the face of changing architectures. InfluxData's modern open source platform is designed for analyzing metrics and events, empowering developers to build next-generation monitoring, analytics, and IoT applications faster and easier.

InfluxData's Video Series discusses dealing with large amounts of data, a growing challenge as companies collect more data from devices and endpoints. A time series database is considered a solution to help ingest and analyze this data at speed. According to InfluxData VP of Products Tim Hall, the vast majority of existing data systems are not set up to handle this scale of data ingestion. The video provides insights into how a time series database can address these challenges and enable faster analysis of large datasets.

The time series data market is gaining momentum, with recent entrants like Amazon Web Services (AWS) launching their own hosted time series database, Timestream. This development serves as validation for InfluxData's existing platform, which is a purpose-built solution designed to meet the unique needs of modern enterprises. The entry of AWS into the market highlights the need for robust and specialized solutions like InfluxData's, as building on existing multipurpose databases is not suitable for time series data management. With more competition in the market, InfluxData sees an affirmation of the broader market opportunity and a further validation of its platform's value.

InfluxData has introduced a new language called Flux, which decouples querying and storage, allowing users to define complex queries through functional transformations on data. The language is designed to be more flexible and powerful than traditional query languages, with features such as input and output plugins that enable customization and extension. According to InfluxData's CTO Paul Dix, Flux is not just a query language but a full-fledged programming language that enables developers to build complex applications and workflows. The introduction of Flux aims to simplify data analysis and querying in time-series databases, making it easier for users to extract insights from their data.

Flux, the new query language and engine for InfluxData's time series database, has been compared to Pandas Joins by Anais Dotis-Georgiou. The comparison highlights the benefits of Flux Joins over Pandas Joins, particularly when it comes to readability, flexibility, and efficiency in handling time series data. With Flux, joins can be performed with a more intuitive syntax, eliminating the need for resetting indices or specifying left and right suffixes. Additionally, Flux allows for joining on multiple columns, which is useful for eliminating duplicated or verbose columns in time series data. The author's experience has changed their perception of Pandas Joins, recognizing that they are not optimized for time series data like Flux is.

Chronograf can be securely configured with OAuth 2.0 in Microsoft Azure Active Directory for added user authentication and authorization, allowing users to access the application resource after entering their email and password on the Azure AD login screen. This configuration involves creating a new application within Azure AD, noting down metadata information such as APPLICATION-ID, TENANT-ID, and APPLICATION-KEY, registering a reply URL, and exporting environment variables for Chronograf, enabling user login with centralized security provided by Azure AD. By following these steps, users can securely access the Chronograf Web-UI while maintaining role access based on their Azure AD profile or default settings set by the administrator.

InfluxData's Chronograf now includes improved onboarding that makes it easy to set up predefined dashboards for metrics, which can be accessed quickly and easily without having to think about data visualization. The author uses Telegraf to collect metrics from their MySQL database and sends them directly to InfluxDB, where they can be visualized in Chronograf's dashboard. With the new feature, users can leverage existing metrics with minimal effort, and predefined dashboards are now available for various services such as Kubernetes, Redis, Apache, and more, making it easier to monitor what matters most.

The InfluxData team has released a technical preview of Flux, a new query language and engine for time series data, which enables users to perform math across measurements. The first example demonstrates how to calculate the batch size of lines written to the database per request by filtering data from the telegraf database, isolating write requests and points, joining them together, and calculating the mean batch size over a specified time range. A second example illustrates how to monitor the efficiency of a heat exchanger over time by collecting temperature readings from four different sensors, applying join and map functions to calculate the efficiency across all measurements, and visualizing the results using Chronograf's Flux editor. The new language provides features such as pipes for readability, joins with suffixes, and Map function for complex calculations, enabling users to perform advanced data analysis and visualization tasks.

Telegraf 1.9.1 has been released with improvements to several plugins, including agent and metric memory management, as well as fixes for issues in plugins like graylog input and kafka consumer input. InfluxDB 1.7.2 includes updates on its Flux technical preview, addressing issues with DROPs and DELETEs, and improving sub-query performance. Chronograf 1.7.4 has fixed an issue where string values were being interpreted as numbers in Table graphs, enhanced the Log Viewer to fix scrolling and loading issues, and added a new configuration setting for persistence of topics. Kapacitor 1.5.2 includes features like adding links to notifications sent to PagerDuty v2 API and additional metadata to notifications sent to Sensu, as well as improvements to its barrier node functionality.

Installing a TIG stack on Ubuntu 18.04 involves navigating to the Linux server environment where system admins can access better data about their servers. The TIG stack, consisting of Telegraf, InfluxDB, and Grafana, is being presented as an admin-friendly option for collecting and visualizing server data. To begin, users must update and upgrade their Ubuntu installation before installing the necessary components, including a database, data collection software, and visualization tools. Once installed, users can set up their data source and create a dashboard to visualize their data in a more meaningful way.

The evolution of open source tools is driving changes in cloud native DevOps patterns and practices. Legacy open source-powered toolchains such as Chef, Puppet, Ansible, and others continue to play a critical role, but are being replaced by newer open source-powered alternatives. Industry experts and senior executives note that the status quo can quickly change, with many tools moving to the cloud or being replaced by new ones. Open source platforms like InfluxData's platform are being used for monitoring newly built cloud apps and DevOps toolchains, while also enabling open source-powered workflows for provisioning, securing, connecting, and running any cloud infrastructure.``` The response is neutral and objective as it neither praises nor criticizes the evolution of open source tools in cloud native DevOps. It simply summarizes the key points of the text, focusing on the changes happening in the industry and the role of new and legacy open source tools.

The Developer's Guide to Not Losing the Metrics You Need` discusses the challenges of storing and managing metrics, logs, events, and traces in a distributed system. The author emphasizes the importance of setting retention policies for these data types and using tools like Kapacitor to process and aggregate them. The guide highlights the limitations of time series databases like InfluxDB and the need to separate storage into different "tubes" with varying sizes. It also touches on the challenges of collecting system and application data, including logs, events, and tracing, and how they can be used together to improve debugging, monitoring, and observability. The author shares personal experiences and lessons learned from troubleshooting distributed applications, highlighting the importance of using these tools effectively to make debugging and monitoring more efficient.

In the Flux sandbox, users can create an InfluxDB database and seed it with data using line protocol or the InfluxDB API. The Flux query language is enabled by default in the sandbox, allowing users to start querying their database immediately. Users can use Chronograf's Explore tab to visualize their data and write Flux queries to retrieve specific points from their database. To implement Flux queries in a Rails app, users make a raw HTTP request to InfluxDB's query endpoint using the `net/http` gem, sending their Flux query as part of the request body. The response is then parsed into a usable format using PORO objects and methods.

In a container-based infrastructure, capturing and analyzing metrics is becoming increasingly challenging due to the complexity of modern computing environments. To manage these environments effectively, it's essential to capture and analyze the right metrics, which requires proper storage and management. Modern data platforms are designed to support time series data, providing a solution for system visibility in this virtualized world. By implementing new data platforms and following concrete steps, developers can maximize the benefits of their metrics and gain better visibility into their systems.