August 2018 Summaries

In a recent article on opensource.com, Dan Barker discusses the importance of monitoring in software development and highlights four open source monitoring tools that can help developers track time series data and metrics aggregation. The article notes that there is confusion surrounding the terminology used in monitoring, which has led to the creation of "poor tools" that claim to offer all-in-one solutions. Instead, Barker focuses on time series data and its primary application in metrics aggregation. He profiles the distinctive features of time series data and introduces several tool options, including InfluxDB Time Series Database, a popular open source solution for storing and managing time series data.

A new maintenance release for Telegraf, version 1.7.4, has been made available. The update includes several bug fixes and improvements to various input/output components of the system, such as the agent, InfluxDB output, PowerDNS input, Burrow input, Ping input, Syslog input, CGroup input, and HTTP output. Additionally, some minor updates have been made to the input tests and authentication credentials are now sent when using the HTTP output. The release addresses several issues with metrics serialization and error handling in certain components of Telegraf.

The renewable energy sector is benefiting from the use of scalable software solutions built with open-source tools, which are helping companies optimize day-to-day operations and reduce costs. These tools are improving processes by leveraging technology to monitor equipment and enhance data management. Open source solutions are particularly well-suited for the renewable energy industry due to their ability to quickly enable innovation and bring ideas to fruition faster. By utilizing open-source tools such as InfluxDB, companies like A&S Energie were able to improve efficiency and make all data continuously available by replacing traditional systems with more efficient ones.

Flux is a data scripting language designed to make querying and analyzing time series and other kinds of data quick and easy, working with data from various sources including InfluxDB, Prometheus, relational databases, CSV files, S3, and any other API or data source. The language's learning curve is comparable to that of learning an API, and its syntax is designed to be familiar for users who have experience with JavaScript. Flux has four main boundaries of functions: input functions for reading from sources, combining tables in the resulting stream, splitting them apart, applying to each table aggregates, selectors, and sorting, and finally output functions for sending results to data sinks. The language aims to reduce cognitive load for developers when working with data, making it readable and understandable even for newcomers, with a design goal of making it look like other popular languages. Early builds of Flux will be released in the coming months, including an alpha version of InfluxDB 2.0 and a separate flux executable.

The TICK Stack is an open-source monitoring system that enables the creation of a modern monitoring system for applications and infrastructure. The authors note that DevOps, cloud computing, and containers have changed how we write and run our applications, requiring a new approach to monitoring logs. Gianluca Arbezzano explains the importance of choosing log monitoring tools and using InfluxDB for log storage in his article on DZone. He concludes by providing guidance on setting up a modern monitoring system using InfluxData's full stack of open-source projects, including the TICK Stack.

This article, written by David G. Simmons, explores how to translate C code to Go using the InfluxDB Go Library and demonstrates its use on a Raspberry Pi, showcasing the ease of getting data into InfluxDB. The author discusses porting a library from C to Go, writing a new library in Go, and utilizing these libraries with the InfluxDB Go Library to store data in InfluxDB. Simmons concludes by highlighting that this project is part of a larger initiative he plans to unveil in the near future.

The open-source community needs to reassess its approach to developing successful open-source software, as it requires continued investment to sustain itself. The community relies on subsidies from individuals' free time or other successful businesses to drive growth. Building a business in the open-source infrastructure space has become increasingly challenging due to cloud providers dominating the market. To adapt, companies are exploring alternative models, such as open core, where they create both open-source and proprietary projects around their core software.

InfluxDB Enterprise version 1.6.2 is now available with several new features including Anti-Entropy service improvements, support for LDAP as a backing store for user permissions, additional TLS configuration options, and back-pressure features to prevent clusters from being overwhelmed with writes. This release marks the first official release of InfluxDB Enterprise on the 1.6 line. Customers can download the binaries directly from the InfluxDB Enterprise portal. The new version also includes all defect fixes from versions 1.6.1 and 1.6.2, making it a great option for community members as well.

The TICK Stack, developed by InfluxData, is an open-source set of projects designed to provide a modern and flexible monitoring toolkit. The stack consists of Telegraf, a metrics collection agent that can be easily installed on servers; InfluxDB, a time series database that stores and manages the collected data; Chronograf, a dashboard for managing and visualizing data; and Kapacitor, a native data processing engine that can take proactive action on system issues. The TICK Stack is designed to simplify the process of monitoring applications and infrastructure, providing a more efficient way to extract information from logs and time series data, and enabling users to easily set up and manage their own monitoring systems.

The author, David G. Simmons, is not an experienced Go developer but decided to learn it by taking four actions: attending Gophercon, porting a C library to Go, writing a new Go library, and using the InfluxDB Go library to store data in InfluxDB. The author successfully ported a BME280 sensor library from C to Go, which was surprisingly easy, and then wrote another library for the SenseAir K30 CO2 sensor. To use these libraries with InfluxDB, the author used the InfluxDB Go Client library and created a batch point every 4 seconds, adding temperature, humidity, and CO2 readings to it. The author's goal is to expand their knowledge of Go and use existing libraries to build a small project using sensors, which will be part of a larger project to be unveiled in the near future.

The author, Paul Dix, argues that the open source community needs to accept the reality that developing and sustaining open source software requires ongoing investment, which must be subsidized by a business or other successful entity. He suggests that this can be achieved through various models, including open core, where a company produces an open source project and then creates proprietary software around it, generating revenue that supports continued development of the open source component. Dix also notes that some companies, like Facebook and Google, produce significant open source projects primarily subsidized by their successful advertising businesses, while others, like Matt Klein's Envoy project, rely on sponsorship from a single large company. The author defends RedisLabs' decision to license certain enterprise modules under a new license, which will prohibit them from being used freely by hosting providers and other software makers that profit directly from them, arguing that it makes sense for the business and ensures the longevity of the open source Redis project. Dix also emphasizes the importance of understanding the motives of organizations driving open source projects and notes that the community should take a deep breath and think about the realities of OSS development, which requires people's time, effort, and passion.

InfluxData has released an upgraded version of its Time Series Platform InfluxDB Enterprise, adding LDAP integration for secure user authentication and advanced analytics capabilities for mathematical and financial modeling. The new release also includes self-healing replication that automatically detects and resolves data inconsistencies. According to InfluxData, time series data is crucial for observability and mission-critical for organizations, making this upgrade particularly significant.

The Cloud Native Computing Foundation (CNCF) has accepted OpenMetrics, an open source metrics exposition format and evolution of Prometheus, into its Sandbox project. This move is seen as a positive step for collaboration and further integration with the broader vendor ecosystem, particularly given Prometheus' widespread adoption in the industry. The Prometheus exposition format will continue to be supported by InfluxData, which will also adopt OpenMetrics as a first-class citizen, moving the standard forward with other open source companies and organizations. The project's progress can be followed on GitHub, where discussions about future developments and contributions are actively taking place.

Eventual consistency is a model used by many distributed systems, including InfluxDB Enterprise edition, which ensures high availability but requires maintaining data accuracy. Anti-entropy (AE) is a service that checks for inconsistencies in the system and aims to minimize data loss. AE works in conjunction with the Hinted Handoff queue (HHQ), which saves data during node outages, to maintain consistency. The HHQ has practical limits, such as storage size and time constraints, and can only address common scenarios. Starting from InfluxDB Enterprise 1.5, AE examines each node's shard distribution and copies missing shards automatically. With the introduction of InfluxDB Enterprise 1.6, AE can also review consistency across nodes and repair inconsistencies. The service relies on at least one copy of a shard being available to perform its heroism, and it cannot compare or repair hot shards.

InfluxData has released an updated version of its InfluxDB Enterprise platform with improved analytics, self-healing capabilities, and enhanced LDAP integration. The new features aim to make it easier for administrators to ensure mission-critical data is available and secure by verifying user authorizations and roles. According to InfluxData's CMO, time series platforms are gaining popularity due to their ability to monitor and analyze IoT, DevOps, Cloud, and Containers, as well as provide real-time analytics for decision-making.

Monitoring PostgreSQL database health requires tracking various key metrics beyond just PostgreSQL-specific statistics. The pg_stat_database view provides information on each database's activity, including connections, commits, and disk blocks read and shared buffer cache hits. The pg_stat_bgwriter view offers insights into the checkpoint process, which is crucial for determining load on the database while it updates or replicates files. Additionally, monitoring system resources such as CPU, network bandwidth, disk space/disk utilization, and RAM can provide a good indication of overall database health. Tracking variables like shared buffer usage, connections, and disk usage can also help identify potential issues with slower queries and application performance. By keeping an eye on read and write query throughput, reliability metrics, and system resource utilization, users can ensure their PostgreSQL databases are running efficiently and effectively.

The new release of InfluxDB Enterprise from InfluxData offers improved LDAP integration, advanced analytics capabilities, and self-healing features to its time series database platform, aiming to enhance system observability and availability in the IoT industry. The update makes it easier for administrators to manage mission-critical data securely by verifying every requested action. With this release, InfluxData aims to provide a comprehensive set of tools and services to help organizations accumulate, analyze, and act on their metrics and events data via powerful visualizations and notifications.

The latest release of InfluxDB Enterprise delivers LDAP integration, advanced analytics capabilities, and self-healing features to its leading time series platform, empowering organizations to build next-generation monitoring, analytics, and IoT applications while preserving data security and privacy. The enhancements make it easier for administrators to manage critical time series data and users by checking and verifying every requested action against a user's stored authorizations and role. Advanced analytical functions enable complex financial modeling, while advanced data replication with self-healing automatically detects and resolves data inconsistencies, providing peace of mind knowing that the data is always available.

In a "metrics first" approach, log analytics aims to provide developers with means to ingest, correlate and visualize all time series data at three levels. According to InfluxData, this approach enables pinpointing valuable contextual information in log files by utilizing metadata that accompanies metrics. The goal is to make log analysis more efficient and effective for developers, allowing them to make better-informed decisions based on accurate data visualization.

You can apply high-performance machine learning models to your time series data using Loud ML and InfluxDB, enabling offline predictions with minimal setup. Docker containers are used to create a bridge network for the InfluxDB, Chronograf, and Loud ML services, allowing for easy configuration and deployment. The tutorial covers how to create and train a machine learning model with Loud ML on your InfluxDB database, predict values, and use Chronograf to visualize those predictions. Additionally, it provides an overview of 1-Click Machine Learning, which simplifies the process of creating high-performing models and evaluating them without requiring extensive knowledge of machine learning for time series data.

Telegraf 1.7.3 has been released with updates including locking of output buffer to prevent concurrency issues, preservation of metric type when using filters in output plugins, and a reduced required docker API version to 1.21. InfluxDB 1.6.1 has also been released with performance improvements such as faster merging of large series sets and improved storage read service performance. Additionally, Kapacitor 1.5.1 includes defect fixes, while the binaries for all releases can be found on the downloads page.

As a new user of Docker, it's essential to be aware of common mistakes and take steps to avoid them. One of the most critical errors is using `docker container ls` instead of `docker ps` to list running containers, which can lead to confusion and inefficiency. Another mistake is creating containers with incorrect configurations, which can result in wasted time and resources. It's also crucial to assign a name to your container to make it easier to identify and manage. Additionally, new users should be aware that containers are not automatically DNS-resolvable and take steps to ensure they are connected to each other. Finally, users should be mindful of file system sharing issues and take measures to resolve them. By following these tips and being aware of the common mistakes made by new Docker users, you can set yourself up for success and navigate the learning curve with ease.

This DZone article by Katy Farmer discusses the importance of gathering metrics in measuring success in game development, highlighting the wealth of statistics that games can provide to monitor performance, and how metrics help tune game performance and improve gameplay.

Chronograf 1.6.1 has been released, featuring several improvements including enhanced dashboard import functionality, updated template variable behavior, and fixes for query replacement and log viewer issues. The new version now allows users to reconcile sources on imported dashboards, updates text type template variables to include a default state of empty, and resolves issues with template variable replacement in queries and display of histogram displays in the Log Viewer. Additionally, pressing play in the Log Viewer will now start live updating at "now", and crosshairs and hover legend are displayed properly when building Alert Rules.

The InfluxDB Java Client has undergone significant improvements in performance and functionality, making it more suitable for mission-critical systems. The latest version (2.10) boasts a substantial increase in ingest rate, with a 40-50% reduction in overhead compared to the previous version (2.9). This is attributed to improved execution speed of certain methods, such as `String.replace(CharSequence, CharSequence)` and enhanced MessagePack support. MessagePack, an efficient binary serialization format, delivers better performance when querying data in this format. The client also features reliability, resilience, and robustness enhancements, including batch writing, write interval jitter, and retry mechanisms for errors like cache-max-memory-size exceeded or unable to parse. These improvements make the InfluxDB Java Client more enterprise-ready and consistent with the behavior of the Telegraf client.