March 2018 Summaries

InfluxData has announced its expansion into EMEA with two new officials: Rob Gillam as EMEA Sales Director and Dean Sheehan as Senior Director of Pre- and Post-Sales. The company's EMEA regional expansion follows its $35M Series C funding round in February 2018, which aims to support global growth in sales, marketing, and R&D. InfluxData continues its worldwide expansion efforts with these new appointments, solidifying its presence in the region.

The influxdb-python library is a Python client for interacting with InfluxDB, a time series database. It allows developers to collect, store and query data in a structured format. The library provides an object-oriented interface to interact with InfluxDB, including creating connections, writing points, querying data, and managing databases. With the influxdb-python library, developers can easily integrate InfluxDB into their Python applications, making it easier to build real-time monitoring systems, IoT applications, and other use cases that require time series data storage and analysis. The library is well-maintained by a community of volunteers and provides additional functionality such as SeriesHelper and DataFrameClient, which provide convenience features for writing points in bulk and integrating with PANDAS and DataFrames.

Chronograf has released version 1.4.3, which includes improvements to its notifications system, generic OAuth2 support, and configuration options for Kapacitor. The new feature release introduces a redesigned notification system that stacks on the right side of the browser, enhancing user experience. Additionally, Chronograf now supports Okta as a generic provider, making it easier for users to configure their applications. Furthermore, users can now configure Kapacitor to use unsafe SSL connections through the UI, facilitating local testing and development.

Batch processing versus stream processing are two different approaches to data processing in Kapacitor tasks. Batch processing involves grouping data points into a specific time interval, querying InfluxDB periodically, and using limited memory. This approach is suitable for cases where aggregate functions need to be performed on the data, downsampling is required, or low latency is not a critical factor. On the other hand, stream processing creates subscriptions to InfluxDB, allowing every data point written to InfluxDB to also be written to Kapacitor, and is ideal for real-time transformations, cases where lowest possible latency is paramount, and high volume query loads on InfluxDB. The choice between batch and stream processing depends on the specific requirements of the task, including memory availability, latency needs, and the type of data being processed.

InfluxData has attracted Jacob Marble, a software engineer looking for a team with nice and smart individuals, as well as a company that values collaboration and interaction. Marble was impressed by the company's products, particularly InfluxDB, which he uses in his personal projects. He appreciates the open-source nature of InfluxData's product code, allowing him to contribute to the community and potentially advancing the technology through collaborative efforts. With a background in Java, Golang, data pipeline frameworks, and API development, Marble is excited to leverage his skills with the company's storage team to improve user experience and reduce Time to Awesome.

I've instrumented my kids' health with InfluxDB, a time-series database that allows for efficient data collection and analysis. I've created an IoT demo using sensors, an IoT gateway, and LoRA boards to collect temperature readings remotely. The data is then sent to InfluxDB, where it's stored and processed by Kapacitor, which triggers alerts and visualizations on a dashboard. This setup provides real-time monitoring of my kids' temperatures without the need for direct contact, making it an ideal solution for parents who want to keep track of their children's health without compromising hygiene. The project showcases the versatility of InfluxDB in monitoring various aspects of life with IoT devices.

The company is expanding its European presence to meet growing demand for its time series database metrics and events platform, which has seen a significant rise in revenue and growth. InfluxData has appointed two key executives, Rob Gillam as EMEA Sales Director and Dean Sheehan as Senior Director of Pre- and Post-Sales, to help expand the company's presence in the region. The company's unique features enable customers to quickly build monitoring, alerting and notification applications, IoT applications, and real-time analytics applications, which are focused on streaming data and anomaly detection. InfluxData has rapidly built its developer and customer base across industries by delivering a modern Open Source Platform that enables customers to derive better business insights, data-driven real-time actions, and a consolidated single view of their entire infrastructure.

In a recent article by Russ Savage, Product Manager at InfluxData, he discusses the various tools available for writing TICKscripts, which are used to monitor and alert on events in Kapacitor, an open-source monitoring and alerting tool. The article covers several developer environments, including Chronograf Web UI, Vim Integration, Emacs Major Mode, Atom Integration, Jetbrains Plugin, and Visual Studio Code Plugin, providing users with the necessary tools to optimize their development process for TICKscripts.

Kapacitor and Continuous Queries: How To Decide Which Tool You Need` Kapacitor and Continuous Queries are two powerful tools in InfluxData's TICK stack, designed to handle different use cases. Continuous Queries (CQs) were created to aggregate data, reducing storage requirements by storing query results in a new measurement, while Kapacitor is a processing engine that can perform more complex tasks such as querying data on a schedule, performing transformations, and alerting based on conditions. To decide which tool to use, consider factors such as the number of CQs being run, the complexity of data transformations needed, and whether downsampled data is required, with Kapacitor often recommended for more complex queries or large-scale use cases, while CQs are suitable for simple aggregation tasks requiring high storage efficiency.

InfluxDB 1.5.1 and InfluxDB Enterprise 1.5.1 have been released with new maintenance updates available. The latest open source release can be found on the downloads page, while customers of InfluxDB Enterprise can download binaries from their portal.

Intelligent Monitoring: Automating Dashboard Annotations in Chronograf` Chronograf has added support for annotating dashboard cells with interesting time-events, allowing users to note events such as abnormally high resource utilization or changes to infrastructure. These annotations are replicated across all dashboards and their respective cells, enabling correlation of events during root cause analysis. Chronograf exposes an annotations endpoint via its API, which can be used by Kapacitor to automatically annotate dashboards based on predefined rules for any metric or event. To leverage this feature, users must configure Kapacitor to POST to the annotations endpoint with a custom payload, which is achieved through editing the HTTPpost section of Kapacitor's configuration file. By following these steps, users can automate dashboard annotations and improve their ability to analyze events in real-time.

InfluxData has contributed a Go implementation of Apache Arrow, a standardized columnar memory format for efficient analytic operations on modern hardware. This implementation is built using c2goasm and SIMD (Single Instruction Multiple Data) optimizations to improve performance. The project aims to provide an alternative way to exchange data between InfluxDB's query processing engine and language, Flux f.k.a. IFQL, and participate in the broader ecosystem of data processing and analysis tools. Apache Arrow is gaining popularity among open source projects and commercial software offerings due to its standardized format, performance, and language-independence features. The Go implementation provides support for memory management, array and builder support, primitive types, parametric types, type metadata, SIMD math kernels, and sum operations for 64-bit float, int, and unsigned int arrays. The project also aims to automate the code generation process using clang and c2goasm, making it easier to generate assembly files for different target architectures.

InfluxData has announced its support for Apache Arrow with a new implementation in the Go programming language, expanding the cross-language development platform's capabilities by providing CPU-specific optimizations to improve performance and analytic workloads. This contribution aims to enhance the overall efficiency of data analysis tasks. The addition of this Go implementation is seen as a significant step forward for Apache Arrow, which is gaining popularity due to its increasing adoption in various industries.

The Apache Software Foundation has announced a new contribution from InfluxData, a company that provides an Open Source Platform for metrics and events data, to the Apache Arrow project by implementing its Go language expertise in this cross-language development platform for in-memory data. This implementation aims to increase performance against analytics workloads and broaden the integration of Arrow-enabled systems. The contribution is part of InfluxData's efforts to support the greater Open Source community and align with the goals of the Apache Software Foundation. By donating its expertise, InfluxData increases the adoption of Go across the industry and provides a highly performant, in-memory format that can be shared between an ecosystem of data analytics tools.

Indexing is a crucial aspect of optimizing database queries for time series applications, as it enables faster search and retrieval of data by creating shortcuts to specific values. However, indexing can become oversized if too many columns or fields are indexed, leading to memory consumption and slowing down processes. To mitigate this, it's essential to be aware of the schema at all times and use indexing judiciously. Another critical factor is query scope, which can significantly impact performance; using time ranges, sub-queries, and filtering data can help limit queries and avoid data overload. Retention policies are also vital in managing high volumes of time series data by setting up processes to delete expired data, while cardinality refers to the number of unique values in a column or series, which can significantly impact performance; optimizing for fewer series with more points rather than more series with fewer points can improve efficiency. By understanding and addressing these factors, developers can craft efficient queries that meet the demands of time series applications.

DZone | Instrumenting Your Node/Express Application: Viewing Your Data``` Margo Schaedel, an InfluxData DevRel, provides guidance on creating a dashboard in Chronograf and exploring data in InfluxDB to gain better insights into a Node.js/Express application. She covers the process of building charts for monitoring CPU usage, event loop latency, heap usage, garbage collection rates, http requests, and database queries. This article is a follow-up to her previous work on instrumenting Node/Express applications, aiming to deliver more effective data analysis and visualization tools.

To monitor a Kubernetes architecture, it's essential to consider both the underlying resources and the services, ingress, and pods deployed. A dedicated namespace for monitoring should be created, and InfluxDB and Chronograf should be deployed on this namespace. Persistent volumes must be managed carefully to ensure data persistence. Telegraf can be used as a daemonset to collect metrics from hosts and services, using a sidecar approach with etcd to expose Prometheus-like metrics. This setup allows for easy deployment and sharing of monitoring configuration, and provides benefits such as optimized network traffic and load on InfluxDB. By using this architecture, Telegraf will follow the application forever, making it easier to manage and maintain monitoring configurations.

InfluxDB version 1.5 introduces an open source backup utility that allows for both online and offline backups, as well as restore operations, without requiring system downtime. The new backup tool can generate portable files in the same format used by InfluxDB Enterprise, while also supporting legacy file formats for compatibility with existing automation tools. Additionally, the restored data is no longer erased from the target system during the restoration process, and the process now imports data through a streaming API provided by `influxd`. The improved restore function can be activated using either the portable or online flags, allowing users to choose between importing data in the new portable backup format or the legacy backup format.

Three maintenance releases are now available for Telegraf, Chronograf, and Kapacitor, each addressing a small number of issues reported by the community and customers. The open source versions of these tools have been updated to 1.5.3, 1.4.2.3, and 1.4.1 respectively, and can be downloaded from InfluxData's downloads page. Enterprise customers of Kapacitor also receive an update with the same fixes, which is available through the InfluxDB Enterprise portal.

The article discusses how to visualize data collected by an instrumented Node.js application using Chronograf, a tool that allows developers to create dashboards to monitor their application's performance. The author uses AmazonBay, a Node.js application, as an example, where they collect various metrics such as CPU usage, event loop latency, garbage collection rates, HTTP request duration, and database query times. By visualizing these metrics in different types of graphs and charts, developers can gain insights into the health of their application, identify potential performance bottlenecks, and make data-driven decisions to optimize their application's performance.

InfluxData is responding to the growing need for IoT tools and mechanics by developing its open source 'metrics and events' platform for building IoT analytics and monitoring applications. The company has joined the PTC Partner Network and now integrates with PTC's ThingWorx IoT package, making its time series data technology accessible to developers through the ThingWorx Marketplace. According to Brian Mullen, VP of business development at InfluxData, all sensor data is inherently time series data due to its temporal nature, which requires efficient storage and analysis capabilities to extract meaningful insights from it.

InfluxDays is a 2-day event that will take place in London, UK in June, featuring talks from industry experts and InfluxDB users, as well as hands-on workshops. The first day will focus on full-day talks, while the second day will be dedicated to workshops for those new to or advanced with InfluxDB. The event will kick off with a talk by InfluxDB's Founder and CTO, Paul Dix, discussing the new query engine IFQL and its progress. Early registration tickets are currently available for £125 until April 6, 2018, offering attendees a unique opportunity to learn from experts and engage with the community.

Sending alerts from Kapacitor via MQTT allows users to bypass the need for a Node-Red instance and send color-coded alerts directly to an MQTT broker. This is achieved by editing the kapacitor.conf file, creating a new TICK script in Chronograf, and using template variables to map temperature values to corresponding colors. The process involves triggering an alert when the change in temperature exceeds a certain threshold, and then posting the resulting message to the MQTT broker under a specific topic. With this setup, users can send alerts from Kapacitor directly to their MQTT broker without relying on external services or JavaScript transformations.

In February 2018, ten hot startups raised VC funding, including InfluxData which secured a Series C round of $35 million led by Sapphire Ventures. The open-source platform enables the development of monitoring, analytics and IoT applications for analyzing time series data such as metrics and events. This funding round was also participated in by other prominent venture capital firms like Harmony Partners, Battery Ventures, Mayfield Fund, and Trinity Ventures.

Optimizing data queries for time series applications is crucial for improving application efficiency and performance, according to Katy Farmer in her article on The New Stack. Common pitfalls such as indexing, query scope, retention policies, and cardinality can prevent database queries from being smart and performant if not addressed. By paying attention to these challenges, developers can optimize their queries and improve overall system efficiency.

Instrumenting Your Node/Express Application` discusses the importance of instrumenting a Node.js/Express application to collect metrics, enabling smarter decision-making for its progress and efficient issue handling. The process involves installing the `appmetrics` library, initializing it in the server file, and collecting CPU usage metrics. Telegraf is then used to send data to InfluxDB, where it can be stored and queried. The article concludes with next steps, including exploring collected data and building a dashboard within Chronograf.

InfluxDB 1.5.0 and InfluxDB Enterprise 1.5.0 have been released, featuring a General Availability of the Time Series Index (TSI) which allows for storing more series than ever before, and enhanced portable backup capabilities that enable seamless data migration between open source and enterprise editions.

InfluxDB 1.5 introduces significant logging improvements to simplify the analysis of log data for troubleshooting purposes. Structured logging allows for easier extraction of specific elements from log events, enabling efficient parsing and analysis with off-the-shelf tooling. The new configuration options improve collating log events and enable HTTP access logs to be written to a separate file, allowing standard HTTP log analysis and monitoring tools to parse the data without further processing. These changes aim to reduce the time required to diagnose issues by providing more valuable insights from log data.

Telegraf, an open-source project from InfluxData, has significantly expanded its ecosystem of supported applications for collecting metrics and events data, with over 80% growth in agents supported in the past year, driven by community demand. The project's ease of use, minimal memory footprint, and ability to enable quick extension by developers have contributed to its popularity. Telegraf supports thousands of metrics, built by over 400 contributors, and is used by more than 400 customers, including major companies like Cisco Systems and Tesla. The platform provides a comprehensive set of tools and services for accumulating, analyzing, and acting on metrics and events data, enabling customers to build next-generation monitoring, analytics, and IoT applications.

In this DZone article, Noah Crowley explains how adding monitoring to your stack can provide visibility into your application, using process monitoring as an example. He provides a step-by-step walkthrough on gathering process counts with the metric collection agent Telegraf and its TICK Stack. The article concludes with next steps, including alerting tips and tool options, aiming to help readers enhance their application's performance and reliability.

Time series data is unique in its characteristics, requiring different considerations for storage and retrieval compared to other types of data. This type of data is voluminous and evanescent, making it crucial to use a database designed specifically for time series data. Scalability is a key benefit of using a Time Series Database, as they can handle high volumes of data with eventual consistency across distributed storage. Additionally, these databases offer specialized query languages that are optimized for accessing time-series data in the context of time, allowing for efficient aggregation and trend analysis. However, this comes at the cost of trade-offs, such as potentially sacrificing durability or atomicity, which must be weighed against the need for speed, accuracy, or volume.