August 2020 Summaries

This internship at InfluxData provided a great learning experience for the author, who tackled issues with the Flux language, focusing on optimization. The author dispels three common myths about working in the tech industry: one needs to have all the skills before joining, can make a difference in the company, and form close relations through remote working. The author shares their personal journey from studying Business to becoming passionate about computer science and technology. They also highlights the amazing culture at InfluxData, which promotes a sense of family among employees, encourages collaboration, and has a focus on creating a cohesive community. The author participated in various projects, including optimizing queries, re-designing the Parser, and hosting a SlackBot slash command. Despite remote working, they formed close relationships with colleagues and had interesting conversations with team members from different functions. Overall, the internship was an amazing experience that helped the author grow both personally and professionally.

InfluxDB is a time series database that requires careful consideration of data layout and schema design to optimize resources, ingest rates, query performance, and security. General recommendations for schema design include encoding metadata in tags, limiting the number of series or reducing series cardinality, keeping bucket and measurement names short and simple, avoiding encoding data in measurement names, separating data into buckets when assigning different retention policies, minimizing instance size by optimizing data layout, and applying downsampling to reduce storage and query performance. It is essential to consider user experience (UX) and security when designing the schema, as well as common mistakes that can lead to runaway series cardinality, such as storing logs or IDs as tags. By following best practices and taking a strategic approach to schema design, users can optimize their InfluxDB instance for improved performance, scalability, and resource efficiency.

A new maintenance release for Chronograf is now available, featuring improvements in security, data explorer, predefined dashboards, TICKscript editor, and other bug fixes. The update includes enhancements such as added security measures to protect against XSS attacks, improved data exploration capabilities, and updated features in the TICKscript editor. Additionally, this release provides bug fixes for various issues, including sorting problems with dashboards and retention policies. For community members, the latest version can be downloaded from the official website.

This internship at InfluxData provided a welcoming environment where the intern felt supported and encouraged to ask for help when needed. The intern gained valuable technical skills such as understanding of GitHub, React, and Go programming languages, as well as experience in working with large codebases. They also developed problem-solving and self-reliance skills, learned to effectively communicate with teammates, and took ownership of their internship experience. The company culture was friendly and inclusive, with the executive team taking time to explain their roles and each team member making an effort to get to know the intern.

InfluxDB provides several graph type visualizations to allow users to easily monitor their data. However, most of those graph types are only helpful if your data can be represented numerically. The mosaic graph type is well-suited to tracking status over time by plotting the change in the value of a given category of data over time, with colored boxes representing certain values and the length of the box representing how long the value lasts. A team of interns created this visualization type during their summer internship, facing challenges such as organizing data into intervals, handling string values, and implementing the graph's backend. The project taught them valuable lessons about understanding complex codebases, quantifying work, and the importance of mentorship and support from their team.

Throughout the summer, Rose Parker, a remote intern at InfluxData, gained valuable insights from the people she met and worked with. She received advice on the importance of having a code of conduct, not forcing-push unless absolutely necessary, finding mentors in her career and life, not solving an issue before it arises, and prioritizing activities that bring joy to her daily routine. These lessons helped shape her experience at InfluxData and provided valuable takeaways for anyone looking to navigate the tech industry with integrity and purpose.

InfluxDB Cloud 2.0 is a time series database as a service that can be prone to errors, especially for beginners. The most common error `specified column does not exist in table: _time` occurs when using aggregate functions like `count()`, `mean()`, or `median()` on columns without the `_time` column. To fix this, users should use `aggregateWindow()` instead of the raw function and preserve the `_time` column. Another error, `unsupported aggregate column type string`, happens when trying to apply aggregators on columns with string types; fixing this requires converting strings to numerical values using functions like `int()` or `float()`. The third error, `failed to initialize execute state: could not find bucket "bucket-name"`, is caused by accidentally including a whitespace in the bucket name. To fix this, users should use the Flux Query Builder and switch to the Flux Script Editor to verify their bucket exists without white spaces.

The latest maintenance release for InfluxDB OSS is available now, with improvements including optional feature to add custom HTTP response headers, performance enhancements such as parallel query planning and batching of tombstone writes, and fixes for various issues like Flux regression and TSI edge cases. Additionally, a new maintenance release for InfluxDB Enterprise 1.8.2 has been released, featuring hinted handoff queue improvements, increased throughput during recovery, and improved logging. The release also includes updates to influxd-ctl CLI, LDAP configuration, and fixes for cluster read service issues.

InfluxDB Cloud has achieved significant performance improvements in aggregations, query response times, and overall dashboard speed, thanks to data locality breakthroughs, automatic optimizations, and new features such as auto-aggregations, pause functionality, and improved visualization capabilities.

The array.from() function in Flux is a new method introduced by the community that allows users to construct tables on the fly from an array of objects, overcoming limitations associated with alternative approaches such as manually editing Annotated CSV files or using the influx write command. This function writes data with timestamps in a human-readable RFC3339 format and saves time by not requiring authentication parameters, making it useful for constructing new temporary data, generating specialized tasks, and testing tasks to validate their operation.

The author created a system to track Fortnite player performance using InfluxDB, Telegraf, and Python. The system consists of three main components: data gathering, dashboards, and alerting. The author used the InfluxDB template for Fortnite, which includes pre-built dashboards and query variables. The system can be customized to track specific players or teams, and alerts can be sent when a player wins a match. The author provides a step-by-step guide on how to set up the system, including installing Telegraf, Python, and InfluxDB, as well as configuring the system for data gathering and alerting.

The tutorial covers three methods for writing data to InfluxDB 2.0: uploading data via the Chronograf UI, importing data directly into InfluxDB OSS using a CLI command, and using Telegraf with the Tail plugin for InfluxDB OSS and Cloud. The tutorial provides step-by-step instructions for each method, including setting up the necessary configuration files and navigating the InfluxDB interface to create buckets and import data. The author also addresses common issues and provides troubleshooting tips to help readers get started with writing data to InfluxDB 2.0.

This tutorial provides a step-by-step guide on how to stream data into InfluxDB v2 using Telegraf with two different plugins: the Exec Plugin and the Tail Plugin. The tutorial uses the Alpha Vantage Stock and Crypto API to retrieve intraday time series data, which is then written to InfluxDB in line protocol format. The tutorial covers the configuration of both plugins, including specifying the precision of the data, setting the interval for data collection, and omitting the 'host' tag. The tutorial also provides examples of how to write real-time data to InfluxDB using Python scripts with both plugins.

InfluxDB 2.0.0 Beta 16 is now available for download, featuring potentially breaking changes such as removing the /packages route from the CLI and requiring defaults when using environment references in Templates. This release includes enhancements to the CLI tooling, including managing dashboards and improving query capabilities, as well as improvements to InfluxDB UI performance and Flux Query Engine upgrades. The release is intended for testing and feedback purposes only and requires users to join the InfluxDB Community Slack channel for support and assistance.

InfluxDB provides a monitoring workflow that involves querying data, assigning a status level based on specific conditions, storing the output in a measurement, and sending notifications to an endpoint based on notification rules. To create a check, users can navigate to the Alerts tab, select a threshold or deadman check type, name their check, define aggregation functions, configure thresholds, and assign custom statuses. Users can also view the output of their checks by navigating to the default _monitoring bucket in their InfluxDB instance. If data transformation is required before creating a check, users can create tasks to perform data transformations and then create a check on the transformed data. This allows for more complex data analysis and notification rules, enabling users to add custom statuses beyond the standard four levels (CRIT, WARN, INFO, OK).

InfluxDB provides several ways to monitor states with Flux functions, including stateDuration(), stateCount(), monitor.stateChanges(), and monitor.stateChangesOnly(). These functions allow users to determine how long a state persists, count consecutive states, detect changes in specific columns, and calculate average charging cycle times. Additionally, users can apply custom statuses using tasks and use the rename() function to modify column names, although this feature enhancement is currently not available.

InfluxDB helps retail organizations prepare for the Cyber Five Weekend by providing real-time analytics and insights into their data. Retail companies rely on InfluxDB to monitor 700+ terminals in shops, improve online customer experience, and detect issues in real-time. Wayfair, a large online retailer, uses InfluxDB to collect website traffic data, track customer activity, and monitor performance metrics. The company's IT infrastructure is designed to handle high volumes of data during the Cyber Five Weekend, with multiple data centers and cloud services providing redundancy and scalability. By using InfluxDB, Wayfair can improve its customer experience, reduce downtime, and increase sales.

Telegraf 1.15 introduces a new Starlark Processor Plugin that allows users to perform various operations in Telegraf using the Starlark language similar to Python, enabling real-time aggregates on data before sending it to InfluxDB. The Defaults Processor Plugin ensures certain fields always exist with a specified default value, solving the problem of missing data. A new Execd Output and Processor Plugin is released, allowing external programs to run as daemons and continuing to stay running, providing flexibility in Telegraf operations. Other plugins include Network Interface Name Processor, Port Name Lookup Processor, Reverse DNS Processor, NGINX Stream STS Input Plugin, and Redfish Input Plugin, which gather metrics and status information about physical server metrics like CPU temperature, fan speed, power supply, voltage, hostname, and location details. The New Relic Output Plugin writes to New Relic Insights using the Metrics API, allowing for dual-write capabilities with other output plugins. Additionally, Telegraf 1.15 includes enhancements such as authentication support for the HTTP Response Input Plugin, older version compatibility for the Microsoft SQL Server Input Plugin, and additional fields for the MongoDB Input Plugin.

The Telegraf Starlark Processor Plugin allows users to perform various operations on their data using the Starlark language, which is a dialect of Python. The plugin provides capabilities such as math operations, string operations, renaming tags, logic operations, and scaling metrics. Users can write custom Starlark functions to transform their data, calculate percentages, scale bytes to kilobytes, megabytes, or gigabytes, and even rename tags. Additionally, the plugin supports conditional logic and dropping metrics based on numeric conditions. With this plugin, users can enhance their data collection and processing capabilities, making it easier to work with time-series data in Telegraf.

Telegraf 1.15.2 has been released as a maintenance update, addressing issues with RPM permissions and the Tail Input Plugin in the Agent plugin, while also providing security and extensibility enhancements in other related products such as Chronograf and Kapacitor.