July 2023 Summaries

The InfluxDB 3.0 Python CLI is a bolt-on application that leverages the new Flight endpoint and uses the client library as its core development component. It provides a data browsing tool with interactive REPL capabilities, allowing users to query and write data in various formats such as SQL, InfluxQL, and CSV. The CLI also supports writing data to InfluxDB using line protocol and has an experimental feature that utilizes ChatGPT for generating InfluxQL queries. The future development of the Python CLI aims to improve OpenAI functionality, find better packaging and distribution methods, extend write functionality, provide post-query exploration support, integrate delta sharing, and more.

The Python client library for InfluxDB 3.0 is a new development that provides a simple way to write data to and query InfluxDB 3.0, leveraging capabilities of Arrow Flight and supporting SQL and InfluxQL-based queries. The library is designed to be easy to use and provides various configuration parameters, including token, host, org, database, flight_client_options, write_client_options, and **kwargs, allowing users to customize its behavior. It also supports writing data in different formats, such as basic point building, Pandas DataFrame, and file uploads. The library can query InfluxDB 3.0 using SQL or InfluxQL, with various return modes available. Future plans include merging the Write API from the V2 Client, adding prepared statements for queries, improving Polars support, integrating delta sharing, and more.

DevOps and DevSecOps are two methodologies that have gained attention in recent years for their emphasis on team collaboration and software development efficiency. While both share similarities in automation and collaboration, they differ significantly in terms of security integration, expertise, compliance, and risk management. DevOps prioritizes software delivery speed and efficiency, whereas DevSecOps focuses on integrating security practices throughout the software development life cycle. Organizations can combine the principles and practices of DevOps and DevSecOps to build a culture of collaboration, agility, and security. Adopting a DevSecOps approach is critical in today's digital landscape due to growing security threats, improved collaboration, increased customer trust, regulatory compliance, and the need for secure software delivery. By integrating security into their development process, organizations can deliver more secure software faster and respond to business needs effectively.

GitOps is an approach to continuous delivery that emphasizes treating infrastructure as code (IaC) using a single source of truth (SSOT), typically a Git repository. It aims to automate the deployment of applications and infrastructure changes by utilizing Git for managing all deployment-related artifacts. DevOps, on the other hand, is a software development and delivery process that focuses on collaboration between development and operations teams, emphasizing automation tools to improve speed, efficiency, and quality of software development and delivery. While both approaches share similarities, they differ in their core principles and implementation, with GitOps focusing on managing IaC properly using best DevOps practices and DevOps automating the entire software development process from start to finish. The choice between adopting GitOps or DevOps largely depends on an organization's needs and goals, as both methods have their benefits and disadvantages.

Docker secrets provide a secure way to store sensitive information such as passwords, private keys, and API keys in a Docker swarm mode cluster. This approach ensures the immutability of sensitive data by encrypting it across all manager nodes and distributing it to containers associated with relevant services. Docker secrets are encrypted and can be accessed only by individuals with the vault key, which is assigned to service nodes that require access to the information. The secrets can contain various types of data but have a maximum size limit of 500 KB. Docker swarm mode is required for using Docker secrets, and they work in conjunction with services to orchestrate containers across multiple hosts. By utilizing Docker secrets, developers can enhance the security of their containerized applications by storing sensitive information securely.

Observability is the ability to understand how your system behaves at any given moment by collecting various data points pertaining to your system's behavior from multiple sources. It's used for complex systems like distributed systems or microservices architecture, and involves measuring output, collecting metrics, logs, and traces, and analyzing them to deduce internal state or behavior. Monitoring is the process of periodically observing your system and collecting and analyzing certain data points that indicate how the system is functioning, such as CPU usage, memory usage, network traffic, and error rates. Observability focuses on the whole system's scope, current state, proactivity, and granularity, while monitoring has a limited scope to performance indicators, reactive approach, and less granular data. The relationship between observability and monitoring involves using each to detect different types of issues, with observability detecting present problems and monitoring identifying historical bottlenecks. Striking the right balance between both is essential for achieving different goals, such as understanding real-time behavior or diagnosing issues via metrics and alerts.

The Intel team has been contributing to Telegraf since 2020, providing telemetry from various platforms and frameworks such as Intel Resource Director Technology, Data Plane Development Kit (DPDK), Libvirt, P4 Runtime, and Reliability Availability Serviceability (RAS). The Intel PowerStat Input Plugin monitors power statistics on Intel-based platforms, while the Intel RDT Input Plugin collects information from monitoring features of Intel's Resource Director Technology. Other plugins include Intel Performance Monitoring Unit Plugin, which exposes Intel PMU metrics, RAS Daemon Input Plugin, which gathers machine check error metrics, and Intel Dynamic Load Balancer Input Plugin, which provides load-balanced scheduling statistics for DPDK applications. Additionally, Telegraf has plugins for Intel Baseband Accelerator, Libvirt, P4 Runtime, S.M.A.R.T., and Data Plane Development Kit (DPDK) to collect various system and network metrics. These plugins provide insights into platform reliability, utilization, congestion, and configuration issues, which can be used to trigger remediation actions as part of an observability solution in closed-loop systems.

InfluxData discontinued InfluxDB Cloud service in two regions: AWS Sydney and GCP Belgium, causing some users to lose their data. The company has successfully recovered the time series data for the GCP Belgium region and is working to retrieve the data from the impacted AWS Sydney region. InfluxData apologizes for the incident and thanks the Google Cloud Platform team for their assistance in recovering the data. The company is conducting a root-cause analysis to improve its operating processes and will publish the findings on its status page before the end of the month.

Data observability is essential for gaining insights from data, ensuring its accuracy, reliability, and quality, and making informed business decisions. It involves collecting and analyzing data from all system parts, including logs, metrics, and traces, to gain a complete view of the system and identify areas for improvement. Unlike monitoring, data observability provides comprehensive coverage, scalability, and traceability of data, allowing for better analysis of the impact of changes. It facilitates the transition from understanding what's happening to understanding why it's happening, ensuring data quality in terms of accuracy, completeness, consistency, timeliness, validity, and uniqueness. Data observability is important because accurate and relevant data is crucial for business decision-making, and outdated or incomplete data can lead to wasted time and resources, as well as undermine confidence when making data-driven decisions. Implementing data observability enables seamless collaboration between data engineers and scientists, aligns with business digitization, and eliminates data downtime by applying DevOps best practices.

InfluxData has discontinued InfluxDB Cloud service in two regions: AWS Sydney and GCP Belgium, catching some users off guard and resulting in lost data. The company had notified users over several months but acknowledged that not all users were aware of the discontinuation. To mitigate this, InfluxData invested significant effort into informing users and assisting them with migration, including sending email updates, reaching out to customers directly, and updating the homepage of the UI. Despite these efforts, some users were still caught off guard and lost access to their data. The company is now working on restoring data for affected users and has identified areas where it could have done better, such as improving communication processes and providing a 30-day data retention grace period.

This blog post explores the differences between Security Orchestration, Automation, and Response (SOAR) and Security Information and Event Management (SIEM), two key technologies in the cybersecurity domain. SOAR is a suite of tools designed to help security teams manage and respond to cyber threats more efficiently by integrating various security tools, automating repetitive tasks, and providing a centralized hub for incident response and management. On the other hand, SIEM is a technology that aggregates, correlates, and analyzes data from multiple sources to detect threats and manage security events. While both technologies help organizations streamline their security posture, they serve different purposes. By understanding the differences between SOAR and SIEM and evaluating an organization's specific needs, IT teams can make an informed decision about which solution is best suited for their organization. The integration of these two technologies allows security teams to harness the strengths of both systems, contributing to a more effective security posture and increased operational efficiency.

The founders of Ockam and InfluxData, Paul Dix and Mrinal Wadhwa respectively, decided to rewrite their respective products from C and Go to Rust due to issues with the current coding languages. Both Ockam and InfluxDB have undergone significant rewrites in Rust, with Ockam's rewrite being a full transition from C and InfluxDB 3.0 offering improved performance tools such as Apache Arrow, Apache Parquet, and DataFusion. The choice of Rust was driven by its high performance capabilities, fearless concurrency, borrow checker, and async/await tool, which allowed for more efficient and secure development. However, the rewrites came with challenges, including a steep learning curve and lengthy compile times, which required developers to adapt their coding styles and optimize their code for better performance.

InfluxDB 3.0 has been designed to balance data availability and usability with storage costs, allowing users to keep large volumes of historical data without sacrificing data fidelity or storage expenses. The database uses Parquet files as its underlying file storage, achieving significant compression gains and reducing storage requirements on disk. In a distributed environment, InfluxDB can store these files in inexpensive object storage while maintaining high-performance querying capabilities, effectively eliminating trade-offs between data costs, availability, and fidelity. According to real-world customer experiences, InfluxDB 3.0 has been shown to save up to 96% on storage costs compared to its predecessors, offering dramatic benefits over open-source solutions like ClickHouse.

Node-RED is an open-source visual programming tool that simplifies event-driven application development, automation workflows, and integration of various services and devices using Node.js. It offers a browser-based flow editor with a user-friendly interface for connecting nodes to create flows by dragging and dropping elements on the canvas. The tool has gained popularity due to its versatility in diverse domains such as IoT, home automation, data processing, and integration projects. Its features include visual programming, rich node library, flow-based programming, accelerated prototyping, optimized for event-driven architectures, active community support, and system requirements that include Node.js and npm.