June 2025 Summaries

Integrating OpenObserve with Splunk via the HTTP Event Collector (HEC) allows users to combine OpenObserve's real-time observability with Splunk's advanced analytics and visualization capabilities. This integration involves enabling HEC on a Splunk instance, configuring OpenObserve to send data, and creating pipelines to forward log data from OpenObserve to Splunk. The process benefits various use cases, such as correlating logs from microservices with firewall data, centralizing observability data for long-term retention and compliance, and leveraging Splunk's analysis tools. The setup requires a working Splunk instance with HEC enabled, a configured OpenObserve, and a Splunk token with the proper HEC input configuration. Once these prerequisites are in place, users can validate event ingestion in both OpenObserve and Splunk to ensure the seamless transfer of data, troubleshooting common errors like incorrect tokens or invalid JSON payloads if necessary. This integration not only simplifies the management of logs across platforms but also enhances the analytical capabilities available to teams.

Avatar, a transportation technology company, faced significant challenges in achieving SOC 2 Type II compliance due to a fragmented logging system that impeded operational efficiency and compliance. The company hired Graham Brooks in April 2024 to address these issues, specifically the need for a consolidated and compliant logging infrastructure. After evaluating several platforms, Avatar selected OpenObserve for its use of standard SQL, flexibility, and exceptional customer support, which allowed for a seamless integration into their existing systems without extensive training. OpenObserve's capabilities enabled Avatar to unify their observability strategy, simplifying troubleshooting and enhancing system reliability, while also providing cost-effective data processing compared to other platforms. This new setup not only helped Avatar achieve SOC 2 Type II compliance but also improved their application server availability and laid a foundation for continued growth and infrastructure scalability.

OpenObserve has announced a significant update to its pricing policy, transitioning to a fully usage-based model effective June 2, 2025, with no minimums, which aims to provide more cost-effective options for users ingesting smaller data volumes or utilizing limited platform features. The update includes discontinuing the free-tier Cloud service, requiring current users to switch to a paid plan within 30 days, while new customers can access a 14-day free trial. The new pricing structure includes specific rates for data ingestion, query volume, pipelines, session replays, error tracking, and action scripts, removing the flat monthly fee for current Standard-tier customers, who will also transition to the pay-as-you-go model after a 30-day exemption period. OpenObserve emphasizes that this change aligns costs with actual usage, promotes scalability, enhances transparency, and supports ongoing platform development. Users preferring not to incur charges can opt for the free self-hosted open-source version, which offers complete control with no usage fees. OpenObserve maintains its commitment to open source and provides support to users transitioning to the new model, ensuring they understand the pricing implications and options available.

Leveraging Amazon DynamoDB and Amazon Kinesis Data Streams allows developers to efficiently monitor item-level changes for applications requiring real-time data processing, such as financial systems or mobile apps. DynamoDB, a fully managed NoSQL database, excels in delivering fast, predictable performance, making it suitable for applications like gaming and IoT. Kinesis Data Streams captures modifications in DynamoDB tables, replicating them for near-instant analysis, while Kinesis Data Firehose facilitates the transformation and delivery of streaming data to AWS services like Amazon S3 or OpenObserve. The setup process includes creating a DynamoDB table, populating it with data, and connecting a Kinesis Data Stream to monitor changes. Real-time monitoring is further enhanced through OpenObserve, which enables the visualization of changes, creation of alerts, and generation of time series charts. Best practices involve optimizing shard management for performance, ensuring data consistency, and implementing security measures through IAM policies. This monitoring framework supports use cases such as real-time analytics, data synchronization, and event-driven application workflows, ultimately enhancing the responsiveness and efficiency of data-driven applications.

Anomaly detection in machine learning involves identifying unusual patterns or outliers in data, which is crucial for applications such as cybersecurity, finance, IT operations, and real-time monitoring. It uses algorithms like Random Cut Forest (RCF) and autoencoders to automate the process, enabling proactive incident responses by spotting deviations in time series data. OpenObserve is an open-source platform designed for real-time monitoring and analysis of logs, metrics, and traces, integrating seamlessly with existing toolchains and offering cost-effective solutions for large-scale data analysis. RCF, an unsupervised anomaly detection algorithm, excels in detecting anomalies in high-dimensional or complex time series data without needing labeled data. It is particularly useful for real-time log anomaly detection, financial data analysis, and monitoring network performance. Implementing a real-time anomaly detection system involves setting up OpenObserve, configuring the environment, training the model, and deploying it to monitor streaming data, with visualization and alerts for anomalies. Regular retraining and threshold adjustments are necessary to address challenges like false positives and concept drift, ensuring a scalable and intelligent monitoring system.