December 2023 Summaries

OpenObserve's transition from Plotly to Apache ECharts in data visualization was driven by the need to address scaling challenges and enhance user experience, marking a strategic milestone in their operations. While Plotly had been a reliable tool, it started showing limitations such as issues with displaying vertical scrollbars and hidden X-axis labels as project complexity increased. Apache ECharts offered a solution with its robust performance, efficient handling of large datasets, extensive customization options, and a supportive community, which collectively contributed to improved load times and user interactions. The transition, although challenging, was facilitated by comprehensive documentation and community support, leading to a significant transformation in user experience by offering more intuitive and responsive visualizations. This strategic choice reflects OpenObserve’s commitment to leveraging advanced technology for clearer insights and better decision-making.

OpenObserve is highlighted as an innovative and adaptable monitoring solution that served as a robust alternative to DataDog for a company's Azure Kubernetes Service (AKS) cluster. The choice to deploy OpenObserve was driven by its high compression ratio, low memory footprint, and cost-effectiveness in storage, making it an ideal fit for the Azure environment without performance degradation. The deployment process involved the use of a customizable version of OpenTelemetry collector to gather logs, metrics, and traces from applications, and the article provides a detailed guide on deploying OpenObserve via helm, leveraging Azure storage, and configuring PostgreSQL for metadata storage. The monitoring transformation included setting up an Nginx ingress for external access, and the use of OpenObserve's alerting feature to create and manage alerts through customizable templates, such as Slack notifications. This deployment not only enhanced the organization's monitoring capabilities but also demonstrated OpenObserve's versatility across different cloud environments and its ability to support various applications regardless of language or framework.

Jiyue, a high-intelligent car robot brand, significantly enhanced its application monitoring and troubleshooting capabilities by migrating from Elasticsearch to OpenObserve, an open-source observability platform. This transition addressed several challenges, including limited trace sampling and high operational costs associated with Elasticsearch's scalability issues and resource demands. By deploying OpenObserve, Jidu achieved 100% trace data sampling, leading to a comprehensive understanding of application behavior and the ability to swiftly identify and resolve performance bottlenecks. The efficient architecture of OpenObserve reduced infrastructure burdens, with notable decreases in CPU and memory usage and daily storage requirements, ultimately resulting in significant cost savings. This transformation not only optimized resource utilization but also improved application stability and operational efficiency, culminating in an enhanced customer experience and serving as a model for organizations aiming to elevate their system monitoring and resource management strategies.

The blog post explores how to effectively filter logs at their source using the otel-collector, an open-source observability data collector that is part of the OpenTelemetry project, to manage log volume sent to centralized logging backends. It emphasizes the use of two specific processors, the filter processor and the attributes processor, to selectively filter entire log records or specific attributes, which can help reduce costs, noise, and network load. The post provides practical examples of filtering logs from a specific namespace and removing unnecessary log attributes, discusses the challenges of learning and implementing OpenTelemetry Transformation Language (OTTL), and introduces the visual pipelines available in OpenObserve for easier log, metric, and trace management. By applying these techniques, users can optimize their observability data, enhancing efficiency and reducing unnecessary data transmission.