August 2022 Summaries

A full observability stack aims to provide centralized visibility into application performance and behavior, with key components including metrics, logs, and traces. Open APIs and CNCF projects such as OpenTelemetry, fluentd, Prometheus, Jaeger, and ChaosSearch can be used to build a cost-effective observability solution. Metrics monitoring helps identify potential problems, while log investigation uses tools like OpenSearch and Kibana to analyze log data. Traces are used for distributed context propagation, transaction monitoring, and root cause analysis with Jaeger as the CNCF-sponsored open source system. A combination of best-of-breed components and open APIs can provide a flexible and future-proof observability solution.

ChaosSearch has helped Equifax simplify and power its operational analytics by addressing five key challenges faced by the company as it scaled its operations in the cloud. By adopting ChaosSearch, Equifax was able to democratize access to log data, manage logging costs with high data ingest, accelerate time to insights, simplify data analytics workflows, and minimize the management burden of analytics. With a centralized cloud data repository in Amazon S3, Equifax eliminated siloed deployments and lost insights, reduced storage costs, and accelerated its time to insights. ChaosSearch's schema-on-read approach eliminated complex data pipelines and made log data available for analysis much sooner after it was generated. The platform also simplified data analytics workflows and reduced the management burden of log analytics, allowing Equifax to deliver new opportunities to the business and discover usage patterns in its systems.

At the Black Hat USA 2022 security conference, Amazon AWS and Splunk partnered with other vendors to create the Open Cybersecurity Schema Framework (OCSF), a collaborative effort aiming to simplify data ingestion from various security products for enterprises to better protect against threats. `This initiative acknowledges the challenge of dealing with diverse security product data, providing a vendor-agnostic taxonomy to empower security teams to respond quickly and effectively. `The OCSF schema uses JSON and encourages open collaboration to make systems and data more secure. `ChaosSearch is excited about this project's evolution and its potential to improve operational and security data lakes for monitoring and alerting on security, application, Kubernetes, infrastructure logs, CloudWatch, CloudTrail, VPC FLow Logs, Splunk, Cloudflare, Fastly, Signal Sciences, Okta, Auth0, etc. `The OCSF project can be found at https://github.com/ocsf/.

Wes Gelpi, Director of Research & Development at SAS, joins a popular podcast to share his insights on leading a team and leveraging Kubernetes to drive innovation in cloud application management. He emphasizes the importance of creating a safe space for creatives to think effectively and work, allowing them to capitalize on their initiative and drive return on investment. Gelpi believes that Kubernetes is an untapped potential for companies to improve resiliency within platforms and applications, and its adoption is currently low due to a lack of recognition of its benefits. He also discusses the importance of data management through lakes, warehouses, and virtualization, highlighting the need for organizations to tailor their approach depending on the situation or project. By pairing these systems together, companies can create a customizable data view for various users and purposes, simplifying data storage and meeting each user's needs without significant technical involvement.

Datadog is a popular cloud-native application performance monitoring tool that measures system health and status based on telemetry data, including machine-generated logs. However, as organizations scale, they face challenges such as log retention schedules, data transformation and rehydration processes, and cost. These challenges can hinder the team's ability to generate insights, limit their ability to achieve regulatory compliance and legal requirements, and require workarounds that are not ideal. To overcome these challenges, Datadog users can consider using alternative solutions like ChaosSearch, which offers unlimited data retention at a starting price of $0.30/GB, significant discounts at scale, and faster time to insights by analyzing data in place without processes like data pipelines and rehydration. By combining Datadog with ChaosSearch, organizations can achieve best-in-class monitoring and a deeper understanding of their systems to drive business growth.