November 2022 Summaries

AWS EventBridge can help organizations build loosely coupled, event-driven architectures and deploy new features faster, while also leveraging it for observability purposes to collect and send data across applications and services. By using EventBridge as an event bus, organizations can centralize monitoring and reduce costs by complementing it with other tools that are stronger in certain areas. Data retention is crucial for root cause analysis, and tools like ChaosSearch offer unlimited data retention via cloud object storage, minimizing costs and eliminating data movement. The combination of EventBridge, ChaosSearch, and Amazon S3 can provide a streamlined centralized logging solution, allowing organizations to scale data analytics while optimizing costs.

The 7 Costly and Complex Challenges of Big Data Analytics` highlights seven major challenges that enterprise DevOps teams, SREs, and data engineers face when navigating the growing costs and complexity of big data analytics. These challenges include `Data Pipelines`, where modern tools often run into scalability issues; `Data Preparation`, which can consume up to 60-75% of resources in an organization's data analytics program; `Data Destination` and `Governance`, where data governance strategy and controls must span the entire data pipeline, but localized around the final destination element. The challenges also include `Data Platforms`, where most platforms provide a trade-off between generic power and flexibility; `Data Analytics`, where operational and business analytics are often run separately by two different departments with big data silos; and `Data Lifecycle`, where long-term data retention is a complex challenge to manage at scale. ChaosSearch addresses these challenges by leveraging the power of cloud object storage, delivering an idealized data destination and optimized approach to governance, and transforming your cloud storage into a hot analytical data lake.

In this episode of "Data Legends: Stories from the IT Trenches", David Noblet, Co-Founder + Chief Architect at ChaosSearch, shares his insights on how to build a culture of passion in a high-pressure startup environment, expand at a sustainable rate, and prioritize curiosity in the face of rapid technological change. David emphasizes the importance of hiring team members who share the company's passion for technology, measuring progress by the sum of all vectors, and finding common ground between innovation and consistency. He also stresses the value of surrounding oneself with curious people to inform new product ideas and improve existing ones, and advocates for an incremental approach to growth and development.

Cribl Stream is a vendor-agnostic observability pipeline tool that enables enterprises to consolidate logs, metrics, and traces from any source, clean, transform, normalize, and enrich the data, and route it to multiple destinations. Cribl's unique approach allows customers to collect, transform/filter, and route observability data to where it's needed without imposing data retention limits that restrict log analytics use cases. By using Cribl Stream with ChaosSearch, organizations can create a consolidated view of observability data sources, replay logs with instant visibility, and keep the same data analytics tools while reducing costs. This combination provides a cost-effective solution for enterprises to maintain security and oversight of complex IT systems without imposing data retention limits that restrict log analytics use cases.

The Right Time to Right-Size Your Observability Process Every client we meet has been using multiple tools to satisfy their observability needs. We rarely find a greenfield opportunity, as they have pointed out when the time is right to add ChaosSearch into the fold. This is usually due to a combination of high log data volume, unpredictable costs and ineffective results. By the time we talk to clients in this state, the pain and frustration are incredibly high. However, after adding ChaosSearch, life gets much better, with clients able to keep their existing dashboards, avoid rehydration woes, reduce costs and gain better insights through investigations across all log data. This leads to optimal economics, as well as room to add ChaosSearch into the mix, resulting in cost savings of 20-50% in overall TCO.

JSON FLEX is a solution that enables organizations to automatically index and process JSON data for hassle-free business insights. With its ChaosSearch technology, JSON FLEX leverages flexible indexing choices and schema-on-read transformation features to efficiently store and query JSON structures. This allows analysts and end-users to specify, quickly search, and visualize their valuable business data without worrying about the JSON permutation explosion. By selectively controlling the indexing of JSON source and using rules to exclude or include only target data content, organizations can reduce storage footprint, maximize scan performance, and filter out unneeded data. Additionally, views with JSON transformations and schema-on-read features in the Refinery enable materialization of columns from JSON strings, making them searchable with Elastic nested query paths, and transforming arrays for analysis benefits.