April 2019 Summaries

The author has gained access to a large dataset of GitHub users and projects through the GHTorrent project, which is an effort to build an offline version of all data available in the GitHub APIs. The data was made available on Google Big Query for free, and after uploading it to Amazon S3, the CHAOSSEARCH platform quickly indexed the data, allowing the author to analyze and visualize various trends and patterns. The author analyzed the growth of user creation across the entire dataset, which showed a fairly even growth with some large spikes in users getting created, especially towards the end of the year. They also found that the majority of users didn't fill out their location details, but those who did were mostly from the US, India, and China. The author graphed the growth of users over time for these countries and found that user creation started growing around 2011-2012. In terms of state popularity in the US, Massachusetts ranked low due to its high number of large enterprises with limited software development activity, while cities like San Francisco and Austin were more popular among GitHub users. The author was able to use the CHAOSSEARCH platform to quickly search for their own user accounts using prefix and postfix wildcard queries, which allowed them to find both of their user accounts within seconds. In their next post, they will continue analyzing the dataset to learn more about the projects that these users are creating in GitHub.

Amazon S3 has become the bedrock of storage for the cloud as we know it, used to store a wide range of data types including static web content, application log and event data, AWS logs, backups, images, movies, metadata, satellite data, IoT and mobile data. The marriage of CHAOSSEARCH and S3 is perfect, allowing users to "Store everything. Ask anything." With Logstash, it's easy to get data into S3 from various sources such as the AWS CLI, Fluentd, or Filebeat. Once data is in S3, CHAOSSEARCH can extract value quickly and easily.

The origin of CHAOSSEARCH and its Data Edge technology is rooted in the desire to make information smaller, reducing data storage, processing, and complexity. The founder's "ah-ha moment" led to the development of a new database index called Data Edge, which aims to disrupt the analytics space by providing a cost-effective solution for big data management and analytics. The technology is based on first principles, including minimums, distribution, and analysis thereof, which guide architectural decisions and lead to the creation of a unified data platform that unifies storage and analytics into a single solution at a disruptive price. This mission is driven by the vision that anyone can store everything and ask anything of their own data, revolutionizing business opportunities.

Announcing GA – Search and Analytics for the Cloud` CHAOSSEARCH is a search and analytics platform that converts Amazon S3 into a searchable database cluster at high-performance and low cost. The platform, now in general availability (GA), delivers an industry first by providing a fully managed service that allows customers to store all their data and quickly search and analyze it without moving it from their own S3 infrastructure. CHAOSSEARCH is designed for businesses struggling with managing their own Elasticsearch database or ELK Stack for logging, offering a disruptive pricing model starting at 50% the cost of traditional solutions. The platform provides a range of features, including built-in data cataloging and organizing services, one-click schema detection, normalization, and indexing, as well as a data refinery console for advanced data transformations. With its multi-tenancy and account management capabilities, CHAOSSEARCH enables organizations to capture and index 100% of their machine data at a lower cost than traditional solutions.

AWS provides best practices documentation that outlines numerous services like Lambda, Glue, Quicksight, and Athena, but users must tie these services together to gain insights into their data, which can be challenging due to the lack of ability to search across data. To overcome this challenge, customers are moving their logs out of Amazon S3 using tools like Elasticsearch and the ELK Stack, but they soon realize that managing large volumes of data becomes expensive in terms of both capital and human resources. A new solution called CHAOSSEARCH offers a fully managed service that integrates with S3 buckets, indexes data back into the account, and provides highly compressed storage, allowing users to derive value from their data without having to move it out of S3. This approach enables users to continue using existing tools like Kibana, as CHAOSSEARCH extends the Elasticsearch API on top of the user's data on Amazon S3.

The volume of data is projected to grow exponentially, from 33 zettabytes in 2018 to 175 zettabytes by 2025, driven by the internet, cloud, and connected devices. This growth has transformed raw data into a critical component of business operations, but transforming it into valuable information is a more complex and expensive process. The refining of data involves much more compute resources than generating and storing it, making information creation a costly endeavor. Traditional database solutions have been effective in the past, but they are now facing challenges due to the growth in the 3Vs (velocity, variety, and veracity) of data, which has led to the development of new styles of databases that increase costs. However, an alternative approach called CHAOSSEARCH aims to make information inexpensive by employing a patent-pending technology and architecture that simplifies access to information without breaking the bank.

Elasticsearch is an open-source, RESTful, distributed search and analytics engine widely used by developers, providing powerful query capabilities on large datasets at scale. Its management can be complex, requiring sizing, node allocation, index building, and data placement, which may lead to high costs and operational headaches. While Elasticsearch offers flexibility, some companies opt for managed solutions to avoid these complexities. CHAOSSEARCH provides a simpler alternative by indexing data in S3, allowing customers to store and analyze their logs without the need for manual management or additional infrastructure. By separating cluster management from data storage, CHAOSSEARCH enables users to focus on insights rather than operational overhead.