March 2019 Summaries

Amazon continues to focus on empowering its customers by providing infrastructure services that allow them to concentrate on their core business logic. The company has made significant investments in the AI/ML space, including the AWS DeepRacer project, which enables engineers to develop machine learning models using reinforcement learning. However, many customers are still struggling to find a solution for search and analytics on their AWS S3 data, highlighting the need for tools like CHAOSSEARCH that can help reduce Elasticsearch complexity and costs. Amazon's recent announcement of Amazon Glacier Deep Archive has also provided a new option for storing massive data volumes at a lower cost. Overall, the event showcased Amazon's commitment to innovation and its efforts to support customers in achieving their business goals.

The time a threat actor is present undetected in a network is known as dwell time, with most corporate environments experiencing an average dwell time of around 99 days. Many companies keep only 7-14 days worth of data due to high costs associated with running Elasticsearch clusters or the ELK stack. While Amazon Athena allows querying security and compliance events, managing query costs remains a challenge, often requiring additional tools for visualization. The complexity of integrating multiple services like S3, Glue, EMR, and Athena can be overwhelming. Some companies have turned to the open-source ELK stack, but this comes with its own set of challenges, such as choosing between retention and AWS bills. CHAOSSEARCH offers a solution by indexing all fields in Amazon S3 documents, allowing for seamless querying without the need for database servers or additional tools like Kibana or Tableau.

This Wednesday, March 27th, CHAOSSEARCH will be sponsoring their first 2019 AWS Summit in Santa Clara, CA, a FREE event at the Santa Clara Convention Center, as part of eight AWS Summits being held in the Americas this year. Dr. Werner Vogels will kick off the summit, providing context for attendees before they dive into technical sessions covering topics such as Serverless, ML/AI, and Hybrid Cloud. The CHAOSSEARCH Team will have a booth at Startup Central, where they'll connect with AWS users and partners to discuss how their platform can help manage log data and provide access to months of historical data using Amazon S3.

We are hearing from customers that companies want to store more data due to rapid growth in mobile computing, IoT, edge applications, and cloud computing, aiming to differentiate themselves through innovation and value creation. Storing more data should not be cost-prohibitive or consume a disproportionate percentage of IT spend. Customers see Amazon S3 as the perfect place for storing data at scale, but often struggle to leverage its value due to lack of direct access to their data. Managing Elasticsearch is also complex and costly, with limitations such as scalability complexity and inability to perform relational queries. Customers desire easy text search on log data, programmatic APIs for integration, security control, and flexibility in choosing vendors. Ultimately, customers want a platform that can unlock the power of data stored in S3 without being locked into one vendor or platform.

The Blue Bikes dataset was used to analyze rider behavior, with two main types of riders: Customer and Subscriber. The average trip duration for Subscribers is around 12 minutes, while Customers take about an hour. An outlier in the data showed a nearly year-long ride, which was likely due to a bike not being properly locked back into its docking station. Further analysis revealed that Bike 1053, with this unusual ride, had multiple disappearances from and re-appearances in the network, suggesting it may have been taken for an extended period without being properly secured. The bike eventually returned to normal circulation after another year of use, coming to rest at the Museum of Science.

CHAOSSEARCH, a data search and analytics platform, allows users to query and analyze datasets without needing to parse or configure the schema on their side. The author recently used this platform to dive into Boston's Blue Bikes bike share program dataset, which is publicly available on Amazon S3. By indexing and querying the data, the author was able to extract insights such as the rate of rides over time, user demographics, and popular stations for users. The analysis revealed interesting patterns, including a seasonal decrease in non-subscriber usage during winter months, consistent 20% growth among subscribers, and an imbalance in gender representation with men outnumbering women by about 3:1. The author also identified the most popular start and end stations for users, as well as a least used station that appears to be used primarily for round trips between a small group of destinations.