May 2019 Summaries

The Boston DevOps meetup, hosted by CHAOSSEARCH, featured an open space style discussion format where attendees wrote down topics they wanted to discuss and voted on which ones to focus on. The event saw discussions on various topics including opensource observability tooling, over-engineering of platforms, and the challenges faced by different groups such as startups and hospitals. The event was a great success, with many attendees appreciating the opportunity to learn from others and share their own experiences in solving common DevOps challenges. The organizers and attendees alike expressed gratitude for the event and looked forward to the next meetup.

CloudTrail is a service provided by Amazon Web Services (AWS) which logs every single API call within an AWS account, providing deep visibility into activity and allowing for security and operational troubleshooting. While there are no charges to use CloudTrail, standard S3 charges apply to logged data. To analyze this data, customers must ingest it into a third-party service such as Redshift or Athena, or use ChaosSearch, which automatically identifies fields, indexes data, and provides a Kibana interface for visualization. With ChaosSearch, users can identify anomalies in their CloudTrail data, run queries on specific events, and gain insights into API usage and user activity in minutes, without needing to deploy a database or create schemas and mappings.

'' Terraform is a powerful automation tool that can be used to create and manage infrastructure in the cloud. In this example, it's used to set up CHAOSSEARCH for searching and indexing data. The process involves creating a bucket to store data, setting up encryption using KMS keys, and configuring SQS queues for notifications. Terraform is used to automate the creation of these resources, making it easier to manage infrastructure. However, errors may occur during the process due to eventual consistency issues with AWS services. Once the setup is complete, CHAOSSEARCH can be tested and used to search and index data. The infrastructure can then be cleaned up using Terraform's `destroy` command. Overall, this example demonstrates how Terraform can be used to simplify the process of setting up and managing infrastructure for CHAOSSEARCH.

AWS Elastic Load Balancer (ELB) Log Analysis on S3 is a critical aspect of maintaining high-performance load balancing and application delivery for services. Traditional approaches like provisioning an ELK Stack or leveraging AWS Athena can be costly and complex, especially as data volumes grow. CHAOSSEARCH offers a more efficient solution by indexing raw ELB log data directly in the S3 bucket, eliminating the need to move data out of the bucket or configure data visualization with Athena. This platform provides auto-regex parsing for semi-structured log datasets, automatic schema creation and mapping, and real-time data indexing capabilities. With CHAOSSEARCH, users can navigate to a fully integrated Kibana interface, visualize data, and gain deep insights into their ELB logs without the complexity and expense of an Elasticsearch cluster. The platform also offers unlimited retention of log data, enabling users to ask larger and broader questions across months and years.

Our goal at CHAOSSEARCH is to provide a simple and low-cost log management platform that reduces configuration friction, allowing users to easily integrate new services with minimal documentation review. We have developed automation templates using AWS CloudFormation to streamline the setup process for integrating AWS S3 buckets with our platform, eliminating the need for manual configuration and reducing errors. These templates are available on our documentation page and can be used as-is or customized to meet unique needs. The templates automatically build necessary resources, including IAM Roles and Policies, and enable scalable logging processes using AWS SQS and s3 notifications.

CHAOSSEARCH technology and architecture core principles aim to reduce complexity and cost associated with log management by transforming S3 storage into a search and analytic database, automating data management, and providing patent-pending innovation that offers a holistic solution for customers. By removing the need for external scaffolding required to extract, transform, and load data, CHAOSSEARCH pricing is disruptive, but the platform and philosophy are even more so, enabling businesses to view their operations and offerings completely differently when information is less expensive.

This year's DevOpsDays Austin conference was held at a unique venue, the Darrell K Royal–Texas Memorial Stadium, and featured high-quality talks from industry experts such as Damon Edwards and Wendy Nather on day one, and Patrick Dubois on day two. The event also introduced a new addition, DevOps Trivia hosted by Patrick himself, which proved to be a fun way to end the first day of festivities. The conference was well-organized and inclusive, with a strong focus on retaining hard-to-hire positions in organizations, a topic that resonated with the speaker who gave an updated version of their 2013 talk on this subject. Overall, it's clear that DevOpsDays Austin continues to be a premier event for developers and DevOps professionals.

The most popular software languages used to create projects on GitHub are JavaScript, followed by Java, with tremendous growth in new projects starting around 2012. The most popular open source licenses used for these projects are MIT and Apache 2.0, with BSD and GPL licenses following behind. A specific analysis of the Golang language shows that nearly three times as many projects use the Apache 2.0 license compared to MIT, likely due to companies wanting to enforce their trademarks. The study also reveals some interesting outliers, such as the University of Illinois - NCSA Open Source License being used by only two projects.