November 2016 Summaries

Logz.io has enhanced its alerting mechanism, a key feature integrated with Elasticsearch and Kibana, by introducing capabilities such as triggering alerts based on field aggregations and integrating with PagerDuty. Users can now set alerts using various aggregation types like max, min, avg, and sum, allowing them to monitor specific conditions within their ELK Stack environment and receive notifications through PagerDuty when those conditions are met. The process involves retrieving a PagerDuty service key, creating new endpoints, and setting up alerts with specific conditions and thresholds, which can help in real-time monitoring and troubleshooting of IT environments. Additionally, Logz.io supports grouping aggregations, enabling users to save time by configuring alerts that apply to similar conditions across multiple values of a specific field, thus streamlining the monitoring process. This evolution of Logz.io's alerting feature reflects user feedback and promises future integrations with other alerting and messaging applications.

In the context of log aggregation, Kafka and Redis are two prominent tools, each with distinct functionalities and performance characteristics. Kafka, designed as a distributed, partitioned, and replicated commit log service, excels in handling large volumes of data with its disk-based storage, offering scalability and fault tolerance through parallelism and replication. It supports log aggregation by allowing logs to be fed into various topics, which are then processed by consumers. In contrast, Redis operates as an in-memory data store, ideal for real-time data processing and scenarios requiring minimal latency but less suited for long-term data retention due to its limited memory capacity. Performance tests highlight Kafka's ability to handle high throughput, especially in asynchronous replication settings, while Redis benefits from pipelining, allowing multiple requests without waiting for replies. Both tools, when integrated with solutions like Fluentd, can enhance systems like Elasticsearch by protecting data streams and enabling robust log management. While Redis is suitable for short-lived message processing, Kafka is preferable when dealing with large data volumes that need retention.

Docker containers can be logged using various drivers, including the awslogs driver, which facilitates logging to AWS CloudWatch for centralized management. This process involves setting up AWS with the necessary permissions, configuring Docker to use AWS credentials, and using the awslogs driver to send logs to CloudWatch. Once the logs are in CloudWatch, they can be exported to an S3 bucket for further analysis with systems such as the ELK Stack offered by Logz.io. The integration with Logz.io involves setting up a pipeline from CloudWatch to the ELK Stack using S3 batch export, allowing for enhanced monitoring and analysis of Docker container logs. Additionally, methods such as Kinesis and Lambda can automate the export process, offering alternative solutions for managing Docker logs.

Logz.io has secured a $16 million Series B funding round led by OpenView, with continued support from existing investors 83North and Giza Venture Capital, to enhance its platform that transforms infrastructure and application troubleshooting into a data-driven science. By leveraging the ELK Stack, Logz.io offers an enterprise-grade, cloud-scalable solution that allows companies to efficiently aggregate, process, store, and analyze vast amounts of log data. The introduction of the Logz.io Cognitive Insights™ artificial intelligence engine further enhances this platform by autonomously identifying crucial events through machine learning, processing extensive online documentation, and building a large database of software problems and solutions. This approach combines human knowledge and AI to provide insights into machine data, making it accessible and actionable for more than 1,000 companies worldwide, including major brands like British Airways and Electronic Arts. The company's growth is credited to its dedicated team, supportive investors, and a strong customer community.

Elasticsearch mapping serves as a crucial component in defining how documents are indexed and stored, similar to a database schema, and is essential for obtaining accurate search results. The process involves determining field types, such as strings, integers, or dates, and setting custom rules for automatic updates when new fields are introduced, using either static or dynamic mapping. As Elasticsearch has evolved, mapping types have been deprecated, prompting users to adopt alternatives like indexing per document type or utilizing a custom type field. The shift from the string data type to text and keyword types in Elasticsearch 5.0 has optimized full-text searches and exact-value searches, respectively, while preventing potential mapping explosions is critical to maintaining system performance. Understanding these mapping intricacies is vital for effectively managing Elasticsearch deployments, whether through third-party services like Logz.io or self-managed systems.

AWS re:Invent is a major global event focused on cloud computing, attracting approximately 20,000 attendees with diverse sessions, keynotes, and workshops. The event features a vast array of content, including around 192 breakout sessions across eight tracks, making it challenging for participants to navigate. Highlights include sessions on Netflix's global architecture, the integration of AWS Lambda and CloudWatch, container management solutions, and the security of container-based applications. Other notable topics cover SaaS optimization, serverless architectural patterns, and cloud monitoring. Logz.io, a big data and log analysis company, is a Silver sponsor of the event and recommends sessions related to their expertise, such as big data and architectural sessions. Attendees are encouraged to reserve seats for popular sessions in advance and are advised to stay hydrated and rested throughout the event.

The first-ever annual Logz.io DevOps Pulse survey reveals several key insights into the current state of DevOps practices, highlighting that open source technology has gained mainstream acceptance with 57% of respondents heavily relying on it, while the ELK Stack emerges as the most popular monitoring and logging tool. The survey, conducted among hundreds of DevOps professionals, shows that despite the hype around cloud-based infrastructure, many companies remain cautious, with over 40% having no more than a quarter of their infrastructure in the cloud. AWS leads in cloud service usage among respondents, but the transition to microservices architecture is gaining momentum, with 78% of organizations adopting it to some degree. Interestingly, only a small portion of IT professionals currently use container orchestration services, and email remains the preferred method for receiving alerts over newer platforms like Slack. The survey also provides insights into DevOps salary ranges across different regions, suggesting that while the field continues to evolve, traditional practices and cautious adoption rates still play a significant role in the industry.