April 2016 Summaries

In the blog post, Gianluca Borello shares an intriguing experience of uncovering a bug in Amazon's Elastic Load Balancer (ELB) while managing the deployment of Sysdig Monitor, a container monitoring tool. The issue arose during a rolling deployment process, where ELB's behavior of abruptly dropping connections upon deregistering an instance caused spikes in system load. Despite initially suspecting the application code, a thorough investigation using Sysdig and Wireshark revealed that the ELB was closing connections prematurely due to a glitch associated with the removal of an internal ELB node when a backend instance in a single availability zone was deregistered. AWS confirmed the bug and suggested maintaining at least two instances per zone as a workaround. This case highlights the importance of comprehensive monitoring and troubleshooting to understand and resolve complex infrastructure issues.

Sysdig, a company focused on container monitoring solutions, recently raised $15 million in a Series B funding round led by Accel and Bain Capital Ventures, who also invested in previous rounds. This funding underscores the growing potential of the container market and the success of Sysdig's open-source and enterprise products, with the company having already gained over 100 enterprise customers. The investment will be used to enhance Sysdig's engineering capabilities, expand its go-to-market strategies, and continue supporting the open-source community. Sysdig plans to develop new features for its Sysdig Cloud platform, focusing on passive instrumentation, ecosystem integration, and the shift towards microservices, while also working on democratizing access to complex container information through a community-based approach. The company expresses gratitude for the support it has received and is actively hiring to further its mission in the container and open-source arenas.

Gianluca Borello, an engineer at Sysdig, describes a performance issue encountered with Cassandra, a database known for its scalability and flexibility, when used for storing and processing streams of binary blobs. The problem arose when querying large streams, leading to degraded response times due to Cassandra processing all columns in a row, even when only specific columns were queried. Through tests and system tracing with tools like sysdig, Borello identified that Cassandra was reading entire data files instead of the specific data requested, due to the way it handles CQL row semantics. To resolve this, the team refactored their schema to distribute blobs across multiple rows, reducing the size of each row and significantly improving query performance. This experience highlights the importance of monitoring and troubleshooting at the system level, demonstrating how system call analysis can efficiently identify and solve database performance issues without delving into the application's internal code.

DC/OS, or Data Center Operating System, created by Mesosphere, integrates open-source projects like Mesos and Marathon to pool distributed resources into a unified system for running complex applications, with or without containers. The shift from virtual machines to containers is significant, and DC/OS simplifies the management and deployment of containerized environments, essential for leveraging container technology effectively. Sysdig has partnered with Mesosphere to provide monitoring solutions that integrate seamlessly with DC/OS, enabling real-time topology mapping and resource visualization. This integration helps users understand the allocation and performance of applications within DC/OS, illustrating both physical and logical topologies to aid in troubleshooting and comprehension of the system's operations.

The Sysdig cheatsheet provides a comprehensive guide for Linux administrators to troubleshoot server issues using command-line tools such as strace, htop, lsof, tcpdump, and iftop, with a particular focus on using Sysdig and csysdig as alternatives or complements. It explains how these tools function, including their capabilities to trace command executions, monitor system calls, list open files, and capture network traffic, while highlighting the distinctive real-time filtering and analysis features of Sysdig. The cheatsheet also offers practical examples comparing traditional tools with Sysdig's syntax, emphasizing Sysdig's flexibility in analyzing live or captured data with advanced filtering options, and demonstrates the interactive nature of csysdig, which enhances user experience by providing live system insights.

Sysdig 0.9.0 introduces significant updates, including native support for monitoring Mesos, Marathon, and DCOS, allowing users to specify URLs for Mesos Master and Marathon API to create specific views and event outputs. It offers enhanced network monitoring capabilities by enabling subnet-based filters using CIDR notation, providing a more intuitive data filtering method. The release also improves Kubernetes authentication by adding support for SSL-based and bearer token authentication against the Kubernetes API server. Additional updates include new actions in csysdig views such as 'lsof' and 'renice', and a new case-insensitive filter comparison operator 'icontains'. Users can download the latest version from Sysdig's website or access the source code on GitHub.

Sysdig offers comprehensive monitoring and troubleshooting support for Apache Mesos, Mesosphere Marathon, and DC/OS environments through its open-source and commercial products. The integration enables users to shift focus from monitoring physical resources to monitoring logical resources such as applications and services, providing deep insights into the cluster's layout, configuration, and performance metrics. This capability is demonstrated through a scenario involving a conflict between Redis servers running in Mesos and Docker containers, highlighting Sysdig's ability to diagnose and resolve issues through detailed container-level analysis. Sysdig's tools, including sysdig, csysdig, and Sysdig Monitor, facilitate real-time cluster status monitoring, historical data replay, and automatic detection of Mesos frameworks, with current metadata support specifically for the Marathon framework. The platform aims to expand its framework-specific support based on customer needs, enhancing its utility for users managing distributed systems.

The blog post discusses the importance and utility of metadata in Kubernetes environments, particularly in facilitating the organization and understanding of container orchestration across various services and infrastructure. It highlights that Kubernetes metadata, primarily in the form of labels and annotations, provides a logical framework for monitoring and troubleshooting applications by transforming a physical view of hosts and containers into a logical view of services and microservices. The article uses examples such as a 3-node deployment on Google Kubernetes Engine (GKE) to illustrate how metadata can aggregate metrics and create a hierarchical view of components like namespaces, replication controllers, pods, and containers, thus allowing for efficient performance monitoring and resource management. The post concludes by encouraging readers to explore the potential of metadata for enhancing their system's efficiency and hints at a future discussion on adaptive alerting using metadata.

Sysdig's Spring 2016 release introduces several enhancements and new features to its Monitor tool, significantly improving performance, scalability, and integration capabilities. The update supports tens of thousands of containers per customer with faster dashboard load times and integrates seamlessly with Docker Data Center and Kubernetes 1.2, offering new views for AWS ECS, Docker, and Kubernetes environments. Troubleshooting capabilities are enhanced by allowing system captures triggered by alerts, providing deeper insights beyond standard dashboard alerts. Sysdig Monitor also introduces a number panel feature for real-time metric monitoring and over 20 new application views for instant data access, complementing its auto-app discovery feature. Users can now customize columns in the Explore view, akin to a pivot table for infrastructure, and a new Dashboard Wizard has led to a tenfold increase in dashboard creation.