August 2016 Summaries

In an intriguing case study, the author explores the challenges of troubleshooting a persistent 502 error encountered in a multi-tier AWS environment using various monitoring tools such as AppDynamics, DataDog, and Sysdig. Despite having these commercial tools, the root cause of the issue remained elusive until the open-source tool Sysdig was employed. Sysdig's capture feature, which records every system call on a host, was pivotal in identifying the problem: an inverted HTTP header in the Java application's response that was being rejected by HAproxy. This case highlights the limitations of traditional monitoring tools in providing deep insights into specific issues and underscores the value of incorporating open-source tools like Sysdig into a developer's troubleshooting toolkit for more comprehensive diagnostics.

The blog post discusses the complexities and challenges developers face with software largely built upon legacy systems, using a specific troubleshooting incident involving Sysdig and Linux as a case study. The author, Luca Marturana, recounts an issue where a customer's statsd metrics weren't being captured due to php-fpm not attaching appropriate container metadata when using Alpine Linux, which relies on the fork system call rather than the more modern clone. The root cause was identified as a parsing oversight in the Sysdig tool, which only accounted for clone events, leading to a solution that involved modifying the code to parse fork and vfork events as well. This anecdote highlights the broader theme that operating systems, while reliable, carry a form of technical debt that requires engineers to navigate historical quirks and maintain compatibility with older software behaviors, underscoring Linus Torvald's principle of "never break userspace."

Sysdig Cloud offers a comprehensive solution for monitoring container-based applications and Kubernetes environments, integrating seamlessly with platforms like Google GKE and Deis Workflow. It provides real-time dashboards, historical data replay, dynamic topology mapping, and alerting features, which are crucial for gaining insights into app performance and resource usage. The blog guides users through setting up Sysdig Cloud to monitor a Kubernetes cluster, installing Deis Workflow, and deploying a demo guestbook app. It emphasizes the ability to create customized dashboards for specific namespaces and illustrates how to conduct stress tests using ApacheBench to evaluate app performance under load. By leveraging Sysdig Cloud's capabilities, users can effectively track system properties and resource consumption, making it easier to identify and address performance issues.

The blog post discusses the vulnerabilities associated with SQL injection attacks, which occur when unsanitized user inputs are exploited to execute malicious SQL code, potentially giving attackers access to sensitive data and system control. It highlights the importance of input sanitization to prevent such attacks and suggests additional security measures such as limiting database privileges and removing unused functionalities. The post introduces Sysdig Falco, an open-source behavioral activity monitor that detects anomalous activities post-exploitation by examining system calls and identifying suspect behaviors. Falco operates in various environments, including containerized, virtualized, and bare-metal Linux deployments, and uses a ruleset to monitor activities such as reading sensitive files, replacing system files, spawning unauthorized shells, and opening network connections. By providing real-time alerts for these activities, Falco adds an extra layer of defense against attacks, encouraging users to contribute to its ruleset to enhance community protection.

Falco 0.3.0 introduces significant enhancements, primarily aimed at reducing CPU usage through various optimizations such as reordering rule condition operators, implementing set membership tests, avoiding unnecessary string manipulations, utilizing efficient string comparison operators, and grouping rules by event type. These changes lead to improved performance when compared to the previous version, as demonstrated in tests like Phoronix's pts/apache and pts/dbench, Sysdig Cloud Kubernetes Demo, and Juttle-engine examples. Additionally, the release introduces a new output type, "program," which allows formatted events to trigger program executions, such as sending emails for each notification, and includes the capability to handle all events, even those flagged with EF_DROP_FALCO, to enhance overall functionality. The update is accessible through standard distribution channels, and users are encouraged to review the changelog for a comprehensive list of modifications.

Loris Degioanni's blog post explores an efficient and less cumbersome approach to system profiling using sysdig tracers, particularly beneficial for developers who prefer minimal setup. The post discusses the use of sysdig, an open-source tool, to profile MongoDB queries by comparing the latency and network bandwidth usage of the aggregate functionality introduced in MongoDB 2.2 against the traditional map-reduce method. Degioanni demonstrates how sysdig tracers can be integrated into scripts to mark the beginning and end of query executions, allowing for the collection of performance data with reduced noise. The analysis reveals that the aggregate method is significantly faster and more bandwidth-efficient than map-reduce. The post concludes by emphasizing the advantages of using aggregate queries in MongoDB and encourages developers to adopt sysdig tracers for easier and effective profiling in containerized environments.