November 2018 Summaries

Kubernetes 1.13, released on December 3, 2018, introduces several new features and updates targeted at improving configuration management, core components, hardware support, storage, and cloud integration. Notable enhancements include the dynamic audit configuration, which allows audit events to be forwarded to remote APIs, and the support for webhook conversion for custom resources, enabling different schemas for different CRD versions. The API-server dry-run feature moves to beta, allowing users to simulate API requests without modifying the state, and the kubectl diff command provides a preview of changes before they are applied. Kubernetes cluster creation is simplified through improvements in kubeadm, while taint-based evictions and scheduler performance enhancements further optimize cluster operations. Additionally, Kubernetes 1.13 enhances hardware support with monitoring plugins for third-party devices and expands storage capabilities by making raw block devices available via persistent volumes and stabilizing the Container Storage Interface for third-party volume plugins. Cloud integration is strengthened with features like the AWS ALB ingress controller and the debut of the AWS cloud-controller-manager, along with upgrades to Azure availability zones and cross-resource group node support. This version also marks the deprecation of support for etcd2, reflecting ongoing efforts to streamline and modernize the platform.

Before the release of Java 9 and 10, deploying and monitoring Java in Docker containers presented several challenges, including the spawning of a complete Java Virtual Machine (JVM) for each container, issues with container isolation awareness, and inconvenient monitoring processes. Java 9 and 10 have addressed these issues by improving integration with Docker, recognizing container CPU and memory limits, and enhancing monitoring capabilities. Java 10, in particular, allows for better resource management by accurately detecting CPU and memory allocations, making it more efficient to run multiple JVM instances in containers. Additionally, monitoring has been simplified with tools like Sysdig, which eliminates the need for complex network configurations and supports native JRE and custom JMX metrics. These advancements facilitate easier deployment and monitoring of Java applications in dynamic, auto-scaling environments, offering a more seamless experience for developers and organizations transitioning to containerized solutions.

Harry Perks discusses the use of Falco, an open-source behavioral monitoring tool that leverages system calls, to detect unexpected network activity by writing custom rules. Falco's flexibility allows users to create rules for various activities, including network connections, by utilizing macros to monitor outbound connections and exclude local ones. Challenges arise with dynamic DNS, which can cause hardcoded IP addresses to become obsolete, leading Sysdig's engineering team to introduce new filter checks in version 0.24.0 that allow domain names to be specified instead of IP addresses, ensuring up-to-date resolution and better management of network policies. These enhancements enable users to define trusted domain names, triggering alerts when connections occur outside the specified domains, and integrate with Sysdig Secure to apply rules across specific infrastructure scopes, such as Kubernetes deployments, for tailored network security. The article emphasizes the power of system calls for intrusion detection and encourages users to share their experiences with Falco.

Falco 0.13.0 introduces support for Kubernetes Audit Events, marking a significant enhancement since its initial release, by enabling the detection and notification of suspicious activities within Kubernetes clusters. This update provides a second stream of events, alongside system call events, and allows the integration of additional event sources in the future. The release includes new Kubernetes audit rules that identify notable activities, such as unauthorized user actions, creation of privileged pods, and management of sensitive resources like configmaps with private credentials. Falco’s architecture remains consistent, with events being processed against rule sets, but now includes a civetweb-based webserver to accept Kubernetes audit events through POST requests. The rules are classified into those detecting suspicious activities, tracking resource changes, and displaying audit events, offering enhanced runtime observability. Additional features include better kernel module management, configuration reloads without restarting Falco, and the integration of netcat for easier event management. The release is accessible via various platforms, including RPM/Debian packages, Docker images, and GitHub, with further support available through the Sysdig open-source Slack team.

The blog post discusses how to instrument Go code with custom metrics using the expvar library, a standard interface in Golang for exposing application metrics via HTTP. It explains how expvar can be used to track application performance and health by exporting custom variables and built-in metrics like memory allocation and garbage collection stats. The post details the process of setting up expvar, including registering an endpoint for JSON-formatted metrics and creating custom metrics using various expvar types such as String, Integer, Float64, Map, and Structs. Additionally, it provides a complete example of a Go application utilizing expvar to export custom metrics and discusses challenges and solutions for collecting these metrics in a containerized environment using Sysdig Monitor. Sysdig Monitor is highlighted for its ability to simplify expvar metric collection and enhance monitoring by automatically discovering application components and offering real-time insights without requiring significant code changes.

Cota Healthcare, a company focused on improving cancer patient outcomes through technology and precision medicine, has transitioned to using Kubernetes on Google Cloud, enhancing its infrastructure with Sysdig for effective monitoring and container security. This move supports Cota's mission to handle large data sets securely while maintaining high developer productivity and reducing costs. Sysdig offers Cota the ability to gain comprehensive insights into their cloud infrastructure, which facilitates rapid troubleshooting, capacity planning, and the secure management of sensitive healthcare data in compliance with HIPAA standards. By leveraging custom metrics from Prometheus and StatsD, Cota can efficiently monitor application performance and ensure their infrastructure is right-sized, which has led to significant cost savings. The integration of Sysdig has also improved the speed of issue resolution, enhanced developer velocity, and provided the necessary visibility into system behavior to maintain a strong security posture.

Mateo Burillo's blog post focuses on troubleshooting three common Java production issues: network bottlenecks, memory leaks, and thread locking, utilizing open-source tools like Sysdig for diagnosis. For network bottlenecks, the post explains how high request latency can occur due to factors like client request surges or software bugs, and it suggests using monitoring systems to detect anomalies in request handling. Memory leaks are addressed by demonstrating how to identify high memory usage and Java class loading through Sysdig, with a focus on detecting undetected leaks that can lead to crashes. The post also covers thread locking issues, such as deadlocks and race conditions, by using Sysdig to track process behavior and identify synchronization barriers causing CPU usage drops. While the examples provided are simplified, the post emphasizes the importance of adapting Java troubleshooting to modern container platforms and microservices, offering Sysdig Monitor as a tool to enhance system inspection and monitoring.