January 2018 Summaries

Red Hat's acquisition of CoreOS is seen as a strategic move that benefits the Cloud Native community by enhancing Red Hat's capabilities in container technology. CoreOS, known for its contributions to open-source projects and its innovative approach to container host operating systems with Container Linux, complements Red Hat's enterprise offerings. This acquisition allows Red Hat to strengthen its position in the container host OS market and integrate CoreOS' enterprise Kubernetes distribution, Tectonic, with its own OpenShift platform. Additionally, Red Hat's involvement with the Open Container Initiative and other container runtime projects is bolstered by CoreOS' technology, providing end users with more choices in container runtimes. This move is expected to accelerate Cloud Native strategies across enterprises, with Red Hat investing in improvements that will benefit the entire community.

The blog post provides a comprehensive guide on deploying an OpenShift Origin multinode cluster on Amazon AWS using CloudFormation and Ansible, targeting non-production environments. It critiques existing solutions for being either too complex or too simple, and presents a middle-ground approach developed by the author. The process involves using a simple CloudFormation template to set up the AWS infrastructure, which includes a master node, worker nodes, and a secure network configuration. The installation of OpenShift Origin on CentOS is automated with Ansible playbooks, while specific configurations like region labels, authentication setup, and pre-flight check adjustments are highlighted to optimize resource use and deployment efficiency. Post-deployment, the guide suggests installing the Sysdig agent for monitoring and securing the OpenShift cluster, providing detailed instructions on setting up service accounts and deploying the Sysdig daemonSet. The narrative emphasizes the method's adaptability and encourages readers to customize the provided scripts for their specific use cases.

The article explores a technical issue related to a Linux container that fails to start due to a mismatch between the container image and the host kernel version. This problem arises from an incompatibility between the vsyscall and vDSO mechanisms, with the older container image relying on the deprecated vsyscall interface, which the customer's kernel does not support in its 'none' mode, leading to a segmentation fault. The issue is demonstrated through a detailed troubleshooting process that involves system calls, page faults, and memory access analysis, revealing that while the vsyscall mechanism provides a fast-path for frequent system calls, it poses security risks and performance inefficiencies compared to the vDSO. The narrative underscores the importance of using updated container images for compatibility and performance, as well as understanding the evolving interactions between system libraries and kernel configurations.

Sysdig Monitor has enhanced its capabilities by integrating with Sysdig Inspect, facilitating more efficient troubleshooting of container issues in production environments. This integration enables users to automatically trigger Sysdig capture files upon any alert condition, providing comprehensive system call and OS event data to aid in root cause analysis. By eliminating the need to log into production hosts or transfer files across tools, the combined platform offers significant time savings and a streamlined process for identifying and resolving problems. Users can explore detailed summaries and metrics within Sysdig Inspect to diagnose issues, such as frequent Kubernetes pod restarts, by examining system calls and application activities. This approach speeds up troubleshooting efforts, as exemplified by quickly pinpointing the cause of a CrashLoopBackOff condition in a Kubernetes environment through intuitive navigation and analysis of capture files.

Meltdown and Spectre are two significant hardware vulnerabilities that exploit the speculative execution feature in CPUs, affecting billions of devices across various manufacturers like Intel, ARM, and AMD. Meltdown allows attackers to access kernel memory, particularly impacting Intel and some ARM CPUs, while Spectre targets the isolation between applications, posing a risk to user data across all CPU manufacturers. Patches have been released to mitigate these vulnerabilities, but they often lead to a notable decrease in system performance, with impacts ranging from 5% to 30% on CPU performance, particularly in database workloads and virtual machines. Organizations are advised to implement monitoring and alerting systems, consider auto-scaling or vertically scaling instances, and possibly disable kernel fixes if performance is severely degraded. Despite the hype surrounding these exploits, their widespread nature and potential impact on system performance underscore their seriousness.

The blog post discusses an experiment conducted to understand how attackers exploit Kubernetes clusters by setting up a honeypot with an intentionally exposed API server port. The experiment used Sysdig Secure to monitor for suspicious activity, which detected two notable attacks: a cryptojacking attempt and a second attack involving Linux.BackDoor.Gates DDOS malware. The cryptojacking attack involved trying to exploit the host's resources for Bitcoin mining by manipulating the system's crontab file, although it failed due to a malformed entry. The second attack saw an intruder using a container's shell access to run a program that attempted to embed itself in system directories and persist through reboots. The detailed forensics provided by Sysdig Secure and Sysdig Inspect allowed the researchers to reconstruct these attacks and understand the attackers' methods, highlighting the need for robust security measures in containerized environments. The post concludes by suggesting the closure of the open Kubernetes instance to prevent further unauthorized access.