June 2021 Summaries

A large-scale attack targeting Kubeflow instances to deploy malicious TensorFlow pods for Monero cryptocurrency mining in Kubernetes environments has been identified by Microsoft. Kubeflow and TensorFlow, popular open-source frameworks for machine learning tasks, provide an ideal target due to their high processing power and GPU access. The attack exploits the configuration of Kubeflow dashboards exposed online via the Istio ingress gateway, allowing attackers to deploy TensorFlow images and execute malicious code without detection. Although the value of cryptocurrency has decreased, such attacks remain prevalent, exploiting legitimate images within machine learning clusters to maximize computational gains. Detection and mitigation strategies include implementing strong authentication mechanisms, keeping management dashboards private, and using tools like Falco and infrastructure monitoring to detect unusual resource usage and suspicious activities.

Carlos Tolon discusses the integration of the Prometheus Blackbox exporter with Sysdig for monitoring availability metrics, emphasizing its utility in Kubernetes environments. The Blackbox exporter allows exploration of endpoints over various protocols, generating metrics such as endpoint status, response time, and certificate expiration. It can be easily deployed in a Kubernetes cluster using Helm, with configurations allowing for HTTP probes and authentication options. The Blackbox exporter works in tandem with the Sysdig Agent through Prometheus service discovery to monitor and alert on key metrics such as probe success, probe duration, and certificate expiry, aiding in the detection of endpoint failures and latency issues. The setup facilitates comprehensive monitoring and visualization of target metrics, with the potential for alerts on critical events like certificate expiration, enhancing the reliability and security of services.

Sysdig's June 2021 update highlights various enhancements across its platform, focusing on improved security, user experience, and integration capabilities. Key updates include the introduction of Falco Feeds, providing continuously updated rules for threat detection, and the beta release of the Falco Policy Tuner, which automates the tuning of noisy rules and false positives. Enhancements in Kubernetes Network Security offer finer configuration and improved user interfaces, while the addition of GCP rules for cloud security aims to parallel AWS CloudTrail integration. The update also introduces the PromQL Library and Prometheus Remote Write to extend Sysdig Monitor's capabilities. Additionally, Sysdig Secure now supports the CIS RedHat OpenShift Container Platform v4 Benchmark, and various UX improvements have been made to the "Investigate" navigation and Activity Audit features. The Sysdig Agent has seen updates, including support for AWS ECS Anywhere and a new slim agent for eBPF probes, alongside numerous bug fixes and enhancements in the Sysdig CLI, Python SDK, and other tools.

Vulnerability assessment and management are crucial practices for minimizing potential security threats in software infrastructure, considering the increasing complexity of software and the pressure for rapid feature deployment. Despite inevitable software vulnerabilities, implementing best practices such as vulnerability scanning during development, continuous image scanning, and host scanning are essential to evaluate and manage security risks effectively. Emphasis is placed on blocking vulnerabilities before they reach production through inline scanning and maintaining updated versions of software to mitigate exposure to risks. Runtime security tools like Falco are recommended for detecting and managing threats as they occur, and conducting forensic investigations post-exploitation to prevent future incidents. Effective vulnerability management involves implementing company-wide policies to prioritize vulnerabilities based on severity and impact, as well as using comprehensive reporting tools to facilitate quick response and compliance checks. Sysdig Secure is highlighted as a solution that consolidates these practices into a single platform, offering a streamlined approach to securing cloud environments and ensuring compliance across infrastructure.

AWS has launched the CloudFormation Public Registry, a searchable collection of extensions that simplifies the discovery, provisioning, and management of AWS and third-party resources, including a new Sysdig extension. This extension automates the deployment of the Sysdig agent to Amazon EKS clusters, allowing for quick and secure installation at scale through various interfaces. Users need a Sysdig API Token to connect installations to their Sysdig account and must configure an IAM role with specific permissions for the extension to access EKS clusters. The article outlines the steps to enable and customize the Sysdig extension, including creating IAM roles and configuring the CloudFormation template, which helps streamline Sysdig agent deployment on EKS clusters. The integration of CloudFormation with Sysdig supports infrastructure as code practices, enhancing the ability to scale deployments, reduce manual intervention, and accelerate application deployment in production environments.

Cloud security is increasingly complex due to the dynamic nature of cloud environments and the adoption of DevOps practices, which accelerate application delivery but also require relinquishing some control over configurations. Misconfigurations remain a prevalent cause of data breaches, as highlighted by high-profile incidents and reports from industry experts like Gartner. To combat this, organizations are encouraged to adopt continuous cloud security posture management (CSPM), utilizing tools that provide real-time monitoring of cloud configurations and suspicious activities. The integration of cloud and container security into unified platforms helps streamline operations, enhance security, and reduce the complexity of managing multiple tools. Open-source solutions, such as Falco and Sysdig, are gaining traction by offering community-driven innovation and interoperability, enabling organizations to build robust security frameworks that are adaptable to the evolving threat landscape. The shift towards open-source technologies mirrors broader industry trends, emphasizing the need for standards and collaboration in facing modern security challenges.

This blog post by David de Torres Huerta discusses the intricacies of setting appropriate Kubernetes resource limits to optimize performance and prevent issues such as CPU throttling and memory starvation. The article, part of a Kubernetes capacity planning series, emphasizes the importance of detecting containers lacking resource limits and offers strategies for setting limits using Prometheus for monitoring. Two approaches are suggested: a conservative strategy, which bases limits on the maximum resource consumption observed, and an aggressive strategy, which allows for occasional spikes by setting limits at the 99th percentile. The post also addresses the risks of resource overcommitment at both the cluster and node levels, advising on strategies to maintain balance and avoid unnecessary infrastructure costs. Additionally, it highlights the utility of Sysdig's out-of-the-box Kubernetes dashboards for identifying underutilized resources and facilitating effective capacity planning without needing expert knowledge.

Sysdig's recent introduction of the automated Falco Rule Tuner aims to streamline the process of customizing Falco's security rules, reducing alert fatigue by automatically identifying and suggesting exceptions for noisy policies. While default security rules are essential for broad threat detection, they often generate excessive false positives, overwhelming administrators and potentially leading to missed critical alerts. The Falco Rule Tuner automates the analysis and refinement of these rules daily, allowing security teams to decide which exceptions to retain. This enhancement also includes an upgrade to Falco's rule syntax, benefiting the broader Falco community by enabling more efficient management of security alerts across both container runtimes and cloud infrastructures. Despite advancements in machine learning for security, human oversight remains crucial, and Sysdig's tool provides a balanced approach to managing complex security environments effectively.