May 2021 Summaries

The blog post provides a collection of ten practical Prometheus query examples intended for monitoring Kubernetes clusters, leveraging Sysdig's expertise from assisting numerous customers. It includes queries for counting pods per namespace, identifying containers without CPU limits, detecting pod restarts, and listing pods not ready for troubleshooting purposes. Additionally, the article covers queries to detect CPU and memory overcommit, the number of ready nodes, node readiness flapping, and underutilized CPU and memory resources. Readers are encouraged to explore further resources like a PromQL cheatsheet, the Awesome Prometheus alerts collection, and a free hands-on lab to practice in a real environment, ensuring effective use of Prometheus for Kubernetes monitoring.

Amazon ECS Anywhere extends the capabilities of Amazon Elastic Container Service by allowing users to run containers on-premises as well as in the cloud, simplifying management through a consistent, managed control plane. This flexibility enhances deployment options and facilitates quicker application development while addressing security concerns with tools like Sysdig, which offers security and monitoring across hybrid cloud environments. Sysdig, integrated with the open-source Falco detection engine, provides runtime security, compliance checks, and visibility into container and cloud operations, aiding in threat detection and compliance with industry standards. Additionally, ECS Anywhere supports AWS Fargate for serverless computing, further broadening deployment choices without compromising security. This approach allows organizations to maintain infrastructure control for compliance while benefiting from AWS's managed services, offering a unified solution for container security and monitoring in diverse operational settings.

Sysdig's May 2021 update highlights the introduction of Falco Feeds, which enhances the capabilities of the Falco open-source security tool by providing companies with access to expertly crafted rules that are regularly updated in response to newly identified threats. This addition aims to bolster cloud defense strategies by offering continuously refreshed security insights.

CVE-2021-25737 is a low-severity vulnerability in Kubernetes' kube-apiserver that allows an authorized user to hijack network traffic by redirecting pod traffic to private networks through EndpointSlices, potentially leading to sensitive data leaks. This issue affects specific versions of Kubernetes, including v1.16.0 to v1.21. The vulnerability is due to the lack of validation preventing modifications of EndpointSlices to include addresses in the 127.0.0.0/8 and 169.254.0.0/16 ranges. Mitigation involves upgrading kube-apiserver to patched versions or implementing a validating admission webhook to prevent the creation and modification of EndpointSlices with such addresses. Detection of exploitation attempts can be achieved using the open-source tool Falco by enabling Kubernetes Audit Logging to monitor suspicious activities. The article emphasizes the importance of adhering to container security best practices and the role of tools like Sysdig Secure and Falco in safeguarding Kubernetes environments.

AWS App Runner is a new service designed to simplify the deployment and management of containerized applications without the need for users to handle infrastructure. It allows users to deliver their source code or container images, automatically building and deploying them while managing load balancing, encryption, and scaling. Sysdig, in collaboration with AWS, provides security enhancements for App Runner by utilizing tools like Falco to detect and alert on security anomalies, as well as scanning images for vulnerabilities. Sysdig Secure extends this functionality by offering Cloud Activity Insights, which helps identify cloud misconfigurations and provides a comprehensive overview of cloud activities, thus enabling users to promptly address security incidents and reduce risks. This integration of workload and cloud security insights aims to improve incident response and protect applications and infrastructure.

PostgreSQL monitoring with Prometheus is facilitated by the PostgreSQL Exporter, which provides an accessible way to track the health and performance of PostgreSQL instances, whether self-hosted or on AWS RDS. Key metrics for effective monitoring include checking if PostgreSQL is running, postmaster service uptime, replication lag, database size, available storage, number of available connections, latency, cache hit rate, memory availability, and requested buffer checkpoints. These metrics help in creating alerts for potential issues like server downtime, unexpected restarts, storage limitations, and performance bottlenecks. The article also suggests downloading a cheat sheet for deeper insights and offers pre-configured PostgreSQL monitoring dashboards to integrate with tools like Grafana or Sysdig Monitor.

Sysdig and LogDNA have partnered to create an integration that enhances observability by allowing users to view log events from LogDNA alongside other events within Sysdig, facilitating faster issue resolution and a more comprehensive understanding of infrastructure health. Observability, crucial for swiftly solving problems, typically involves metrics, logs, and traces, with Sysdig specializing in metrics collection in challenging environments like Kubernetes. The integration involves setting up alerts in LogDNA, which are then visible in Sysdig Monitor, complete with details such as alert names, matched log lines, and direct URLs to specific alerts in LogDNA. This collaboration underscores the importance of correlating logs and metrics to enhance visibility and problem-solving capabilities in complex systems.

PromCon Online 2021 showcased several key advancements and insights into Prometheus and related technologies, highlighting the growing ecosystem and community adoption. Among the notable features discussed were the Prometheus remote write function, which, despite being in the early stages of development, has gained popularity necessitating standardization efforts, and the Prometheus Conformance Program introduced to ensure compatibility across projects integrating with Prometheus. Presenters like Anais Urlichs and Bartlomiej Płotka demonstrated using Prometheus for automating Kubernetes rollouts by monitoring canary releases and triggering automatic rollbacks based on error rates. The event also featured lightning talks that covered a range of topics, from calculating historical values of recording rules and validating alert rule syntax, to gathering metrics from SNMP devices and predicting power consumption in data centers. The event emphasized the support available for Prometheus users, including resources provided by Sysdig, which adheres to Prometheus standards.

Kubernetes capacity planning is crucial for infrastructure engineers as it involves balancing cluster stability and resource utilization, ensuring containers have enough resources without unnecessary over-allocation, which can lead to increased costs and scheduling difficulties. The article outlines methods to identify and rightsize cluster capacity using open-source tools like kube-state-metrics and cAdvisor, which help avoid resource underuse by analyzing current resource usage and adjusting requests accordingly. PromQL queries can detect underutilized CPU and memory, allowing for charge-backs to departments with oversized namespaces and identification of the top containers wasting resources. The goal is to set requests between 85% and 115% of average usage, ensuring efficient deployment and resource allocation. Measuring the impact of optimizations can be done by comparing current resource usage against previous benchmarks. Sysdig offers out-of-the-box Kubernetes Dashboards to facilitate this process, enabling users to quickly identify underutilized resources and streamline Kubernetes capacity planning without needing expert-level skills.

AWS Fargate offers a compelling value proposition by allowing users to focus on software development without managing virtual machines, but it presents challenges in achieving deep visibility for security purposes due to limited OS access. Traditional instrumentation techniques like LD_PRELOAD and ptrace either lack accuracy or efficiency, complicating security measures for Fargate workloads. Sysdig addresses this with a new patented technology that optimizes the open-source pdig framework to provide precise data capture with minimal overhead, even for languages like Go. This advanced Fargate instrumentation offers the same level of security and performance as traditional kernel-based methods, ensuring comprehensive visibility and protection for serverless applications.

Kubernetes Dashboards offer a way to explore and understand the complexities of your cluster by providing visual insights and data. The article, published by Jesús Ángel Samitier, highlights the significance of these dashboards in monitoring and managing cluster activity. It discusses how tools like Falco Feeds, developed by Sysdig, enhance the capabilities of Kubernetes by providing access to expertly crafted security rules that are regularly updated to address new threats. The piece also emphasizes the importance of staying informed about cloud security and suggests engaging with experts to effectively protect cloud environments.

Securing AWS Fargate workloads presents unique challenges due to its serverless architecture and shared responsibility model, where users must ensure the security of their own applications despite AWS managing the underlying infrastructure. The article discusses using Sysdig Secure to enhance security by implementing File Integrity Monitoring (FIM), which is crucial for meeting compliance standards like PCI-DSS. It illustrates a scenario where suspicious filesystem changes were detected in a Fargate task using Sysdig's Falco runtime policies, highlighting the importance of threat detection and cloud security features like Cloud Activity Insights. The example demonstrates how unauthorized activities can be traced back to specific users and actions, underscoring the need for proactive security measures such as image scanning and user activity monitoring to prevent breaches. Sysdig's capabilities in providing unified threat detection for both container workloads and cloud infrastructure are emphasized, allowing for a comprehensive security strategy that includes real-time visibility and compliance assurance in AWS Fargate environments.