March 2022 Summaries

The text discusses the capabilities and benefits of using Datadog Real User Monitoring (RUM) for hybrid mobile applications, which integrate web-based content into mobile platforms like iOS, Android, React Native, or Flutter. It highlights the challenges of monitoring hybrid apps, where traditional tools may not offer a complete view by only tracking either web or native sessions. RUM addresses this by collecting and connecting data from both sources, providing a unified insight into user journeys. This allows developers to visualize user interactions, identify performance bottlenecks, and troubleshoot errors more effectively. It also offers performance metrics for both web and native components, aiding in optimizing app functionality and design. RUM's analytics features, including session timelines and product analytics, help developers understand and enhance user engagement, ultimately improving the overall user experience. Existing Datadog customers are encouraged to explore hybrid app monitoring with RUM, while new users can try it with a 14-day free trial.

The Linux kernel's "Dirty Pipe" vulnerability allows an unprivileged process to write to any file it can read, even without write permissions. This primitive enables privilege escalation by overwriting critical files like `/etc/passwd`. The exploit is particularly concerning in Kubernetes environments where containers are isolated from the host system. A proof-of-concept exploit demonstrates how an attacker can escape a container and gain host-level administrative privileges using this vulnerability. To mitigate this risk, it's recommended to ensure containerized workloads don't run as root, use validating admission controllers to restrict image deployment, leverage AppArmor or SELinux for security, and consider using Datadog Cloud Workload Security to detect potential exploits in real-time.

AWS has introduced an update allowing Lambda customers to provision up to 10 GB of ephemeral storage for each function, making them suitable for data-intensive workloads such as machine learning inference, large media file processing, and financial analysis. Datadog is supporting this update by enriching existing metrics with a new tag to help users monitor their functions' ephemeral storage usage and capacity. The Lambda extension now automatically tags all functions by storagesize, enabling users to query or filter them by storage configuration from within Datadog’s Serverless view.

AWS Lambda has announced that customers can now provision up to 10 GB of ephemeral storage for each function, making it well-suited for new, data-intensive workloads such as machine learning inference and large media file processing. Datadog is supporting this announcement by enriching existing metrics with a new tag to help track functions' ephemeral storage usage and capacity. AWS users can configure their functions' ephemeral storage programmatically or through the console, while Datadog's Lambda extension automatically tags functions by `storagesize`, enabling queries and filtering by storage configuration within Datadog's Serverless view.

In 2020, distributed tracing was released for AWS Lambda functions written in Python, Node.js, and Ruby. Since then, support has been expanded to Java and Go runtimes, and real-time distributed tracing is now available for .NET Lambda functions. This feature, powered by the Datadog Lambda Extension, allows developers to trace all requests across their entire infrastructure with minimal overhead. Tracing .NET Lambda functions can help identify issues such as exception errors and cold starts, which can degrade performance and negatively impact user experience. Datadog APM provides an end-to-end view of requests traversing any component of the infrastructure, enabling users to spot pain points without code or configuration changes. It also helps detect cold starts automatically and supports integration with other serverless telemetry available through Datadog's Lambda Library and Lambda Extension.

Datadog has released real-time distributed tracing support for .NET Lambda functions, providing comprehensive visibility into serverless applications and enabling developers to identify and resolve issues such as exception errors and cold starts. With this feature, .NET developers can trace all requests across their entire infrastructure with minimal overhead, spotting pain points anywhere in the system without code or configuration changes. Datadog's APM provides an end-to-end view of requests as they traverse any component of the infrastructure, including Lambda functions, containers, and on-prem hosts, enabling developers to easily identify bottlenecks and determine whether .NET Lambda function errors are behind issues. The feature also detects cold starts automatically, allowing developers to isolate and view their durations, and provides code-level insights into .NET Lambda functions, such as divide-by-zero or null reference errors. Datadog's full support for .NET Lambda functions enables real-time visibility into serverless applications, correlating with other telemetry available through the Lambda Library and Extension to gain further insight into any issue.

The Dirty Pipe vulnerability is a significant security issue in Linux-based systems that allows underprivileged processes to write arbitrary data to readable files, leading to privilege escalation. The vulnerability was introduced in the Linux kernel through a code refactoring and was first publicly disclosed by Max Kellermann on March 7, 2022. It has been fixed for all currently maintained releases of Linux, but many systems may remain vulnerable due to their older kernel versions. The vulnerability can be exploited to break out from unprivileged containers, including those used by Docker and Kubernetes, and can be detected using tools like Datadog Cloud Workload Security. To remediate the vulnerability, users need to ensure their Linux systems are running a kernel version of 5.16.11 or more recent, and major Linux distributions have released dedicated security bulletins to help mitigate the issue.

In this Datadog Spotlight series, Jackie Chua, a Recruiter II based in New York, shares her journey to Datadog and how she has grown alongside the teams she supports. She discusses her transition from agency recruiting to in-house recruiting, overcoming challenges, and supporting diversity and equity in hiring. Jackie also mentions her role in enabling new hires and creating more standardized processes across their teams. She is excited about bringing more diversity to Datadog and shares the impact of her work on the business.

Sara Verdi is excited to introduce Jackie Chua, a Recruiter II based out of New York, as the latest feature in Datadog's Spotlight series. Jackie has grown alongside the teams she supports and has achieved success in her role by strategizing with stakeholders on team-building, learning the ins-and-outs of roles, and mentoring new hires. She overcame a challenge by transitioning from agency recruiting to in-house recruiting, partnering with her manager and shadowing other recruiters to learn how to collaborate effectively. Jackie has grown in her career at Datadog, taking on new challenges such as optimizing workflows and leading training sessions. She supports diversity and equity in hiring by considering diverse perspectives and hiring candidates from underrepresented groups, including women and people of color. Jackie encourages others to join Datadog's teams around the world, highlighting the company's growth and opportunities for impact.

Logrotate is a Linux utility designed to simplify log management and maintenance within large-scale production environments through a process known as log rotation. It enables system administrators to efficiently collect logs from all of their system components, manage disk space usage, and ensure that valuable log data is not lost due to resource constraints. The tool offers various configuration options for controlling log rotation schedules, compressing files, running scripts for additional processing, modifying permissions, adding timestamps to logs, debugging issues with the utility, and configuring environment resources to log at the appropriate level. By leveraging logrotate in conjunction with a log forwarding service like Datadog Agent, teams can centralize, store, analyze, visualize, and alert on their logs for improved visibility into application errors and performance.

Datadog Summit, an in-person event celebrating community and knowledge sharing, will take place on April 13th in Denver. Attendees can expect presentations from engineers and developers who have successfully implemented observability cultures within their organizations. Practical advice from community members using data and insights from Datadog to improve system performance, security, and reliability will also be shared. The event will feature hands-on workshops covering various topics such as infrastructure monitoring, distributed tracing, log management, APM, threat detection, and measuring Core Web Vitals in production. Space is limited, so RSVP now to reserve your seat at the Denver Marriott Tech Center.

Usman Khan` and `Mallory Mooney` are discussing the importance of logging to files, which provides valuable information for administrative tasks such as analyzing application performance, debugging system errors, and investigating security and compliance issues. However, managing large volumes of logs can be challenging due to resource constraints. To address this issue, they introduce `logrotate`, a Linux utility designed to simplify log management and maintenance through a process known as log rotation. Log rotation solves problems such as disk space consumption and the need for manual file searching by creating new files on a schedule and renaming old files. The authors will guide readers through customizing the `logrotate` utility to fit their logging needs, including adjusting the log rotation schedule, compressing files to save disk space, running scripts for additional processing, modifying permissions, adding timestamps to logs, debugging issues with the `logrotate` utility, and configuring resources to log at the appropriate level. The authors emphasize that log rotation is not a substitute for using a log forwarder and will provide examples of how to configure `logrotate` for various Linux distributions, including Ubuntu 20.04.3 LTS.

Datadog is hosting its next summit in Denver on April 13th, celebrating the community that helps improve the platform. The event features talks from engineers and developers who have transformed their organizations through observability, as well as hands-on workshops covering topics like infrastructure monitoring, distributed tracing, security, and more. These workshops are designed to provide practical advice and real-world insights, allowing attendees to understand how their systems are performing and quickly find and resolve issues. The summit will also feature the Datadog team showcasing its latest product features and answering questions. With limited space available, attendees are encouraged to RSVP now to secure a spot at this free event held at the Denver Marriott Tech Center.

Datadog's Cloud SIEM has introduced the impossible travel detection rule type to help identify suspicious login activity indicative of security breaches. This feature analyzes user logs to detect if a user has traveled between locations at an impossible speed, which may indicate unauthorized access attempts. Users can create and apply these rules using Datadog's log search syntax and group by dimensions. To minimize false positives, users can fine-tune their rules with suppression lists or enable baseline user location tracking to exclude logs from trusted users. The feature requires the queried logs to include the standard @network.client.ip attribute for location data extraction.

Datadog's Cloud SIEM now offers an "Impossible Travel Detection Rule" to identify anomalous login patterns, helping detect security breaches by analyzing user locations across the globe. This rule type analyzes logs to determine if a user has traveled between locations at an impossible speed, flagging suspicious activity and generating Security Signals to notify users. To create effective rules, users can specify log search syntax and group values by dimension, while fine-tuning rules with suppression lists to minimize false positives. By enabling this feature, organizations can improve their security posture and detect potential threats in real-time.