June 2024 Summaries

Many teams use Airflow to manage multi-stage workflows, but when scaling from local to production, it relies on Celery or Kubernetes, which can be difficult and time-consuming to set up. Modal is a simpler way to manage GPUs and containerized environments, making it ideal for AI/ML workflows. Modal can be triggered directly from an Airflow DAG and serves as a replacement for the Celery or Kubernetes executor. The process involves installing Modal in the Airflow environment, setting token IDs and secrets, and using either the `lookup` function to deploy functions or creating a custom operator that uses Modal Sandboxes to run Python code. This allows teams to isolate their task environment from their Airflow environment, making it easier to manage GPUs and containerized environments.

Modal developed seccheck, a syscall-based runtime analysis system to detect and stop cryptomining abuse on its serverless GPU container platform, where fraudsters may use stolen payment information to occupy costly NVIDIA GPUs. Earlier metadata-based heuristics, such as checks on user profiles, IP addresses, and payment signals, remain a first defense but can produce false positives and are vulnerable to evasion. Because users can run arbitrary Linux code and download software at runtime, Modal rejected source, binary, and container-image analysis in favor of monitoring system calls, which reveal operational behaviors such as suspicious subprocess launches, file access, and network connections commonly associated with miners. Rather than use the slow ptrace-based approach employed by strace, Modal relies on gVisor’s existing syscall interception infrastructure to stream trace events through a socket to seccheck with limited overhead. The system evaluates selected syscall events, particularly execve calls, against detection rules, immediately terminates flagged containers, and links incidents to user accounts for potential bans; Modal reports no false positives so far and plans to expand rules, correlate sequences of calls, update blacklists, and potentially inspect GPU control activity.