The Data Plane Is Yours: Dagster+ Hybrid on Kubernetes and Azure

A large enterprise Dagster+ Hybrid deployment on Kubernetes and Azure separates Dagster Labs’ managed control plane from a customer-operated data plane containing agents, code servers, and dynamically launched run workers, with a platform team owning infrastructure, security controls, GitOps templates, and capacity management while product teams contribute code and limited declarative configuration through pull requests. The deployment uses namespaces, environment-specific node pools, workload identities, admission policies, approved images, and controlled network routes to isolate workloads within a shared AKS cluster, while recognizing that shared cluster, network, subscription, and Azure quota domains do not provide the same isolation as separate clusters. It emphasizes tracing each workload across Dagster runs, Kubernetes Jobs and pods, and Azure node provisioning or quota events, since bottlenecks may surface in Dagster but originate in Kubernetes scheduling or Azure capacity. Resource sizing begins with conservative defaults and evolves through monitored CPU, memory, workload-shape, and business-cycle data, while concurrency limits must be tied to realistic pod fan-out, node allocatable capacity, and Azure quotas rather than run counts alone. The approach also distinguishes GitOps-managed standing resources from Dagster-managed runtime workloads, supports controlled upgrades and ownership boundaries, and aims to give product teams useful operational visibility without requiring direct cluster or Azure administration.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.