Using Ray Direct Transport for Fast and Easy Weight Syncing in Reinforcement Learning (Part 2)

Ray Direct Transport (RDT) is a Ray Core feature that simplifies RDMA-backed GPU-to-GPU tensor transfers between actors, targeting fast model-weight synchronization for reinforcement learning workflows involving large language models. RDMA bypasses operating-system networking layers to reduce latency and CPU overhead, but its benefits can be undermined by costly memory registration, metadata exchange, staging buffers, and inefficient transfers of many small tensors. Using a trainer and inference-generator example, the post explains how RDT integrates with NIXL to manage these operations while retaining Ray’s actor-based programming model. It recommends preregistering persistent tensor memory with `register_nixl_memory`, receiving data directly into model-weight buffers through `set_target_for_ref` to avoid duplicate memory use, and using preregistered memory pools to bucket small tensors into contiguous transfers. On two GB200 nodes connected through Multi-Node NVLink, these techniques achieved up to 859 GB/s for data transfer and improved end-to-end synchronization performance by as much as 7.5 times over a naive implementation. The authors also describe planned and ongoing integrations with RL frameworks including SkyRL and Miles, alongside further work to reduce metadata and Python-object transfer overheads.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.