Preparing high-quality training data for foundation models requires scalable pipelines that turn heterogeneous documents and videos into structured records. A central challenge is that each parent item can expand into an ordered, input-dependent sequence of children whose counts may be long-tailed — and GPUs need to batch those children across parents while preserving parent relationships, child order, completion status, and result routing.
According to a new paper, existing systems either hide parallelism behind coarse-grained jobs or expose flat records that force applications to manage lineage and regrouping themselves. RayOrch, presented as a programming model and distributed execution engine, aims to keep parent-child relations intact throughout execution.
In RayOrch, programs declare ordered variable-cardinality expansions and matching gathers. The compiler validates each expansion-gather pair, while the runtime records child membership, immediate parents, immutable ordinals, and terminal states. Per-Call FIFO Ready Queues batch ready children across parents for GPU execution, and gathers reconstruct results from declared membership and ordinals rather than from batch boundaries or completion order. Parents can advance as soon as all required children reach a terminal state, and typed parent-scoped failures suppress undispatched siblings of a failed parent while allowing unrelated parents to continue.
The paper reports strong scaling results on NVIDIA H20 GPUs: a 15.14x speedup when scaling MinerU from 4 to 64 GPUs, and a 7.82x speedup when scaling a video pipeline from 8 to 64 GPUs. In end-to-end terms, RayOrch reduces time by 13.1% versus Ray Data and 29.0% versus Daft on MinerU, and by 16.0% versus Ray Data on Docling.
The authors position RayOrch alongside systems like Ray Data and Daft for large-scale multimodal data processing, but targeted at pipelines with multiple stages and levels of granularity — such as documents processed as pages and videos as frames. Code is available at github.com/OpenDCAI/RayOrch.