ORION: Observability Framework for Hybrid Workflow-Centric I/O Monitoring in HPC Systems

An observability framework for hybrid, workflow-centric I/O monitoring on HPC systems. I co-authored it.

Scientific workflows depend on efficient data exchange, making practical, low-overhead I/O performance visibility essential for HPC optimization. Existing I/O monitoring tools force extreme choices, such as tracing all I/O events or using only coarse profiling, which either inflates data volume or hides attribution context. We design ORION, a workflow-centric I/O monitoring framework that enables users to dynamically choose temporal and spatial monitoring granularity for different regions. ORION provides multi-faceted I/O profiling across time, processes, and files at low overhead; captures critical I/O events in high-interest regions while preserving aggregated views elsewhere; and reconciles traces with profiles at 0.35M–1.15M events/s for scalable analysis. On large-scale workloads, including Montage and Llama fine-tuning, ORION Selective Aggregation identifies critical bottlenecks with 6.8×–36× fewer trace events at 100% ordered top-k severe-bottleneck accuracy, and is portable across two heterogeneous LLNL systems. These results demonstrate that adaptive-granularity monitoring makes high-fidelity I/O analysis practical at HPC scale.

Cuts trace volume 6.8–36× while keeping 100% of the top-k bottlenecks.