Makes out-of-core analysis of very large I/O traces fast on HPC systems.
Abstract
I/O analysis is an essential task for improving the performance of scientific applications on high-performance computing (HPC) systems. However, current analysis tools, which often use data drilling techniques (iterative exploration for deeper insights), treat every query independently and do not optimize column data for data-slicing (extracting specific data subsets), resulting in subpar querying performance. In this paper, we designed IOMax, a tool for efficient data drilling analysis on large-scale I/O traces. IOMax utilizes a novel query optimization technique to improve the query performance by 8.6x while reducing the memory footprint required for analysis by 11x. Additionally, it employs data transformation techniques to improve data-slicing performance by up to 11.4x. In conclusion, IOMax optimizes I/O analysis for scientific workflows on the Lassen supercomputer, resulting in up to 7x improvement.
@inproceedings{yildirim2023iomax,
title = {{IOMax: Maximizing Out-of-Core I/O Analysis Performance on HPC Systems}},
author = {Yildirim, Izzet and Devarajan, Hariharan and Kougkas, Anthony and Sun, Xian-He and Mohror, Kathryn},
booktitle = {Proceedings of the SC '23 Workshops of the International Conference for High Performance Computing, Networking, Storage, and Analysis},
series = {SC-W 2023},
pages = {1209--1215},
publisher = {ACM},
year = {2023},
month = nov,
doi = {10.1145/3624062.3624191},
}
AgentStage: Exploiting LLM Thinking for Data Staging in Scientific Agents
eScience ’26 · Conference paper · First author
optimize
Uses an agent’s thinking time to stage data from cold storage into a tiered node-local cache (memory, then NVMe) before the agent asks for it.
A Multifaceted Approach to Automated I/O Bottleneck Detection for HPC Workloads
SC ’22 · Poster · First author
diagnose
DigIO, a tool that finds I/O bottlenecks automatically from a multifaceted view of a workload’s traces; the precursor to WisIO.
Correct Is Not I/O Efficient: Characterizing I/O Behavior of LLM Agents on Scientific Data
GCASR ’26 · Poster · First author
observe
Characterizes the I/O behavior of LLM agents working on scientific data: an agent can reach a correct answer while doing far more I/O than the task needs.
DFTracer: An Analysis-Friendly Data Flow Tracer for AI-Driven Workflows
SC ’24 · Conference paper · Co-author
observe
An analysis-friendly data-flow tracer for AI-driven workflows, developed at LLNL.
GDSight: Per-Operation Cross-Layer Attribution for GPUDirect Storage
PDSW ’26, at SC ’26 (accepted) · Workshop paper · First author
observe+analyze
Per-operation, cross-layer attribution for GPUDirect Storage: where a single GPU read spends its time, from the application down to the device.
Exploring the Impacts of Multiple I/O Metrics in Identifying I/O Bottlenecks
SC ’23 · Poster · First author
diagnose
Shows that I/O time, bandwidth, and IOPS each expose different bottlenecks in HACC, Montage, and CM1.
IOMax: Maximizing Out-of-Core I/O Analysis Performance on HPC Systems
PDSW ’23, at SC ’23 · Workshop paper · First author
analyze
Makes out-of-core analysis of very large I/O traces fast on HPC systems.
Performance Analysis of Containerized OrangeFS in HPC Environment
SC ’21 · Poster · First author
analyze
Compares five container runtimes hosting the OrangeFS parallel file system; Singularity came closest to bare metal.
ORION: Observability Framework for Hybrid Workflow-Centric I/O Monitoring in HPC Systems
ProTools ’26, at SC ’26 (accepted) · Workshop paper · Co-author
observe
An observability framework for hybrid, workflow-centric I/O monitoring on HPC systems. I co-authored it.
Optimizing I/O for an Exascale Implicit Kinetic Plasma Simulation using the Rabbit Storage System
REX-IO ’25, at IEEE Cluster ’25 · Workshop paper · Co-author
optimize
I/O optimization for an exascale implicit kinetic plasma simulation on LLNL’s Rabbit near-node NVMe storage.
Recorder: Comprehensive Parallel I/O Tracing and Analysis
Preprint, under review · Journal paper · Co-author
observe
Comprehensive parallel I/O tracing and analysis across HPC I/O libraries.
StageOnce: Speculative Cross-Job Data Staging for LLM Agent Fleets
SC ’26 (accepted) · Poster · First author
optimize
Speculative cross-job data staging for fleets of LLM agents.
ToolFuse: Provenance-Aware In-Flight Coalescing of Redundant Agent Tool Calls
SC ’26 (accepted) · Poster · First author
optimize
Provenance-aware, in-flight coalescing of redundant agent tool calls.
WisIO: Automated I/O Bottleneck Detection with Multi-Perspective Views for HPC Workflows
ICS ’25 · Conference paper · First author
analyze+diagnose
Automated I/O bottleneck detection for HPC workflows, using correlated views of the same trace from several perspectives.