Correct Is Not I/O Efficient: Characterizing I/O Behavior of LLM Agents on Scientific Data

Characterizes the I/O behavior of LLM agents working on scientific data: an agent can reach a correct answer while doing far more I/O than the task needs.

LLM coding agents are increasingly used to generate scientific workflows, but evaluations stop at output correctness. For data-intensive science, this is incomplete: two workflows can produce the same valid result while placing different pressure on storage systems. We study I/O behavior in correctness-passing LLM-generated scientific workflows. Contributions: a benchmark of scientific data tasks paired with expert implementations and a containerized execution harness; POSIX-level tracing of agent-emitted code via DFTracer LD_PRELOAD inside isolated subprocesses; and per-step decomposition of I/O into the final solution path versus exploration overhead (failed attempts, inspection, verification, retries). We evaluate correctness-passing agent-generated workflows across several models and scientific tasks, then compare them with expert implementations. The results show that valid agent-generated scientific code can issue far more POSIX operations and read substantially more data than expert baselines, and that exploration overhead varies by model. The takeaway is that correctness alone is not a sufficient proxy for I/O efficiency, and storage behavior should be measured directly before deploying agent-generated workflows on shared HPC or cloud storage.