Izzet Yildirim
Notes

Using an LLM’s thinking time to prefetch its data

·6 min read

Give an LLM agent a scientific data task, say a quality-control manifest over a few hundred genome alignments sitting on a parallel file system, and watch where the time goes. Each turn, the model first thinks: it plans the next step, decides which files to look at, and drafts the code. Then the tool runs, and it spends most of its time reading those files from cold storage.

Across the four benchmarks I measured, tool execution takes 53–93% of a session’s median wall time, much of it reading cold scientific data. And while the model is thinking, storage sits completely idle.

That idle window is the opportunity. The reasoning is streamed live, and it describes what the next tool is about to read. If something could read along and move those files into fast local storage before the tool starts, the tool’s reads would land hot. That is what AgentStage does.

The window is real, and wide enough

Two things have to be true for this to work. The thinking phase has to say enough about the files, and it has to last long enough to move them.

On the second point, the thinking phases I recorded ran from 2 to 106 seconds. The storage tiers I measured can stage 15–50 GB into a hot tier within one thinking phase, which covers datasets of up to tens of gigabytes. The six workloads in the paper range from 506 MB to 26 GB.

Here is what that buys, drawn as one turn:

One agent turn (illustration, not to scale)Model thinksTool reads cold dataModel thinksStage filesTool reads hot dataWithout stagingWith AgentStage00s05s10s15s20s25s30s35s40s45s50s

Staging overlaps the thinking the agent was going to do anyway, so the tool starts on data that is already local and decompressed.

Why “just prefetch” doesn’t work

The obvious version, copying files early, runs into three problems.

  • The model rarely names files. It says “the genomic VCF files in the project root” or “the FOO_BAR_*.h5 series,” not a path. In 58 of 72 recorded runs, the reasoning never named a complete file path.
  • The candidate set is bigger than the read set. “The training images” can match a whole workspace (22,500 files in one benchmark), while the tool opens only the batch the agent names. Stage too much and you waste the window; too little and the tool stays on the cold path.
  • Moved bytes aren’t ready bytes. In 5 of the 6 workloads, inputs had to be decompressed, decoded, or paired with a small index file before a tool could read them. A BAM alignment, for instance, needs its companion BAI index for random access.

How AgentStage works

AgentStage handles each problem with one component, without changing the agent or its tools:

copy and decompressopen()stagednot stagedLLM streamCapture proxyTiered detectorRule libraryFilesystem probesls, findStaging daemonCold tierparallel FSHot tiermemory, then NVMeTool subprocessLD_PRELOAD shim
  • A capture proxy wraps the model provider’s streaming client and hands each piece of thinking to the detector the moment it arrives. On a replay of over 10,000 stream events, it added no measurable latency (p99 overhead ≤ 1.001×).
  • An auto-generated rule library turns vague intent into files. Before a session, a generator reads only the task instruction and the workspace’s file inventory and writes rules for the ways a model tends to refer to things: band 08, band-08, and band8 all point to the same scene. No rules are written by hand, and none are tuned per model.
  • A tiered detector matches the stream against those rules and sizes each match: up to 20 files are staged right away, up to 200 when there is room, and anything larger waits until the agent narrows it down. The agent’s own cheap probes, like the ls that opens 96% of sessions, narrow the candidates as they come back.
  • A staging daemon copies files from the cold tier into memory (or local NVMe when memory runs out), decompressing them when the tool expects raw bytes and bringing index files along.
  • A small LD_PRELOAD shim in the tool’s process redirects each open() to the staged copy if there is one, and otherwise lets it fall through to cold storage. I chose LD_PRELOAD over FUSE because a tool makes hundreds of stat and open calls while starting up, and FUSE’s per-call overhead would dominate exactly there.

What it bought

I ran three I/O-heavy scientific tasks (genomics, astronomy, and a two-domain satellite and astronomy set) with four reasoning models: Claude Sonnet 4.5, Claude Haiku 4.5, Gemini 2.5 Flash, and the open-weight Qwen3.6-27B, which I served myself on NCSA’s DeltaAI.1

To separate the mechanism from the model’s randomness, I recorded each live session and replayed it twice against a verified-cold cache, once without AgentStage and once with it, pacing the reasoning at its original speed.

  • 1.74× faster sessions on average across the 36 cells (4 models × 3 tasks × 3 runs), and up to 2.34× on a single cell.
  • The detector fires early. The first correct match comes a median 0.70 s into the reasoning, which leaves about 97% of the thinking phase for staging.
  • The model barely matters. Per-model means ranged from 1.52× to 1.88×, and the open-weight model landed in the same band as the closed ones.

I also ran one data-intensive task from each of three community benchmarks, MLE-bench,2 KramaBench,3 and DSBench,4 which AgentStage was not built around:

Mean session speedup by task00.20.40.60.811.21.41.61.82GenomicsAstronomyCross-archiveMLE-benchKramaBenchDSBenchSpeedup (×)

When it doesn’t help

That spread on the right is the more interesting part, because it is predictable. Two factors decide the outcome:

noyesnoyesIs most of the sessionspent reading cold data?Little to gainDSBench: 1.03×Can the files be stagedbefore the tool starts?Part of the gainMLE-bench: 1.36×Most of the gaincurated tasks: 1.74×
  • I/O share sets the ceiling. Staging can only hide the cold reads a session actually does. DSBench reads three files and spends its time fitting a model, so 1.03× is the ceiling itself, not a failure.
  • Timeliness decides how much of it you get. MLE-bench’s 22,500 small images take over 100 seconds to stage, well past its 47-second thinking window, so it gets only part of its ceiling. The curated tasks stage in 9 to 22 seconds, inside a 35 to 81 second window, and realize 91% of their ceiling on average.

Both factors can be estimated before running anything. A simple cost model, calibrated once per storage tier from a file count and total size, predicts each workload’s cold-read time with R² = 0.998. So you can tell in advance whether a workload is worth staging.

What comes next

AgentStage hides the data side of an agent’s latency. Work like speculative tool execution5 and KV-cache reuse6 hides the compute side, and the two compose. The next questions are about scale: fleets of agents sharing the same data, which my SC ’26 poster StageOnce starts on, and long-running agents whose working set changes from turn to turn.

The paper is in eScience ’26. The paper page has the PDF and BibTeX, and the code is on GitHub.

References

  1. B. Bode et al., “DeltaAI: A National Resource for AI/ML Research,” 2025. doi:10.1145/3708035.3736062 ↩

  2. J. S. Chan et al., “MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering,” 2024. arXiv:2410.07095 ↩

  3. E. Lai et al., “KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes,” 2025. arXiv:2506.06541 ↩

  4. L. Jing et al., “DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?” 2024. arXiv:2409.07703 ↩

  5. Y. Sui et al., “Act While Thinking: Accelerating LLM Agents via Pattern-Aware Speculative Tool Execution,” 2026. arXiv:2603.18897 ↩

  6. W. Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP ’23. doi:10.1145/3600006.3613165 ↩