Skip to content

Evaluation, Tracking & Artifacts

Agentic work produces a lot of output fast. Without structure, you get a directory full of plot_final_v2_REAL.png — at machine speed. The fix is to make the agent show its work the way you'd make a student show theirs.

Artifacts: results live in files

Adopt one convention and put it in CLAUDE.md:

- All results go in artifacts/<task-name>/: plots (PNG), tables (CSV or
  parquet), metrics (metrics.json), and a REPORT.md explaining what was
  run, what came out, and what it means.

REPORT.md is the keystone. A result you can't re-explain in two weeks is not a result; making the agent write the explanation at generation time costs nothing and is the difference between a pile of plots and a lab notebook.

The agent reads images

This deserves its own headline: show the agent its own plots.

> Run the scan, save the corner plot, then look at the plot and tell me
> if anything looks pathological before we proceed.

Claude reads the PNG it just produced — catching the railed parameter, the bimodal posterior, the log axis that should have been linear. Plot → inspect → adjust → re-plot is a feedback loop where the measure is visual, and it's shockingly effective for physics, where half our debugging is "that plot looks wrong."

Tracking and logging

  • Log the configuration, not just the result — every artifact directory gets the exact parameters, code version (git rev-parse HEAD), and command that produced it. The agent will do this with perfect discipline if told once; humans never do.
  • Metrics as JSON — metrics.json per run makes results greppable and comparable; the agent can then answer "which of the 14 runs this week had the best χ²?" by reading files instead of memory.
  • Documentation as a side effect — end long sessions with "update NOTES.md and the docstrings to reflect what we learned." This is the compaction-survival habit doing double duty as documentation.

Git as the audit trail

Commit early, commit often, and let the agent write honest commit messages. The repo history becomes the experiment log: each commit a checkpoint you can diff, bisect, and revert to when an agentic excursion goes wrong. Cheap checkpoints are what make bold delegation safe.

The full loop, assembled

Goal → plan (modes) → implement against tests (feedback loops) → artifacts + report (this page) → commit → update NOTES.md (context). That cycle, run repeatedly, is agentic research. The walkthrough runs it for real.