Evaluating a coding agent across sessions, not one prompt

If you want to know whether a coding agent is good at engineering, what should you watch it do?

One prompt tells you very little. Benchmarks answer a real question, but a narrow one. Take an issue-resolution benchmark such as SWE-bench. Its tasks come from real GitHub issues, and success means the agent's patch resolves the issue and the relevant tests pass. A result there gives evidence that an agent can navigate an unfamiliar codebase and produce a patch that passes those tests. That is useful to know. Still, a benchmark is a bit like a paper: publishing a good number doesn't by itself establish quality. The number won't tell you whether the agent's work holds up over a week, or whether a teammate would want to inherit it.

My view is that a day of real work on a real repository says more than a benchmark score. This essay proposes a more structured version of that observation. The idea is a bounded, multi-session project on one repository, designed to surface three things a one-shot issue task mostly misses: continuity, repair, and maintainability. It is a proposed protocol. I have no results to report.

What an isolated issue task leaves out

A one-shot issue task starts from a fixed codebase and ends with a scored patch. That isolation helps make the task reproducible. It does not test how the agent carries decisions or repairs its own work across later sessions.

Continuity. Does the second session build on the first? Or does the agent rediscover the same constraints, contradict earlier decisions, and redo finished work? A long project also tests whether the agent knows what to forget. Some details are worth carrying forward, and others are safe to drop.

Repair. When the agent's own earlier change causes a regression, does it trace the cause or patch the symptom?

Maintainability. Can a person extend the code, or is it a pile of locally correct edits that happen to pass tests?

Pass/fail on an isolated issue isn't designed to capture these. I'd expect them to show up across sessions, in the path the work took as much as in the final diff. Anthropic's published guidance on evaluating agents makes a related point: an agent's outcome and its trajectory are different things to assess, and an evaluation can look at several dimensions.

A proposed four-session protocol

Pick a bounded project. Use a modest real repository and a feature that needs several connected changes. For example, imagine adding a configurable export format to a small command-line tool. The work should be too big for one sitting but small enough that a reviewer can read every change.

Plan the sessions in advance:

1. Implement the core export path.
2. Add configuration and documentation.
3. Change a requirement so that it conflicts with a decision from session one.
4. Resolve a regression. It can be one the evaluator planted or one the agent introduced earlier.

Each session starts with what a human collaborator would have: the repository, its history, and any notes the agent chose to leave. The agent decides how to carry context forward. The question is whether it handled the handoff well, not which mechanism it used.

Score several dimensions:

• Outcome: tests pass, and new tests exercise the new behavior.
• Continuity: in session three, the agent either honors earlier decisions or revises them explicitly. It doesn't contradict them silently.
• Repair quality: the session-four fix addresses the root cause, and the agent can explain it.
• Maintainability: a reviewer rates the cumulative diff against a fixed rubric for readability, duplication, and scope.
• Trajectory: a reviewer reads the logs for wasted exploration, repeated mistakes, and risky moves such as deleting tests or making broad rewrites.

Leave judgment calls to people. Automation can check outcomes. For maintainability, a reasonable instrument is a reviewer asking, "Would I accept this pull request?" Two independent reviewers who have a way to reconcile disagreements are more defensible than one confident score.

A hypothetical comparison

What follows describes two invented agents. Neither is a real system, and nothing here was observed.

Agent A passes every test at the end of every session. In session three it handles the new requirement by adding a second code path beside the original instead of changing the original. In session four it fixes the regression by special-casing the failing input.

Agent B leaves a failing test at the end of session two because it stopped partway through refactoring a helper. In session three it notices that the new requirement conflicts with an earlier choice, changes that choice, and updates the documentation. In session four it traces the regression to its own session-one shortcut and removes it.

A pass-rate metric favors A. The protocol is designed so that B could score higher on continuity, repair, and maintainability. B's codebase is arguably the one most reviewers would rather inherit.

The comparison isn't as clean as it looks. B's failing intermediate state could be deliberate, or it could be carelessness. The rubric has to tell the two apart, which is one reason to read trajectories instead of scoring only end states.

Where the protocol can fail

The planted regression can leak. If the bug is obvious or sits in a conspicuous commit, the agent can fix it without understanding it. The "repair" score then measures pattern-matching on commit history.

Rubrics drift toward taste. If "maintainable" quietly means "written the way I would write it," the score measures stylistic agreement. Concrete questions keep the rubric honest. Is logic duplicated? Is the change scoped to the request? Could a newcomer follow it?

One repository is one sample. A careful multi-session run on a single project is an anecdote, not a distribution. This approach complements benchmarks. It doesn't replace them.

What each kind of evaluation can show

An issue-resolution benchmark shows whether an agent can complete a narrow, isolated task. A bounded multi-session project could show whether it keeps doing the work well as context accumulates and requirements change. The first produces a number. The second would produce evidence about the behavior that matters when an agent works on a shared codebase over time.