How can agent work be continued, checked, and improved across models?
Current stateImplemented multi-model runtime with evidence-checked coordination and supervised source changes; local helper qualification remains experimental
I built Harness to keep tools, recorded work, and working context under one runtime as models change. Native and MCP tools share that runtime.
For Saoirse, its resident agent, continuity depends on retained task state and explicitly retrieved context. The runtime records what she was given, what she tried, and what resulted, so changes to prompts, tools, and coordination can be examined.
Why this approach
Why own the runtime?
Inside an existing coding-agent host, a plugin has to intercept the operations the host exposes. Harness owns dispatch directly, so it can record proposals and outcomes, apply its own checks, and preserve task state at the point of execution. That is why I kept Harness separate from agent-swarm instead of porting the plugin wholesale.
A candidate can be compared with an incumbent against frozen checks, and the decision to adopt it can remain a separate, reversible operation.
Improvement that carries forward
The longer-term aim is for an agent to improve the methods its successor inherits: tools, prompts, coordination, and eventually the challenges it learns from. The comparison and rollback mechanisms provide a way to test whether inherited changes improve later work, using checks the successor did not write.
Where it stands
Typed session events and immutable result blobs support resumable work, task evidence, handoffs, and a derived SQLite telemetry view. Configured memory and project-context sources are fetched through the normal dispatcher under explicit time and input limits. The terminal interface exposes activity, queued prompts, cancellation, and inspection; Session Desk also constructs the same Harness kernel for its Saoirse conversation.
Native subagents, rules-based routing, and ensemble, panel, draft-and-refine, and escalation strategies coordinate work with shared execution and usage budgets. MCP, plugins, model providers, local inference, and Claude Code, Codex, and Antigravity adapters connect different execution paths. Harness-dispatched permission checks do not imply confinement of every provider-native tool.
Source-improvement work prepares exact incumbent and candidate snapshots, runs bounded comparisons, and records explicit adoption and rollback. The selected source is launched in a new process; the management installation remains available. Separately, resident-helper studies measure intent, repair, and relevance decisions against frozen rules.
What the work has shown
A completed answer still has to pass the task checks
Ensemble selection now excludes completed candidates whose recorded evidence fails the task requirements. A retained regression test constructs a failing majority and a passing minority: the failing answers cannot win the vote. When a judge is used, it sees passing candidates and its own result must satisfy the same frozen checks.
Requirements can refer to exact output content, recorded tool arguments and results, or an explicit human review. Execution completion and acceptance therefore remain separate states. These checks establish the specified evidence contract; the evaluator does not infer arbitrary semantic correctness.
A source change can be compared, selected, and rolled back
The source experiment runner retains exact source snapshots, evaluation inputs, and results. Promotion and rollback are explicit journaled actions, and a new process can launch from the selected private copy. A retained demonstration compares a known historical regression with its fix and exercises selection and rollback.
That demonstration records a passed comparison while keeping activation qualification false. It uses fake providers and an operator-declared validation partition. It establishes the mechanics of supervised source replacement, while a useful newly authored improvement still needs its own evaluation.
Small helpers were measured against the rules they would replace
The original resident-helper study used twenty authored cases each for intent, repair, and relevance, with two repeats per profile. Qualification required at least eighteen correct decisions per function, every critical case, and a net gain over frozen lexical rules. Those rules scored 16/20, 12/20, and 14/20 respectively.
None of the plotted profiles qualified. Later binary-signal and specialist CPU-ranker studies also recorded no activation qualification. The chart reports both decision quality and warm latency.
Selected recorded studies. Each keeps its original conditions, date, interpretation, and limits.
Development measurements · 10 September 2026
Small resident helpers: accuracy and warm latency
The original study measured message intent, repair signals, and scoped context relevance on twenty authored cases each. Each profile ran twice. Qualification required at least eighteen correct decisions per function, every critical case, and a net gain over frozen lexical rules.
Four relevance cases per block were resolved by eligibility rules before inference. The plot includes those decisions in the relevance score. Warm latency excludes startup and warmups; the runs used a shared RTX 5070 host and a four-CPU container quota.
Original saved figure: initial profiles, examples added to prompts, and newer small profiles. Sixty authored development cases, repeated. No held-out qualification.Open full-size figure ↗Read the figure’s measurements as a table
Mean correct decisions across two repeats; latency is the slowest function p95
Profile / setting
Intent / 20
Repair / 20
Relevance / 20
Warm p95
4B Q4 / GPU
16
11
17
184 ms
0.6B Q4 / GPU
10
6
14
443 ms
0.6B dynamic Q4 / GPU
9
5
14
106 ms
0.6B Q8 / GPU
9
5
14
100 ms
1.7B dynamic Q4 / GPU
14
11
9
153 ms
0.6B Q4 / CPU
11
5
14
491 ms
4B Q4 / GPU / examples
15
12
16
143 ms
1.7B dynamic Q4 / GPU / examples
16
9
10
91 ms
0.6B Q4 / GPU / examples
14
7
14
84 ms
Qwen3.5 0.8B dynamic Q4 / GPU
13
11
14
167 ms
Qwen3.5 0.8B dynamic Q4 / CPU
13
11
14
856 ms
Frozen lexical rules
16
12
14
Not measured
Latency and accuracy led to different verdicts
Low latency did not make a helper useful at the required decision boundary. None of the plotted profiles qualified. Each profile was compared with the existing lexical rules.
Subsequent work and limits
Later studies narrowed the task to binary signals and examined a specialist CPU ranker. Their saved summaries also record no activation qualification. They are included separately in the download; they are not points on this plot or fresh held-out confirmation.
How source candidates are selected, launched, and rolled back.
What remains open
Demonstrate reliable correction-to-action continuity during substantial daily work, including interruption and recovery.
Qualify a useful local helper against frozen rules and fresh evaluation cases.
Evaluate newly authored source improvements with acceptance criteria that cover the behavior worth preserving.
Extend the resident conversation across interfaces, including live remote use through Saoirse, while preserving session state and a reliable way to revoke access.
