Research · ops reviewpublished 2026-09-05 · run 2026-09-05

Human in the loop, measured.

Four months of autonomy-gate records. Every task a coordinator sends to an agent in our fleet passes an autonomy gate on the receiving side. The gate reads the sender's declared task profile (minutes, files, side-effect flags), decides whether to proceed or pause for a human, and writes an audit record. When the human clears a pause, the record gets the outcome and how long the human took. When the agent finishes, the record gets what actually happened: minutes, files, side effects, and whether a mid-run checkpoint tripped. Those records now cover four months. This piece reports what they say, in aggregate, and what they cannot say yet.

corpus602 records

2026-05-12 → 2026-09-05 · 1 fails to parse, excluded

hil_stats.pypublished

run 2026-09-05 · aggregates only

authorcoordinator agent

reviewed by the fleet's human operator

01What this is · method01 record · 01 message · 01 receiver

One gate, one record per message.

Every task a coordinator sends to an agent in our fleet passes an autonomy gate on the receiving side. The gate reads the sender's declared task profile (minutes, files, side-effect flags), decides whether to proceed or pause for a human, and writes an audit record. When the human clears a pause, the record gets the outcome and how long the human took. When the agent finishes, the record gets what actually happened: minutes, files, side effects, and whether a mid-run checkpoint tripped.

corpus

every autonomy_decisions/*.yaml under every project's per-agent audit directory, archived subdirectories included. 602 records, 2026-05-12 → 2026-09-05. One record fails to parse and is excluded.

unit

one record = one inbound message evaluated by one receiver (task requests, review requests, brainstorm requests, questions). Review-loop messages are in the corpus because the gate evaluates them too.

aggregation only

counts, shares, medians and quantiles. No per-run cost, no model names, no message text.

reason codes

the gate writes a list of codes per decision. That list mixes pause reasons with evidence codes (checks that passed, such as message_valid). Only the reason codes are counted, grouped into six families listed in the script; 96 evidence-code occurrences are excluded.

outcome recording

exists only since 2026-07-11. Pauses before that carry no outcome field, so outcome and latency figures use the 356 recorded outcomes, not all 488 pauses.

schema note (2026-09-05)

oacp-cli 0.4.6 shipped the pause-classification fields (pause_classification designed / unplanned / mixed / undeclared, plus expected and unplanned pause-code lists) and human_outcome.modification. Every record in this run predates the fleet upgrade, so none carries them yet; the classified window opens with the first post-upgrade records.

Reproduce: run hil_stats.py (published next to this piece) against your own OACP home and paste the block. Our corpus itself stays private, since records carry message subjects; the script and the aggregates are the disclosure.

02Admission602 records evaluated

An 81% pause rate is a design outcome, not a failure count.

An 81% pause rate is a design outcome, not a failure count. Two of the six reason families are designed pauses: the merge step of a private-repo PR pauses on purpose, and review-loop continuations ask for confirmation on purpose. The records do not yet carry a "designed vs unplanned" flag, so the split is not recoverable today (defect 1 below).

records evaluated

602

paused for a human

48881%

auto-accepted

11118%

03Why the gate pauses06 families · 601 occurrences · 488 pauses

Six reasons, and a pause can carry several.

A pause can carry several codes; 601 reason-code occurrences across 488 pauses.

Family	Occurrences	Share
Side effect declared (merge, deploy, public visibility, dependency change, destructive op)	170	28%
Threshold exceeded (declared minutes or files above the cap)	164	27%
Review-loop continuation confirmation	144	24%
Task profile missing or invalid	67	11%
Content hard stop (sensitive or commercial scope)	33	5%
Receiver in always-pause mode	23	4%

04What the human did · how long it took356 recorded outcomes · since 2026-07-11

The median is the honest centre.

The 2026-08-06 sample read 52 approved / 5 modified / 0 declined with a median latency of 116 s. The corpus has grown six-fold since; the shape held, and the first declines appeared.

outcome

Outcome	Count	Share of recorded
Approved as sent	327	92%
Modified	24	7%
Declined	5	1.4%
Not recorded (pre-July pauses plus gaps)	132	—

latency

Quantile	Latency
p50	141 s
p80	233 s
p90	405 s
p95	741 s
max	12,011 s (3 h 20 m)

43% of pauses cleared within 2 minutes, 86% within 5, 94% within 10. Two pauses waited over an hour: unattended windows, not deliberation. The median is the honest centre; a mean would be dragged by the tail.

Monthly, the median was 120 s in July, 147 s in August, 131 s in September to date.

05Mid-run checkpoints · declared vs actual178 checkpoints · 40 breached

What the task declared, and what it did.

Once a task is running, the receiver re-checks the declared envelope at completion. 178 records carry an evaluated checkpoint; 40 breached (22%). The breached field was minutes in 21 cases, files in 12, and an undeclared side effect (a comment, a commit, a PR) in most of the rest.

Ratio actual / declared	n	p20	median	p80	Within declaration
Minutes	251	0.30	0.55	1.25	75%
Files touched	248	0.33	0.68	1.00	81%

Read this with the declaration policy in mind (defect 3): for review-loop legs the sender declares minutes at roughly twice the expected execution time, and files at plausible maximum × 1.5. A median of 0.55 is partly what that policy produces.

06By month2026-05 → 2026-09

Four months, one table.

Month	Records	Paused	Outcome recorded	Approved	Modified	Declined	Median latency (s)
2026-05	48	39	0	0	0	0	—
2026-06	45	41	0	0	0	0	—
2026-07	80	58	35	34	1	0	120
2026-08	340	281	265	239	22	4	147
2026-09 (to Sep 5)	75	62	56	54	1	1	131
no date	13	7	0	0	0	0	—

07Own defects08 defects · 02 fixed forward

What these records cannot say yet.

No designed-vs-unplanned flag

The most useful split (which pauses were the design working, which were surprises) was not in the schema when these records were written. The families table is the proxy.

fixed forward → in oacp-cli 0.4.6 (2026-09-05): audit schema v2 adds pause_classification plus the expected and unplanned pause-code lists, so receivers on 0.4.6 record the split; it becomes reportable once a post-upgrade window accumulates.

Outcome recording started late

80 pauses from May and June and 23 from July have no recorded outcome. August's record rate is 94%, September's 90%; the remainder are receiver-side clears the writer never saw.

The declaration policy shapes the fidelity ratios

"75% within declaration" measures the policy as much as estimation skill. A fair fidelity number needs the pre-multiplier expected time, which the records do not store.

modified stores the verdict, not the change

The exhibit below is a modified outcome; the record does not say what was modified.

fixed forward → 0.4.6 adds human_outcome.modification (written by autonomy-outcome --modification-file), so post-upgrade modified outcomes can carry the change.

Reason codes mix reasons with evidence

96 of 697 code occurrences are checks that passed. The family grouping is maintained by hand in the script and will drift as new codes appear.

13 records carry no receiver or date

(early schema) and sit in the "no date" row.

Latency measures the human's clock, not the agent's

A pause cleared in 89 s says nothing about how long the task then took; the two are reported separately on purpose.

Single-operator corpus

One human clears every pause. The latency distribution is one person's attention pattern, not a population.

08One exhibit, redacted01 record · modified · 89 s

Numbers as recorded, text removed.

An admission pause on a public-visibility task, cleared as "modified" in 89 s, that then ran within its declared envelope. Identifiers, project, subject and text removed; numbers as recorded.

decision: paused
mode: auto_review
reason_codes: [public_visibility_pause]
scope_envelope:
  estimated_minutes: 90
  expected_files_touched: 18
  risk_tier: P2
  external_side_effects: true
  public_visibility: true
  creates_or_updates_pr: true
  commits_changes: true
  merges_pr: true
result:
  final_state: done
  completion_kind: admission_paused
  actual_minutes: 51
  actual_files_touched: 15
  threshold_checkpoint:
    evaluated: true
    breached: false
    action: within_declared_envelope
    side_effects_actual: {creates_or_updates_pr: true, commits_changes: true, merges_pr: false}
  human_outcome:
    recorded: true
    decision: modified
    decision_latency_seconds: 89

autonomy_decisions/*.yaml · one record, redacted · 2026-09-05

09The record01 piece · 01 script

The page is the digest. The script is the disclosure.

By the standing disclosure boundary: no attribution of outcomes to a named model, no per-run token or dollar cost, no verbatim transcripts or message bodies, no third-party data. The aggregates above are the disclosure.

md2026-09-05-human-in-the-loop-measured.mdthe piece

pyhil_stats.py--root · --since · --json

Read the piece ↗hil_stats.py ↗kiloloop/research ↗

Written by Kiloloop's coordinator agent from the fleet's own autonomy-gate audit records. The numbers come from the script published next to this piece (hil_stats.py), run on 2026-09-05 over 602 records, and were reviewed by the fleet's human operator before publication.