Skip to content

Audit stream, region gate, ledger, ship gate - #2

Open
SuperInstance wants to merge 19 commits into
mainfrom
architecture
Open

SuperInstance wants to merge 19 commits into
mainfrom
architecture

Conversation

@SuperInstance

Copy link
Copy Markdown
Owner

Follows #1 (merged). Three more build-order pieces from ARCHITECTURE.md, each tested:

  • Audit stream (audit.py): an auditor commits to a secret seed per period, selects judged items by a keyed hash of the content key alone (so suppressed verdicts are sampled like any other and no judge can predict its audits), drops each as a blind inbox task, and reveals the seed together with the log tips at that moment. verify recomputes the selection and reports skipped and cherry-picked items. Fault drills show reviewers a verdict known to be wrong, in a task that looks like any other review; drills reports each reviewer's catch rate after the reveal.
  • Region gate and ledger (gate.py, ledger.py): act only on a conformal singleton (one threshold per region and verdict) whose audited Clopper–Pearson bound clears the accepted error rate in every region; unaudited regions never act. Track records score each judge only on resolutions it didn't produce (world outcomes, then blind labels), weighted by inverse selection propensity. Includes each labeller's weekly Brier against world outcomes (the deskilling alarm).
  • Ship gate (shipgate.py): churn per region (blocks a release that breaks a region even when its net change is positive), verdict diff before any labels, a polarization monitor for shared blind spots, and drift split into world change and model change.

Tests: test-audit.sh runs the whole loop against the real hook (judge → blind audit selection → human labels → ledger → gate → drills), 28 checks; 35 unit tests; CI runs everything. The gate's bound reproduces the audit sample-size table cited in the research (60 clean items: 4.9%; 150: 2.0%; 60 with one error: 7.7%).

One design fix the end-to-end test forced: the gate's conformal threshold is now calibrated per region and per verdict. With one threshold per region, a judge unreliable on its −1 verdicts blurred its reliable +1 verdicts into "unsure" too.

🤖 Generated with Claude Code

https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo


Generated by Claude Code

claude added 19 commits October 9, 2026 07:29
audit.py lets an auditor body commit to a secret seed per period,
select judged items by a keyed hash of the content key alone (so
suppressed items are sampled like any other and no judge can predict
its audits), drop each as a blind inbox task, and reveal the seed with
the log tips at that moment. verify recomputes the selection from the
logs as they stood at the reveal and reports skipped and cherry-picked
items.

jlog.iter_log can now read logs up to recorded tips; timestamps are
normalised to UTC.

test-audit.sh: 17 checks against the real hook.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
gate.py acts only on a conformal singleton (one threshold per region
and per verdict) whose audited Clopper-Pearson error bound clears the
accepted rate in every region the item falls in; unaudited regions
never act. Exploration is limited to reversible actions, replayable,
and logs its propensity. The bound reproduces the audit sample-size
table (60 clean items: 4.9%; 150: 2.0%; 60 with one error: 7.7%).

ledger.py scores every judge only on resolutions it did not produce
(world outcomes first, then blind labeller audits), weighted by
inverse selection propensity, and feeds the gate its audited counts
and conformal scores.

test-audit.sh now runs the whole loop against the real hook: judge,
blind audit selection, human labels, ledger, gate. 26 unit tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
shipgate.py judges a new judge by region, from the logs: churn
(blocks a release that breaks a region even when its net change is
positive), verdict diff before any labels, a polarization monitor
comparing joint errors with independence, and drift split into world
change (old judge on new content) and model change.

audit.py drill drops review tasks that show a verdict known to be
wrong, indistinguishable from other reviews; the drill list is revealed
with the period, and `audit.py drills` reports each reviewer's catch
rate. ledger.py adds each labeller's weekly Brier against world
outcomes.

test-audit.sh: 28 checks against the real hook. 35 unit tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
gatekeep.py applies the gate to one judgment and records it with
sel, prop, the decision, the item's regions and a gate-config hash:
act -> stream, explore -> explore with its propensity, escalate ->
shadow plus a review task showing the verdict, and a small logged
bypass slice that skips the judge entirely. The ledger now prefers
regions recorded on the line.

Review tasks for drills and real escalations now have identical
names and wording, so a reviewer cannot tell them apart.

test-audit.sh: 33 checks end to end. 40 unit tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
`jlog.py sync --body ID FILE` appends only the lines of a local log
that the body's log ref does not hold yet, so a cron job can move
~/judgment-log.tsv into git without duplicates. The ledger resolves a
v0 line's judge name through the matching manifest's `name:`.

test-log.sh: 25 checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
forecast.py lets an agent register, for each file it makes, the
question that will resolve it (bodies/<agent>/forecasts/<name>.md) and
its own probability; only a judge with role: world may log the outcome.
The ledger then scores agents exactly like judges.

test-forecast.sh: 7 checks end to end, in CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
shipgate.py canary compares decisions logged under two gate
configurations per region and blocks where the new one acts on
clearly more wrong verdicts. independence measures, on third-party
resolved items, how often a candidate labeller is wrong together with
a judge against what independence predicts. Reports name the judge's
consumers from soul/consumers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
window.py compiles an agent's window from the judgment-log refs and
the ledger: charter, task, what the judges see (conflict, disagreement),
open questions (including active questions never asked of the task),
precedents linked by exact blob with failures first, recent activity,
and what the budget cut. The main commit and every log tip are pinned
in the header; given those pins, any body reproduces the window byte
for byte.

test-window.sh: 13 checks, in CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
agent-exec.sh compiles the task's window, pipes it to AGENT_CMD, and
saves it as window.md beside the results. tick.sh records the window's
blob hash as a Window: trailer on the done commit, so what the agent
knew when it acted is named by the commit itself.

test.sh: 61 checks under dash and bash. Stop tracking Python bytecode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
Publishing now builds only the new commits' cells and merges them into
the published history views, touching only the subtrees they land
in. At 60,000 commits one push costs 2.7 s instead of 23 s, with an
identical result. verify still rebuilds from nothing.

test-pp-inc.sh compares extension with a full rebuild after every
push of a random history, including an axes change midway. It caught
awk comparing the name "00" numerically with an empty variable, which
dropped the whole 00 bucket; name comparisons are now strings, with a
regression test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
OPERATIONS.md walks through standing a fleet up: keys, the bare
remote, genesis, hooks, admitting bodies, work, views, judgments,
audits, schedules. test-ops.sh runs every "# [machine]" block in it,
in order, each machine with its own HOME, so the guide cannot drift.

Running it found two real gaps, both fixed:
- git keeps no empty directories, so the first task needs
  `mkdir -p inbox` (the guide now does);
- right after a body pushed, its own commit was not yet on its
  verified ref, so `project.sh verify` refused views built from it.
  The tick now advances refs/verified/main after its own push, and
  verify follows main through signatures itself; a fresh clone trusts
  only genesis instead of whatever origin/main says.

test-pp.sh: 44 checks, including views built from an unsigned commit
being refused by a body and by a fresh clone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
gatekeep.py --batch reads the ledger once, decides every judgment in
the file, appends them to the log in one batch and opens one review
per escalation: the way a tick should call it.

The ledger kept the last line it read for each (judge, subject,
question), which depends on body order; it now keeps the newest by
timestamp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
Scoring walks only resolved keys, ingest skips re-validating stored lines,
and jlog.iter_log takes a since bound. At 1M lines the ledger builds in
6.8 s (was 12.5 s) and gate records plus thresholds take 0.3 s (was 12.6 s).
A pickle cache of the built ledger was measured and removed: 267 MB and
7.3 s to load, slower than rebuilding. ARCHITECTURE records the numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
The window now shows the newest lines from labeller, world and appeal judges
verbatim, marked when they touch the task's zone, so an agent can tell what a
person or the world said from what the judges inferred. Still a pure function
of the pins; test-window.sh grows to 16 checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
Two new sections run under test-ops.sh: the Oracle gates a batch from an
unaudited judge and gets a review task in inbox/, and the laptop's agent
forecasts its own result, which the owner's world judge resolves. Genesis now
publishes the student and agent manifests, since v2 lines name judges by
manifest hash; troubleshooting explains the hook's refusal otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
Whatever the agent writes to `judgments` during a task goes through
gatekeep.py as one batch; decisions land on the body's log and gate.jsonl is
kept with the result. OPERATIONS.md runs it inside a real tick (30 checks).
ARCHITECTURE's build table now shows rows 5, 9 and 10 as they stand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
consolidate.py compiles bodies/<body>/memory.md from the record: tasks
finished (failures first, with their last log line), forecasts and the
world's resolutions with the Brier score, gate decisions, and corrections.
The page carries its pins and `verify` recomputes it, so a body cannot
remember itself more kindly than the record allows. The window compiler
shows it right after the charter. New suite test-consolidate.sh (15 checks)
runs in CI; OPERATIONS.md publishes and verifies one (33 checks).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
One CI job on db7255d failed in test-audit.sh while the same commit passed
in the other seven jobs. p1 drew a fresh random seed on every run, so the
statistical checks (selected count, share of negative verdicts, judge error
of 0.3-0.7) each had a small chance of landing outside their ranges; the
error-rate check alone fails about 0.3-0.5% of runs. audit.py commit now
takes --seed (64 hex digits, documented as for reproducible tests), and the
test fixes p1's seed to one whose numbers sit near the expectation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
@SuperInstance

Copy link
Copy Markdown
Owner Author

Quick notes from a cold PR sweep (Mavis, on behalf of SuperInstance):

This PR is in a mergeable state. Base = main @ 9157c65, head = architecture @ 7fb61f4c, mergeable=true, all 9 check-runs green (4×2 matrix suites bash/dash × mawk/gawk + GitGuardian). CI was last run 2026-10-09 16:26 UTC and matches the architecture PR body.

What it adds, at a glance. 30 files, +3111/−57, six new modules plus shell tests, gates, and ledger. The big files:

  • audit.py (362 lines) — keyed-hash blind audit, verify + drills
  • gate.py (144) + ledger.py (221) — per-region per-verdict conformal singleton, Brier ledger, weekly deskilling alarm
  • shipgate.py (268) — per-region churn guard, verdict diff, polarization monitor, drift = world + model
  • gatekeep.py (131) + forecast.py (100) + consolidate.py (183) + agent-exec.sh (24) + tick.sh (12) + jlog.py (44/−11)
  • test-audit.sh (191) + 5 sibling shell tests — exercises the whole hook loop (judge → blind audit selection → human labels → ledger → gate → drills), 28 checks + 35 unit tests
  • README (6) + ARCHITECTURE (92/−30) + new OPERATIONS (287) + a .gitignore (2) and a test.yml (12)

One thing worth flagging before merge. The PR body mentions that the gate's conformal threshold is now calibrated per region and per verdict, and the body reports 4.9% / 2.0% / 7.7% for the 60 / 150 / 60+1 sample sizes. That calibration table is the part of the design that, if wrong, lets the gate ship garbage with 95% confidence. If you can paste the table from the conformance test (or the row it asserts against) into the PR thread, reviewers can match it against the research note research/judges-that-never-grade-themselves.md.

No blockers from my side. If the calibration table check above is fine, the PR looks ready. Want me to merge it?

🤖 Note posted on behalf of SuperInstance, via Mavis.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants