Repository navigation
Audit stream, region gate, ledger, ship gate - #2
SuperInstance wants to merge 19 commits into
Conversation
audit.py lets an auditor body commit to a secret seed per period, select judged items by a keyed hash of the content key alone (so suppressed items are sampled like any other and no judge can predict its audits), drop each as a blind inbox task, and reveal the seed with the log tips at that moment. verify recomputes the selection from the logs as they stood at the reveal and reports skipped and cherry-picked items. jlog.iter_log can now read logs up to recorded tips; timestamps are normalised to UTC. test-audit.sh: 17 checks against the real hook. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
gate.py acts only on a conformal singleton (one threshold per region and per verdict) whose audited Clopper-Pearson error bound clears the accepted rate in every region the item falls in; unaudited regions never act. Exploration is limited to reversible actions, replayable, and logs its propensity. The bound reproduces the audit sample-size table (60 clean items: 4.9%; 150: 2.0%; 60 with one error: 7.7%). ledger.py scores every judge only on resolutions it did not produce (world outcomes first, then blind labeller audits), weighted by inverse selection propensity, and feeds the gate its audited counts and conformal scores. test-audit.sh now runs the whole loop against the real hook: judge, blind audit selection, human labels, ledger, gate. 26 unit tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
shipgate.py judges a new judge by region, from the logs: churn (blocks a release that breaks a region even when its net change is positive), verdict diff before any labels, a polarization monitor comparing joint errors with independence, and drift split into world change (old judge on new content) and model change. audit.py drill drops review tasks that show a verdict known to be wrong, indistinguishable from other reviews; the drill list is revealed with the period, and `audit.py drills` reports each reviewer's catch rate. ledger.py adds each labeller's weekly Brier against world outcomes. test-audit.sh: 28 checks against the real hook. 35 unit tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
gatekeep.py applies the gate to one judgment and records it with sel, prop, the decision, the item's regions and a gate-config hash: act -> stream, explore -> explore with its propensity, escalate -> shadow plus a review task showing the verdict, and a small logged bypass slice that skips the judge entirely. The ledger now prefers regions recorded on the line. Review tasks for drills and real escalations now have identical names and wording, so a reviewer cannot tell them apart. test-audit.sh: 33 checks end to end. 40 unit tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
`jlog.py sync --body ID FILE` appends only the lines of a local log that the body's log ref does not hold yet, so a cron job can move ~/judgment-log.tsv into git without duplicates. The ledger resolves a v0 line's judge name through the matching manifest's `name:`. test-log.sh: 25 checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
forecast.py lets an agent register, for each file it makes, the question that will resolve it (bodies/<agent>/forecasts/<name>.md) and its own probability; only a judge with role: world may log the outcome. The ledger then scores agents exactly like judges. test-forecast.sh: 7 checks end to end, in CI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
shipgate.py canary compares decisions logged under two gate configurations per region and blocks where the new one acts on clearly more wrong verdicts. independence measures, on third-party resolved items, how often a candidate labeller is wrong together with a judge against what independence predicts. Reports name the judge's consumers from soul/consumers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
window.py compiles an agent's window from the judgment-log refs and the ledger: charter, task, what the judges see (conflict, disagreement), open questions (including active questions never asked of the task), precedents linked by exact blob with failures first, recent activity, and what the budget cut. The main commit and every log tip are pinned in the header; given those pins, any body reproduces the window byte for byte. test-window.sh: 13 checks, in CI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
agent-exec.sh compiles the task's window, pipes it to AGENT_CMD, and saves it as window.md beside the results. tick.sh records the window's blob hash as a Window: trailer on the done commit, so what the agent knew when it acted is named by the commit itself. test.sh: 61 checks under dash and bash. Stop tracking Python bytecode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
Publishing now builds only the new commits' cells and merges them into the published history views, touching only the subtrees they land in. At 60,000 commits one push costs 2.7 s instead of 23 s, with an identical result. verify still rebuilds from nothing. test-pp-inc.sh compares extension with a full rebuild after every push of a random history, including an axes change midway. It caught awk comparing the name "00" numerically with an empty variable, which dropped the whole 00 bucket; name comparisons are now strings, with a regression test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
OPERATIONS.md walks through standing a fleet up: keys, the bare remote, genesis, hooks, admitting bodies, work, views, judgments, audits, schedules. test-ops.sh runs every "# [machine]" block in it, in order, each machine with its own HOME, so the guide cannot drift. Running it found two real gaps, both fixed: - git keeps no empty directories, so the first task needs `mkdir -p inbox` (the guide now does); - right after a body pushed, its own commit was not yet on its verified ref, so `project.sh verify` refused views built from it. The tick now advances refs/verified/main after its own push, and verify follows main through signatures itself; a fresh clone trusts only genesis instead of whatever origin/main says. test-pp.sh: 44 checks, including views built from an unsigned commit being refused by a body and by a fresh clone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
gatekeep.py --batch reads the ledger once, decides every judgment in the file, appends them to the log in one batch and opens one review per escalation: the way a tick should call it. The ledger kept the last line it read for each (judge, subject, question), which depends on body order; it now keeps the newest by timestamp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
Scoring walks only resolved keys, ingest skips re-validating stored lines, and jlog.iter_log takes a since bound. At 1M lines the ledger builds in 6.8 s (was 12.5 s) and gate records plus thresholds take 0.3 s (was 12.6 s). A pickle cache of the built ledger was measured and removed: 267 MB and 7.3 s to load, slower than rebuilding. ARCHITECTURE records the numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
The window now shows the newest lines from labeller, world and appeal judges verbatim, marked when they touch the task's zone, so an agent can tell what a person or the world said from what the judges inferred. Still a pure function of the pins; test-window.sh grows to 16 checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
Two new sections run under test-ops.sh: the Oracle gates a batch from an unaudited judge and gets a review task in inbox/, and the laptop's agent forecasts its own result, which the owner's world judge resolves. Genesis now publishes the student and agent manifests, since v2 lines name judges by manifest hash; troubleshooting explains the hook's refusal otherwise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
Whatever the agent writes to `judgments` during a task goes through gatekeep.py as one batch; decisions land on the body's log and gate.jsonl is kept with the result. OPERATIONS.md runs it inside a real tick (30 checks). ARCHITECTURE's build table now shows rows 5, 9 and 10 as they stand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
consolidate.py compiles bodies/<body>/memory.md from the record: tasks finished (failures first, with their last log line), forecasts and the world's resolutions with the Brier score, gate decisions, and corrections. The page carries its pins and `verify` recomputes it, so a body cannot remember itself more kindly than the record allows. The window compiler shows it right after the charter. New suite test-consolidate.sh (15 checks) runs in CI; OPERATIONS.md publishes and verifies one (33 checks). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
One CI job on db7255d failed in test-audit.sh while the same commit passed in the other seven jobs. p1 drew a fresh random seed on every run, so the statistical checks (selected count, share of negative verdicts, judge error of 0.3-0.7) each had a small chance of landing outside their ranges; the error-rate check alone fails about 0.3-0.5% of runs. audit.py commit now takes --seed (64 hex digits, documented as for reproducible tests), and the test fixes p1's seed to one whose numbers sit near the expectation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
|
Quick notes from a cold PR sweep (Mavis, on behalf of SuperInstance): This PR is in a mergeable state. Base = What it adds, at a glance. 30 files, +3111/−57, six new modules plus shell tests, gates, and ledger. The big files:
One thing worth flagging before merge. The PR body mentions that the gate's conformal threshold is now calibrated per region and per verdict, and the body reports No blockers from my side. If the calibration table check above is fine, the PR looks ready. Want me to merge it? 🤖 Note posted on behalf of SuperInstance, via Mavis. |
Follows #1 (merged). Three more build-order pieces from
ARCHITECTURE.md, each tested:audit.py): an auditor commits to a secret seed per period, selects judged items by a keyed hash of the content key alone (so suppressed verdicts are sampled like any other and no judge can predict its audits), drops each as a blind inbox task, and reveals the seed together with the log tips at that moment.verifyrecomputes the selection and reports skipped and cherry-picked items. Fault drills show reviewers a verdict known to be wrong, in a task that looks like any other review;drillsreports each reviewer's catch rate after the reveal.gate.py,ledger.py): act only on a conformal singleton (one threshold per region and verdict) whose audited Clopper–Pearson bound clears the accepted error rate in every region; unaudited regions never act. Track records score each judge only on resolutions it didn't produce (world outcomes, then blind labels), weighted by inverse selection propensity. Includes each labeller's weekly Brier against world outcomes (the deskilling alarm).shipgate.py): churn per region (blocks a release that breaks a region even when its net change is positive), verdict diff before any labels, a polarization monitor for shared blind spots, and drift split into world change and model change.Tests:
test-audit.shruns the whole loop against the real hook (judge → blind audit selection → human labels → ledger → gate → drills), 28 checks; 35 unit tests; CI runs everything. The gate's bound reproduces the audit sample-size table cited in the research (60 clean items: 4.9%; 150: 2.0%; 60 with one error: 7.7%).One design fix the end-to-end test forced: the gate's conformal threshold is now calibrated per region and per verdict. With one threshold per region, a judge unreliable on its −1 verdicts blurred its reliable +1 verdicts into "unsure" too.
🤖 Generated with Claude Code
https://claude.ai/code/session_01D6xSPWYZQwxtf5YhmoQdFo
Generated by Claude Code