Repository navigation
perf(beacon): cache message hashes, group batches by message and batch gossip checks, as Silver does - #771
Draft
MegaRedHand wants to merge 13 commits into
Draft
perf(beacon): cache message hashes, group batches by message and batch gossip checks, as Silver does#771MegaRedHand wants to merge 13 commits into
MegaRedHand wants to merge 13 commits into
Conversation
A place for validating a loaded registry's keys in one pass; the first version just runs KeyValidate per key.
…tate load Silver decodes its entire registry when a state is loaded, so the first block after a start does not validate tens of thousands of keys inline. warm_pubkeys now validates uncached keys with rayon and caches them by registry index; the beacon actor runs it on the head state before it starts, and the import replay runs it on the anchor state.
…sage hash_to_G2 costs about 100 us and every attester of a slot signs the same root, so blst's one-shot verifiers re-hashed it on every check. Single checks now take H(m) from a two-generation cache (Silver's design, cap 256) and run the pairing over it themselves; verify_signature_sets sorts sets by message, sums r_i * pk_i per message with one multi-scalar multiplication, and pays one Miller loop per distinct message plus one final exponentiation. Soundness is kept explicit now that blst's verify no longer refuses for us: the pairing check rejects an identity key, and a batch whose weighted signature sum is the point at infinity fails. The raw blst calls (hash to G2, pairing accumulation, weighted sums) live in bls/primitives.rs.
…columns in batches Every stateful gossip check ran as its own blocking task, so each message paid a pairing's final exponentiation (or a KZG call's fixed work) alone, even though a slot delivers them in bursts. The rules now split into a prepare step, which runs every non-signature rule and returns the signature (or KZG) checks as deferred steps, and a resolve step that walks them in the original order. A queue per kind lets one draining blocking task take everything waiting, verify all its signatures in a single verify_signature_sets call (one cell-proof batch across sidecars for columns), and fall back to per-message checks only when the batch fails, so a forged message costs its batch one extra pass and nothing else changes hands. The public stateful_checks/check_message functions are prepare plus an immediate resolve, so the Beacon API and the spec vectors see the same verdicts as before. Queue overflow and a missing drainer permit both answer Ignore(Overloaded), as an exhausted permit pool did.
One drainer per kind is what makes a batch, but it ran every message's preparation (signature decode, subgroup checks, committee lookups) and, after a failed batch, every per-message fallback check on a single thread. Before batching those ran in parallel, one blocking task per permit, so a burst of attestations or one forged message in a large batch cost seconds of serial work. Preparation, the resolve pass and the per-sidecar KZG fallback now run with rayon, keeping output order and a panic guard around each item. The single combined signature and KZG calls are untouched; they already parallelise internally.
…task The office-4 replay showed the grouped batch 0.5 ms slower per block than the blst batch it replaced. A block's sets mostly sign distinct messages, which replay never finds in the hash cache, and the batch put up to four messages' hashes, and four Miller loops, on one thread each, where blst's batch ran one set per worker. A hash or a Miller loop is worth a task on its own.
This was referenced Nov 3, 2026
## 🗒️ Description / Motivation
`ethlambda validator` kept no slashing-protection record. Its two
in-memory guards (`AttestationGuard`, `ProposalGuard`) applied
EIP-3076's minimal rule within one run, but a restart emptied them and
nothing stopped a second process from signing with the same keys. The
client warned about this on every start, and `docs/spec_deviations.md`
listed it as a known gap. That gap is the blocker for anyone running
ethlambda with real stake.
This PR makes slashing protection durable. It is the first of two PRs;
the second adds EIP-3076 interchange import/export (CLI and keymanager
API).
**Design.** Before choosing, we surveyed Lighthouse, Prysm, Teku,
Nimbus, Lodestar and Grandine.
- **Teku's model:** one record per validator holding EIP-3076 *minimal*
watermarks (highest block slot, highest attestation source and target).
It persists the rule our guards already enforced.
- **Why not full history:** the full-history clients end up enforcing
almost the same thing. Lighthouse prunes to one epoch and minifies every
import. What full history buys is allowing a byte-identical re-sign
after a crash, and it costs a new C dependency (SQLite), pruning and
migrations.
## What Changed
- **New `crates/validator/src/slashing_protection/`**:
- **`mod.rs`, `SlashingProtection`:**
- the rule: an attestation needs `target > last target` and `source >=
last source`; a block needs `slot > last slot`;
- "never signed" is `None`, so slot 0 and epoch 0 are signable exactly
once;
- `open` and the batch APIs;
- every change is written to disk first and taken into memory only once
the write succeeds.
- **`file.rs`, the on-disk record:**
- `<validators-dir>/slashing_protection.json` is an EIP-3076 v5 minimal
interchange naming the chain's `genesis_validators_root`;
- strict reading that fails closed;
- durable writes: temp file, `fsync`, `rename`, directory `fsync`;
- an exclusive `File::try_lock` on `slashing_protection.lock`.
- **Signing path:**
- **`attestation.rs`:** one slot's batch is checked and recorded in a
single write, inside `spawn_blocking` and before the store's read lock
is taken. Only then are the allowed attestations signed.
- **`proposal.rs`:** the cheap pre-check stays before `produce_block`.
The recording check moves before signing and persists.
- **Write failure:** a failed write signs nothing.
- **Startup (`lib.rs`):**
- the record is opened right after `genesis()`, which supplies the root
it must match;
- a key with no record is refused unless `--init-slashing-protection` is
passed;
- the unconditional "keeps NO slashing-protection record" warning is
replaced by an info line.
- **Keymanager:** `POST /eth/v1/keystores` registers each imported key
before writing it to disk or loading it.
- **Removed:** `attestation_guard.rs` and `proposal_guard.rs`. Their
rule and tests moved into `slashing_protection`.
- **New metrics:**
`ethlambda_validator_slashing_protection_write_seconds` and
`ethlambda_validator_slashing_protection_write_failures_total`.
- **Docs:**
- `docs/spec_deviations.md`: the section is rewritten to describe the
minimal-watermark deviation.
- `docs/cli.md` and `README.md`: the new flag and the
slashing-protection callout.
- `docs/metrics.md` and `CLAUDE.md`: updated to match.
## Correctness / Behavior Guarantees
- **Ordering:** check, then a durable record, then the signature. A
signature that exists is always one the record already accounts for,
whether the process then crashes, hits its duty deadline, or is rejected
by the beacon node.
- **Fail closed:** startup stops on any of these, never reading them as
"nothing signed":
- an empty, malformed or wrong-version file;
- a validator listed twice;
- a record for another `genesis_validators_root`;
- a key with no record.
At signing time, an unregistered key or a failed write refuses the
message.
- **One process:** a second validator client on the same
`--validators-dir` refuses to start while the first holds the lock.
- **Behavior change for operators:** the first start after upgrading
refuses with a message naming `--init-slashing-protection`. That flag
registers each key as never having signed. It is meant only for keys
that have signed nothing anywhere for at least two epochs, and it never
lowers an existing record; the migration window is left to doppelganger
protection, in a later PR.
- **Behavior change for deployments:** `--validators-dir` must now be
writable, for the record and its lock. A read-only mount of the keys (a
Kubernetes Secret, a `:ro` volume) fails at startup.
- **Known cost:** a byte-identical re-sign after a crash is refused, as
in Teku. Documented in `spec_deviations.md`.
- **Not in this PR:** carrying history in or out. Keymanager `DELETE`
still returns the empty, all-zero-root interchange, and `POST` still
ignores `slashing_protection`. Both are PR 2.
## Tests Added / Run
- **`slashing_protection` unit tests:**
- every rule case ported from the guards;
- reopening restores the record;
- each corrupt-file shape fails `open`;
- a validator listed twice fails `open`;
- another chain is refused;
- the lock refuses a second open and is released on drop;
- an unknown key is refused without `init` and registered with it;
- `init` never lowers a record;
- a failed write (read-only dir) changes nothing;
- the file is a valid EIP-3076 v5 minimal interchange;
- a key twice in one batch is refused the second time.
- **Service tests (`MockBeaconNode`):**
- `a_restart_does_not_forget_what_was_attested`;
- `a_restart_does_not_forget_what_was_proposed` (no block is even
requested);
- `a_record_that_cannot_be_written_signs_nothing`.
- **Keymanager:**
`an_imported_key_is_registered_for_slashing_protection`, where the next
start opens without `--init-slashing-protection`.
- **Commands run:**
- `make fmt`: clean
- `make lint`: clean
- `make test`: 2878 passed, 0 failed (after the review follow-ups); the
validator crate alone is at 344 after the bot follow-ups
## Review follow-ups (2808b51)
| Review point | Change |
|---|---|
| Kurtosis battery would fail on `main` | `vc_extra_params:
["--init-slashing-protection"]` on both ethlambda validator clients. The
**Assertoor** check on this PR fails for the same chicken-and-egg
reason: PR runs test the published `unstable` image, which does not know
the flag yet. The post-merge run on `main` tests the merge commit's
image, which does |
| Keys registered without history start as "never signed" | Kept, by
design. Seeding the current slot and epoch was added after the first
review and dropped after the second: it only covers messages signed
elsewhere up to this node's clock, costs the rest of the epoch's duties
on every registration, and would differ from the interchange import's
empty entries. The migration window is doppelganger protection's, in a
later PR |
| `path.exists()` fails open | Only `NotFound` counts as absent. A
symlink, directory or unreadable path is refused instead of replaced |
| A failed import leaves a record behind | Keys are registered in one
write, after they are persisted and before any is loaded |
| Mutex taken on the executor | Every async access (attestations, the
proposal pre-check and record, the keymanager) goes through one
`slashing_protection::blocking` helper |
| Read-only `--validators-dir` | Documented in `docs/cli.md` |
| Refusal counter undercounts | A failed write counts every refused
attestation |
| Permission tests fail as root | Skipped when `geteuid() == 0` |
| Lock described as per key | The docs now say it is per directory |
| Stale "guard" wording | Replaced, including `ProposalRefused` and
`is_retryable` |
| Startup error pointed at an import that does not exist yet | Reworded
to the wait condition (import arrives in #736) |
| Cleanups | The durable write moved to
`secure_fs::write_private_durably` and is shared with
`validator_definitions.yml`, which gains its missing fsyncs. No clone of
the record per write, compact JSON, one write per keymanager import |
## Bot review follow-ups (696e2a8)
| Review point | Change |
|---|---|
| macOS: directory `fsync` is not `F_FULLFSYNC` (Kimi 4) | The directory
flush after the rename calls `File::sync_all` first, `F_FULLFSYNC` on
macOS like the file's own flush, and falls back to a plain `fsync` only
where a filesystem refuses that for a directory. The original error is
kept if both fail |
| File mode and the deterministic `.tmp` path (Kimi 5, 11) | The umask
itself was not a problem (it can only clear bits from `0600`), but a
leftover `.tmp` from a crashed write kept its old mode, since `mode()`
applies only at creation. The temporary is now removed first and created
with `create_new`, so the record never inherits a wider mode and a
symlink planted at the `.tmp` path is never written through. Two tests
pin both |
| Poison vs. disk failure, string errors (Kimi 1, 9) |
`slashing_protection::blocking` returns the crate's `Error` instead of a
`String`. A poisoned lock, or a panic inside the operation, is the new
`SlashingProtectionUnusable`, whose message says the client must be
restarted; a failed write stays `Io`. Never retryable |
| Duplicate-key wording (Kimi 2) | The doc now says a key's later
position is judged against its earlier one's verdict |
| `attest` answers `Ok` with zero published on a record failure (Codex)
| `Attested` gains a `refused` count, so a slot where protection
withheld every signature, or could not record them, is distinguishable
from a slot with nothing to publish. The refusal tests assert it |
| Keymanager ignores imported history (Codex) | That is #736 |
| Already done in 2808b51 (Kimi 3, 12) | No clone per write; compact
JSON |
## Related Issues / PRs
- Follow-up: EIP-3076 interchange import/export (CLI and keymanager
API), with the `eth-clients/slashing-protection-interchange-tests`
v5.3.0 vectors.
- Follow-up: doppelganger protection.
## ✅ Verification Checklist
- [x] Ran `make fmt` — clean
- [x] Ran `make lint` (clippy with `-D warnings`) — clean
- [x] Ran `make test` (`test-consensus` plus `test-node`, at
`release-fast`) — all passing
Stacked on #734, rebased onto its review follow-ups. ## 🗒️ Description / Motivation #734 made slashing protection durable, but history could not move in or out. A staker migrating keys to or from ethlambda had no safe path: a key arriving from another client started as "never signed", and a key leaving got only an empty interchange from `DELETE /eth/v1/keystores`. EIP-3076's interchange format exists for exactly this, and every other client supports it. This PR adds EIP-3076 version 5 import and export, offline through a new `ethlambda slashing-protection` command and online through the keymanager API. It also runs the shared `eth-clients/slashing-protection-interchange-tests` vectors. ## What Changed - **`slashing_protection` (validator crate):** - `import`: parses a complete or minimal interchange and checks the version and `genesis_validators_root`. It then reduces each validator to its highest slot, source and target, and merges the result with the existing record by maximum, in one durable write. - Merging by maximum never lowers a watermark. - Slashable data in the file is absorbed rather than refused, as Lighthouse and Nimbus do, except an attestation whose source is at or past its target outside genesis, which fails the whole import: it is never valid, and read into the record it would refuse every honest attestation until the chain justified past that source. - All or nothing. - `export`: writes the record, or selected keys, as a minimal interchange. Keys with no record are omitted. - `open_offline`: opens the record without a beacon node, creating one for a given chain if there is none. - `interchange_chain`, `contains`, `pubkeys`, `genesis_validators_root`. - `file.rs`: parsing is split from reading. An import must accept a pubkey listed twice, and an attestation whose source is after its target; the vectors require both. The client's own record still refuses a duplicate pubkey. - **`ethlambda slashing-protection import <file>` / `export [--pubkeys ...] <file>`** (`bin/ethlambda/src/slashing_protection.rs`): - offline, against `--validators-dir`; - takes the validator client's lock, so it refuses while a client runs there; - export refuses to overwrite an existing file. - **Keymanager API:** - `POST /eth/v1/keystores` judges `slashing_protection` before touching anything: if it does not parse or is for another chain, every key in the request fails and none is loaded. It is written before any key is saved, so a crash between saving a key and registering it cannot load a returning key on the record it had before it left; the history only ever raises watermarks, so a key that then fails to save leaves nothing dangerous behind. The keys are registered after they are saved, under the definitions lock, so a concurrent `DELETE` cannot interleave. - `DELETE /eth/v1/keystores` returns the deleted keys' real records, replacing the deliberately empty, all-zero-root interchange. - The export is taken after the keys leave the store, so it covers every signature made with them. - A key that is gone but still has a record answers `not_active`. - **Interop vectors:** - `make slashing-protection-interchange-tests` downloads v5.3.0. It is pinned, its SHA-256 is checked, and the download is stamped by version. `make test-node` depends on it. - `crates/validator/tests/interchange_vectors.rs` runs all 38 cases against the `should_succeed` (minimal) expectations, as Teku, Nimbus and Lodestar do. - **Docs:** `docs/cli.md` gets a new `slashing-protection` section and the keymanager notes; `docs/spec_deviations.md`, `README.md` and the crate docs are updated. ## Correctness / Behavior Guarantees - **An import never makes the record less strict:** every field is merged by maximum, including against existing records and repeated entries in the same file. - **The chain must match:** an interchange for another chain is refused by the record and by the keymanager. The offline import into an empty directory creates the record for the file's chain, and the client refuses it at its next start if that is not its beacon node's chain. - **Keymanager ordering:** on `POST`, the history is judged and written before any key is saved, so no key ever signs, or loads after a crash, ahead of its history. On `DELETE`, the keys leave the store before the export. If the export fails, only the keys actually deleted become errors; `not_found` and malformed pubkeys keep their statuses. - **Behavior change:** `DELETE /eth/v1/keystores` now returns real history with the real `genesis_validators_root`, where it used to return an empty one with an all-zero root. ## Tests Added / Run - **The 38 EIP-3076 v5.3.0 vectors, all passing.** They cover 150 signing attempts, 113 of which must be refused. - Mutation check: with the double-vote rule weakened (`<=` to `<`), 10 of the 38 fail. - **Unit tests:** - import raises the watermarks and survives a restart; - import never lowers a watermark; - another chain changes nothing; - export lists only keys with a record and round-trips into a fresh record; - `open_offline` needs a record or a chain. - **Keymanager tests:** - an import brings its history; - history for another chain imports no key; - delete returns the key's history, then `not_active`. - **CLI:** a parse test for `slashing-protection export`. Smoke-tested by hand: import, export, a refused overwrite, and a missing record. - **Commands run:** - `make fmt`: clean - `make lint`: clean - `make test`: 2888 passed, 0 failed (after the review follow-ups) ## Related Issues / PRs - Stacked on #734 (durable slashing protection). - Follow-up: doppelganger protection. ## ✅ Verification Checklist - [x] Ran `make fmt` — clean - [x] Ran `make lint` (clippy with `-D warnings`) — clean - [x] Ran `make test` (`test-consensus` plus `test-node`, at `release-fast`) — all passing
## Motivation The project's Telegram group is now [t.me/ethlambda](https://t.me/ethlambda), but the docs still sent readers to `t.me/lambdaconsensus` (set in #766). ## Changes Replaces the Telegram URL in the three places it appears: - `README.md` - `CONTRIBUTING.md` - `docs/introduction.md` (mdbook) The link label ("ethlambda group") and the surrounding text are unchanged. A sweep of the tracked files finds no other Telegram link, old or new.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports how gattaca-com/silver (main
823d67d) uses BLS and KZG, on the blst and c-kzg backends this repo already has. No new dependency. The stacked PR on top re-points the same patterns atsilver_crypto_simd.What changes
crypto/src/bls/hashed_messages.rs). Hashing a signing root to G2 costs about 100 µs, and one root is signed by every attester of a slot. Two generations of 256 roots, as Silver does. Every single check and the batch take their hash from it, through a hand-built pairing inbls/primitives.rs, which is now the only module with raw blst point calls. An identity key is refused explicitly there, since a hand-built pairing would otherwise skip it where blst's verifier fails it.bls/batch.rs). Each distinct message gets one weighted key sum and one Miller loop, and a weighted signature sum at infinity is refused. Every message and Miller loop is its own rayon task: blst's batch runs one set per worker, and grouping several on one thread made the batch slower than nightly's.bls/warm.rs,warm_registry_pubkeysinBlockChain::spawn_beaconand in the replay benchmark), so the first block after a start does not validate tens of thousands of keys inline.p2p/src/beacon/batched.rs,gossip::Prepared,gossip::column::ColumnPrepared). Subnet attestations plus aggregates, sync committee messages, and data columns are queued, one drainer per kind, up to 1024 per batch, with the work inside a batch spread over rayon. A batch is oneverify_signature_setscall (oneverify_cell_kzg_proof_batchcall for columns) and falls back to per-message checks if it fails. Each rule is split into "prepare" (every non-signature rule, in order, stopping at the first failure) and "resolve" (the first failure in the original order), so verdicts are the ones the one-by-one path gives. A queue past its cap, or a missing permit, answersIgnore(Overloaded)as before. New metrics:lean_beacon_gossip_batch_size{kind},lean_beacon_gossip_batch_fallback_total{kind},lean_beacon_hashed_message_cache_lookups_total{result}.Not ported: Silver's batched G2 signature decode, its blst single verifies, its
ringRNG.Benchmark (ethrex-office-4, Ryzen 9 9950X3D, idle; nightly
2b51a3336as control)Import replay of 3 mainnet corpora, legs rotated; warm = 2 rounds, cold = 1 round. STF ms, mean.
silver_crypto_simd(stacked PR)Gossip-shaped microbench, ms:
What this does and does not buy
verify_batchhave unit tests, but no live node has run it.Open questions
start_actor.Tests
make lint;ethlambda-cryptolib 62;ethlambda-state-transitionlib 732; beacon spec tests 7493 passed, 977 ignored;ethlambda-p2plib 384;ethlambda-blockchainlib 317;ethlambdabin 287. New tests cover the hash cache, a grouped-batch oracle against per-set blst checks, cancelling keysPand-P, a weighted signature sum at infinity, the registry warm-up, every gossip rule's prepare/resolve order, the batcher queue protocol, andverify_batchwith real signatures.