Repository navigation
perf(beacon): cache the total active balance per epoch - #648
Conversation
Import-replay benchmarkMethod.
The ordinary-block saving (224 ms) sits just under the estimated 300-414 ms. Peak RSS rises although the cache holds a few All five performance PRs together (#647, #648, #649, #650, #633), measured with a 1 s delay between blocks so the state writer drains (
Without the delay, the combined build outpaces the state writer and waits ~326 ms per block on it. |
…63-64-633-636-638-gloas-live Conflicted files: - crates/blockchain/state_transition/src/beacon/stf/block.rs - crates/blockchain/state_transition/src/beacon/stf/electra.rs - crates/blockchain/state_transition/src/beacon/stf/fulu.rs - crates/blockchain/state_transition/src/beacon/stf/operations.rs - crates/blockchain/state_transition/tests/beacon_spec/operations.rs - crates/storage/src/store.rs - docs/beacon_stf.md Gloas adaptations: - stf/gloas.rs process_block, process_operations and process_attestation take the ActiveBalanceCache; the gloas block's sync aggregate goes through the shared altair process_sync_aggregate with it. The base reward per increment comes from the cache, and the attester's effective balance is read once per attester (it also feeds the pending builder payment weight), as in the other forks. - Soundness check for gloas: effective_balance has one writer outside the fork upgrades (queue_entire_balance_and_reset_validator, called only by the electra upgrade) and genesis, besides the epoch-end process_effective_balance_updates. Gloas block processing (parent payload application, withdrawals, bids, builder payments and deposits, execution requests, slashings, exits) writes balances, exit epochs, slashed flags and queues, never an effective balance, and exits and activations land at least MAX_SEED_LOOKAHEAD epochs ahead, so the cache is used for gloas too. A full or empty parent changes none of those inputs. The debug cross-check on every hit runs in the gloas sanity and transition fixtures. - The gloas arm of block::process_block keeps its engine-free call and gains the cache argument. - fork_choice.rs and the fixture runners (operations gloas arm) pass the cache; get_total_active_balance walks iter_validators(). Other: store.rs keeps both the liveness tracker and the cache field and accessor; fulu.rs keeps its own process_operations and pending-queue import with the cache argument; docs follow the element accessor names and no longer mention registry_mut.
…636-638-gloas-live
Conflicted files:
- crates/blockchain/src/lib.rs
- crates/blockchain/state_transition/src/beacon/fork_choice.rs
- crates/blockchain/state_transition/src/metrics.rs
- docs/metrics.md
Gloas: the precompute stays enabled, because the base state cannot differ.
In this branch a gloas block's post-state never holds its own payload: the
child applies it (process_parent_execution_payload) inside process_block,
after process_slots, and on_execution_payload_envelope only records and
checks, it stores no second state. The store keeps one state per root and
both on_block (get_state(parent_root)) and checkpoint_state read it, and the
worker clones get_state(head_root) and caches the result under
CheckpointState { E+1, head_root }, so the entry is process_slots of the very
state an import of a child of that root clones. The fulu-to-gloas boundary
needs no case either: the upgrade runs inside process_slots on both paths.
A new unit test (a_gloas_precompute_starts_from_the_state_an_import_would)
checks, for a gloas parent, that the worker's result equals
checkpoint_state's for the key, that an import finds it, and that advancing
the rest of a skipped first slot from it equals plain process_slots.
Other adaptations:
- transition_block and stf::apply_block take the ActiveBalanceCache that
#648 added to state_transition; on_block passes the store's own cache.
- on_block's gloas-specific code between the parent read and the
transition does not use the cloned state, so removing that early clone
(as #652 does) needed nothing else.
- lib.rs: the NewBlock handler keeps the envelope settling that follows the
import and triggers the precompute after it, so a head moved by a settled
envelope's block is seen.
- metrics.rs and docs/metrics.md: both sides' metrics kept.
- Test fixtures of #652 follow this branch's BlockInputs (operations) and
the extra argument.
- docs/beacon_stf.md notes why gloas shares the key.
…63-64-633-636-638-gloas-live
get_total_active_balance built an index Vec of the active validators and then read each one back with a validator(i) tree descent. Sum the effective balances in a single in-order walk over state.validators().iter() instead, with the same saturating add and one-increment floor as get_total_balance. Every caller (block, epoch, fork choice, churn) gets the speedup without a signature change.
An ordinary block asks for the total active balance about nine times (every attestation through get_base_reward_per_increment, twice in the sync aggregate), and each ask is a registry pass. Its value is fixed by the epoch and the block root at the last slot of the previous epoch: effective balances are written only by that epoch's effective-balance step, and every activation or exit lands at least MAX_SEED_LOOKAHEAD epochs ahead. Keep it in an ActiveBalanceCache held by the Store next to the committee cache, keyed by that (epoch, decision root), so sibling branches never share an entry they should not. The cache is external to BeaconState: no new state field, no SSZ change. The state-aware half (key derivation, one-pass computation, a debug cross-check on every hit) lives in helpers::accessors as ActiveBalanceCacheExt, mirroring CommitteeCacheExt. Block processing takes the cache next to the committee cache; on_block uses the store's. Spec runners, block production and unit tests pass a fresh default. Epoch processing, churn helpers and fork choice keep the uncached one-pass function. Adds lean_beacon_total_active_balance_lookups_total.
The three process_attestation variants (altair, deneb, electra) read the attester's effective balance with a validator(i) tree descent inside the per-flag loop, up to once per newly earned flag. Read it lazily once per attester instead: the first newly earned flag fetches it, the others reuse it. Staying lazy keeps the error behavior identical, since a validator that earns no flag is never read. The effective balance cannot change inside the read-only phase, so the arithmetic is unchanged.
a4a9958 to
c2b7dc8
Compare
Hoodi ran as 512-block smoke tests because a deneb-era block cost about 4.6 s: get_base_reward_per_increment scans the whole registry once per attestation. This assumes #648 (per-epoch total-active-balance cache) is merged, which removes that scan and lets full Hoodi eras fit the job limit. The entries are 8 (last deneb blocks, electra upgrade in the closing process_slots), 198+199 (fulu upgrade plus the first fulu era) and latest. Mainnet 290 stays excluded: #648 does not touch the phase0 path.
Main gained the gloas stack (#634, #635, #639, #662, #663). Adapted to it: gloas block processing (its attestations and sync aggregate) reads the total active balance from the same cache, which gloas block production passes in, and the cache's registry scan walks iter_validators so it covers gloas's progressive registry.
# Conflicts: # crates/blockchain/state_transition/src/beacon/block_production.rs
…st from the snapshot (#665) ## Motivation On the eth-3 followers (48h of import timings, 10-03 to 10-05), the head recompute at an epoch-start tick sometimes took 0.5-4 s, and the epoch's first block queued behind it: | network | head computes > 0.5 s | slot-0 block queue wait | |---|---|---| | Hoodi | 8 (7 in 2-4 s, 1 in 4-12 s) | up to 2.3 s | | Plataberget | 22 | up to 1.1 s | At that tick the justified checkpoint moves to (E-1, R), and the first head walk builds the justified-balances snapshot from `checkpoint_state((E-1, R))`. When that state has left the 32-entry state cache, the chain actor rebuilds it inline: a full snapshot decode (~0.3 s on Hoodi), plus a whole epoch transition when R's block is before the epoch's first slot (an empty first slot, or an epoch-start block orphaned by a late block). A replay of the logs through a model of the state cache puts the rebuilds at ~100 of ~410 Hoodi epoch ticks. Lighthouse never reads a state at that tick: `BeaconForkChoiceStore::on_verified_block` builds the justified balances at block import, for the block's epoch-boundary checkpoint, into a small cache keyed by (boundary root, epoch), from a state import already has. The tick only looks them up. Prysm keeps a per-checkpoint balances vector too, but builds it from the checkpoint block's un-advanced post-state, which is not what the specification's `checkpoint_states` holds; this PR follows Lighthouse. ## Changes This branch carries two open PRs it builds on, merged in with the resolutions `tmp/bci-...-gloas-live` already uses for gloas (minus #648's active-balance cache and #646's block-production operations, which are not on this branch): - #649, the justified-balances snapshot (`86b07388`) - #652, the epoch precompute and its `transition_block` (`b3017e4b`) The change itself is `6badb4f3`: 1. **A balances cache instead of one snapshot.** The store keeps up to 8 `JustifiedBalances` keyed by checkpoint. When full it evicts the lowest epoch first (ties: oldest insert), and never the current justified checkpoint. Not Lighthouse's plain 4-entry FIFO: on a network with many forks (Plataberget), several blocks crossing one boundary from different parents could flush the canonical entry within an epoch. 2. **Built at import.** - A block at its epoch's first slot caches the balances from its own post-state, under (epoch, block_root): that post-state is what `checkpoint_state` returns for it. - A block crossing from an earlier-epoch parent (empty first slot) caches them from the boundary state, under (epoch, parent_root), inside `transition_block`: from the precomputed state on a precompute hit, or by advancing the cloned parent to the epoch start once on a miss (each slot is still processed once). The boundary state depends only on the parent, so building it before the block is validated cannot cache a wrong value. - `justified_balances()` looks the current checkpoint up and keeps the lazy `checkpoint_state` path for a miss (a restart, or a boundary this node never imported). 3. **The gloas boost reads the snapshot.** `compute_node_weights` fetched the justified checkpoint state on every head walk, only for the gloas boost gate. `is_head_weak`, `is_parent_strong` and `should_apply_proposer_boost` now use the snapshot: the total active balance and vote weights were already there; the equivocator term in `is_head_weak` needs the raw `effective_balance` at the justified state, which the vote weights zero for slashed validators, so the snapshot gains an exact copy as a `u16` count of `EFFECTIVE_BALANCE_INCREMENT` per validator (asserted at build). Committees still come from the head/parent block states. The spec-literal `get_weight`, `gloas_get_weight` and `get_attestation_score` stay state-based as references. On a snapshot hit, no head walk (pre-gloas or gloas) reads the justified state. From review: 4. **The committee total leaves slashed validators out.** The snapshot's total, and the spec-literal `calculate_committee_fraction`, now sum active, unslashed validators only, as consensus-specs #5679 does. Fork choice weighs no vote from a slashed validator, so a boost or reorg threshold sized against their stake outweighs a larger share of the votes that count. #5679 first shipped in v1.7.0-beta.4 and the fixtures are pinned to beta.2, so `docs/spec_deviations.md` records it until the pin catches up; no beta.2 fork-choice fixture tells the two apart. Lighthouse already sums the same way. The raw `effective_balance` copy is unchanged, since `is_head_weak` still needs it for slashed equivocators. 5. **The timer precompute is bounded to a head in its own epoch.** The timer trigger checked only that the head was not past its slot. In a network-wide stall the sync tracker reports synced, so a timer epochs past a stalled head replayed every epoch in between, and again under a new key at each later epoch. A worker now crosses one epoch boundary at most, on either trigger. ## Cost - One in-order registry pass (~25 ms on Hoodi, `lean_beacon_justified_balances_build_seconds`) per epoch-boundary checkpoint, at import, on the importer, as in Lighthouse. - 10 bytes per validator per cached snapshot (8 for the vote weight, 2 for the new raw effective balance): ~12 MB per entry on Hoodi, so ~96 MB with all 8 entries filled. Worth reviewing whether 8 entries (vs 4) earn their memory. ## Testing - New: the cache's eviction policy (bound, lowest epoch, ties, protected justified, replace); a crossing import caches the boundary balances on both precompute paths with the block's state root still verifying; a first-slot import caches from its post-state and other slots cache nothing; a snapshot hit reads no justified state on every head path under both fork rules; a slashed equivocator still counts toward head weakness, matching the state-based reference; a slashed-but-active validator is left out of the proposer score and the snapshot total but keeps its raw balance; a timer whose epoch holds no block, or a stall several epochs long, starts no precompute. - `cargo test --profile release-fast --lib --bins` for storage, state-transition, blockchain and types: 1083 passed, 0 failed. - Beacon `fork_choice` fixtures, consensus-specs v1.7.0-beta.2, mainnet: 206 passed, 0 failed (before items 4 and 5; not re-run locally since, so CI's `test-beacon` jobs are the check). - `cargo clippy --workspace --all-targets -- -D warnings`: clean. ## Not covered - Live effect on the epoch-tick head recompute: worth re-measuring on a follower after deploy. - An `on_block` unit test of the first-slot hook needs a BLS-signed block; it is tested through `transition_block` and the cache helper instead. --------- Co-authored-by: Pablo Deymonnaz <pdeymon@fi.uba.ar>
…on-gloas-live Brings in the justified-balances snapshot that the import-time balances build on. Resolved as tmp/bci-...-gloas-live resolved the same merge (df472fa): gloas's `is_head_weak`/`is_parent_strong` keep their own versions, `compute_node_weights` weighs votes from the snapshot, the snapshot iterates the registry through `iter_validators`, and the gloas `LatestMessage` fields are filled in the moved tests. The metrics hunks take lambdaclass#649's own placement, since lambdaclass#648 is not on this branch.
…-gloas-live The import-time justified balances hook into the precompute's `transition_block`. Taken from tmp/bci-...-gloas-live's resolution of the same merge (23986cf), which also enables the precompute for gloas blocks, minus the active-balance cache (lambdaclass#648) and the block-production operations (lambdaclass#646) that branch threads through `apply_block` and the precompute tests: neither is on this branch.
## Motivation The consensus-spec fixtures check the beacon state transition on small, synthetic states. They never run it on real chain history: full-size registries, real blocks, and fork upgrades applied to the live network's actual state. Replaying the canonical chain from Nimbus `.era` archives covers that, and needs no running node or execution client. ## Description - **`ethlambda replay-era`** (`bin/ethlambda/src/era/`) takes the state stored in era file N-1 and applies every block of file N through `stf::state_transition`. This checks each proposer signature and `state_root`, and every other BLS check runs too. At the end of the era it checks the result against file N: - our state root at slot N·8192−1 must equal the stored state's `state_roots[8191]` - every block root must appear in `block_roots` - the full state root is compared when the boundary slot is empty - **Options:** - `--download` fetches missing files from `https://<network>.era.nimbus.team/` - consecutive `--era` values chain on one in-memory state, so our own `process_slots` performs the fork upgrade - `--max-blocks` gives a smoke run - `--report` writes a JSON report - eras this build cannot process (Sepolia gloas) are refused - **The e2store reader** is hand-rolled, about 360 lines including tests. It uses the `snap` framed decoder already in the workspace. One finding from real files: Nimbus marks an empty slot in the block index by pointing at byte 0 (the Version record), not with the offset 0 the format spec describes. The reader accepts both. - **`.github/workflows/era_replay.yml`** runs on pushes to `main` that touch the STF, types, ssz-tree or era code. It runs the `sha-<7>` image `docker_publish.yaml` publishes for that commit, so it builds no Rust. The matrix: - mainnet `566, 758, 1053, 1422, 1607+1608, latest` - sepolia `222, 518, 869, 1065+1066, 1378` - hoodi `8, 198+199, latest` Each job uploads its report, and a failure on `main` posts to the general Slack channel. `workflow_dispatch` accepts an override such as `mainnet:290 hoodi:9@512`. ## Known limits (details in `docs/era_replay.md`) - **Mainnet phase0 eras are excluded.** Phase0 epoch processing is quadratic in this build: `get_base_reward` (`stf/epoch/rewards.rs:71`) recomputes the total active balance for every validator. Era 290 did not get through one epoch boundary in 30 minutes. This is left for a follow-up. - **Hoodi full eras rely on lambdaclass#648.** Without its per-epoch total-active-balance cache, a deneb-era block costs about 4.6 s, because `get_base_reward_per_increment` scans the registry once per attestation (`stf/deneb.rs:359`, `stf/electra.rs:764`). At that cost a full Hoodi era exceeds the job limit. ## How to test ``` cargo run --profile release-fast --bin ethlambda -- replay-era --network sepolia --era-dir /tmp/eras --download --era 31 ``` Every completed run matched the oracle with no root mismatch: - Sepolia eras 1, 2+3+4 (chained) and 31, run in full at ~15 ms/block - mainnet latest (fulu), first 64 blocks, at 653 ms/block - Hoodi era 8, first 256 blocks
Motivation
get_total_active_balancerescans the registry about ten times per ordinary block: once per attestation (throughget_base_reward_per_increment), twice in the sync aggregate, plus churn and fork-choice callers. Each call built an indexVecof the active validators and then read every index back with avalidator(i)tree descent. In the profile taken for the persistent-tree registry change, this was the largest cost left after hashing (the plan's estimate: about 414 ms per block at mainnet scale; an estimate, not measured here).Changes
perf(beacon): sum the total active balance in one registry passget_total_active_balanceis one in-ordervalidators().iter()pass with the samesaturating_addand one-increment floor asget_total_balance. No indexVec, no descents. Every caller gets it, cached or not. Randomized test (inline SplitMix64) againstget_total_balance(get_active_validator_indices(..)), plus an all-zero registry.perf(beacon): cache the total active balance per epoch in the storeActiveBalanceCache(ethlambda-storage), held by theStorebeside the committee cache (Store::active_balance_cache()). Key derivation and lookup inActiveBalanceCacheExt(helpers/accessors.rs). Attestation processing (altair, deneb, electra) andprocess_sync_aggregatetake it; plumbed up through block processing andstate_transition. New metriclean_beacon_total_active_balance_lookups_total{result}. Docs inbeacon_stf.mdandmetrics.md.perf(beacon): read each attester's validator once per attestationDesign decisions
CommitteeCacheover a non-SSZ cache field onBeaconState. That keeps the state type, its SSZ derive and its fork upgrades untouched: no libssz change, norebase_onadoption, no reset at upgrades.process_effective_balance_updates; the Electra upgrade zeroes only never-activated validators, which are not active at E. The active set at E cannot change inside E, because every activation and exit lands at leastMAX_SEED_LOOKAHEADepochs ahead; deposits append validators that are not yet active; slashing setsslashedbut leaves effective balances alone (the spec's total keeps slashed validators). So(E, block root at the last slot of E-1)identifies the history: two states agreeing on it agree on the total. Sibling branches that diverge during E-1 have different roots, so they get different entries.ActiveBalanceKey) keeps the two apart.unkeyable: computed, not cached.process_effective_balance_updatesand the slot increment a state is in epoch E-1 but already carries E's balances, so a key rooted at the end of E-2 would be wrong there. Block processing never runs in that window (process_slotshas already advanced the state), and epoch processing keeps the uncached one-pass function. Documented onactive_balance_key.debug_assert_eq!, so a missed writer fails the fixtures (release-fastkeeps debug assertions on).on_blockreads the store's cache itself (store.active_balance_cache()), so its signature and the chain actor are unchanged. Other callers (spec runners, block production, unit tests) passActiveBalanceCache::default().Not in this PR
get_proposer_scoreandcalculate_committee_fractionkeep calling the uncached function; another change replaces that total with a justified-balances snapshot.Testing
Measured on release-fast with debug assertions. The import-replay benchmark was not run for this PR, so there is no measured speedup to report.
cargo clippy --locked --workspace --all-targets --profile release-fast -- -D warningscargo clippy -p ethlambda-state-transition ... --features beacon-spec-tests -- -D warningscargo test -p ethlambda-storage --libcargo test -p ethlambda-state-transition --libcargo check -p ethlambda --profile release-fastThe 4 failures per preset are the gossip
matched_fixture_caseschecks: the gossip fixture tree is empty in this environment, so they match no cases. Every other suite passes.