Repository navigation
perf(beacon): keep inactivity scores and the ring buffers on the tree - #650
MegaRedHand merged 2 commits into
Conversation
Import-replay benchmarkMethod.
This matches the estimates: peak RSS was expected at 4.0 → 3.3 GiB and the ordinary-block saving at ~30 ms (47 ms measured). It is the only one of these PRs that lowers memory. RSS at the end of the import was not recorded, only the peak. The peak was consistent across both rounds (3.31 / 3.31 GiB). All five performance PRs together (#647, #648, #649, #650, #633), measured with a 1 s delay between blocks so the state writer drains (
In that build #633's fused epoch pass reads and writes the tree-backed inactivity scores in order ( |
…3-64-633-636-638-gloas-live Conflicted files: - crates/blockchain/state_transition/src/beacon/stf/epoch/altair.rs - crates/blockchain/state_transition/src/beacon/stf/mod.rs - crates/common/ssz-tree/tests/model.rs - crates/common/types/src/beacon/containers/mod.rs - crates/common/types/src/beacon/containers/shared.rs Gloas adaptations: - tree_fields! gets a gloas line: validators and balances (progressive trees) plus block_roots, state_roots, historical_roots, eth1_data_votes, randao_mixes, slashings and historical_summaries, which gloas takes from the shared aliases. inactivity_scores is left out on purpose: gloas declares its own flat libssz ProgressiveList for it, which buffers nothing, so there is nothing to flush or rebase. - ssz-tree: ProgressiveList implements Buffered, so the field list can hold the progressive registry beside the bounded fields. - apply_pending_mutations / has_pending_mutations dispatch over all eight beacon forks through buffered()/buffered_mut(); the lean guard on has_pending_mutations is kept (the merge had dropped it with the old registry match). - rebase_on: the validators/balances list-kind match stays (so a gloas state still rebases onto a gloas base only); the other fields are shared types in every fork and rebase unconditionally; inactivity scores rebase only for a Tree/Tree pair. - Inactivity scores are a tree List before gloas and a flat slice-like list in gloas, so no &[u64] can serve both. altair_validator_lists now returns an InactivityScoresRef view (len, get, in-order iter, ptr_eq, has_pending_updates, PartialEq) as its third element, and inactivity_scores_mut (a &mut [u64]) becomes inactivity_score_mut(index), an element write that dispatches over both kinds. - stf/epoch/altair.rs update_inactivity_scores: decides over an in-order walk of the scores (in step with the summary) and writes only changed scores afterwards, which subsumes both #633's "write if different" and #650's no-op skip; the error order (first eligible validator without a score, then checked-add overflow) is unchanged. - stf/mod.rs tests: BEACON_FORKS now includes Gloas, the historical summaries write covers gloas, and the inactivity score case expects no pending write on gloas. process_slot's registry check uses the new BeaconState::registry_has_pending_updates, since the per-slot roots writes stay buffered by design. - storage store.rs test and the epoch-processing fixture runner comments follow the renamed accessors. Other: shared.rs imports both List/Vector and the progressive alias; model.rs keeps both sides' tests (the base side was empty).
…3-64-633-636-638-gloas-live
1fef7d8 to
203ab96
Compare
…tes (#683) Squash of feat/prune-finalized-states at 43143ae (merge-base aab9864). Adapted onto #628, #634 and #650: the key-layout bump is DB_VERSION 7, since nightly already spends 5 (#634's gloas config keys) and 6 (#628's unrealized justifications). The evict doc keeps #634's three scratch caches, and the mid-interval test bumps its balance through #650's balance_mut. States are now keyed by their block's slot as read from latest_block_header, so the fixtures that stored a state whose header slot is not its block's give it that slot, as this PR does for main's tests: #628's and #634's fork-choice resume tests, #634's proposer-preferences eviction test and #650's tree-sharing test. The slot-peek test covers a gloas block, since #634 made gloas the newest fork.
🤖 Codex Code ReviewLooks good overall. The PR is performance-focused and the key correctness-sensitive changes appear sound: flushing buffered tree writes before state-root computation closes a real cache/pathology gap without changing consensus results, and the inactivity-score optimization preserves semantics while avoiding unnecessary tree unsharing.
Briefly: I don’t see blocking issues in this PR. The added tests around pending mutations and subtree sharing make the optimization credible. Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
🤖 Claude Code ReviewReview of PR 650: move more beacon state fields onto the persistent treeI read the diff and did not build it or run the tests. Overall it looks correct and well tested. I found one minor gap and a few small points. What the PR does
Correctness
Minor
Verdict: approve once CI is green, in particular the lean spec tests, given the Automated review by Claude (Anthropic) · sonnet · custom prompt |
🤖 Kimi Code ReviewI'll review this PR which converts more beacon state fields to tree-backed (SSZ tree) structures for copy-on-write efficiency. This is a significant optimization touching consensus-critical data structures. High-Level AssessmentThe PR converts Overall direction is correct, but I've identified several correctness and safety issues. Critical Issues1. Missing
|
| Priority | Issue | Location | Fix |
|---|---|---|---|
| Critical | Missing historical_summaries in rebase_on |
containers/mod.rs:630-653 |
Add conditional rebase for historical_summaries |
| High | Storage test missing state_roots ptr_eq check |
store.rs:6147-6154 |
Add assert!(decoded.state_roots().ptr_eq(resident.state_roots())) |
| Medium | Comment clarity in test helper | stf/mod.rs:373 |
Fix garbled comment |
| Low | Consider adding historical_summaries ptr_eq to storage test |
store.rs |
After fixing rebase_on |
The core logic is sound. The H256 basic type forwarding is critical and correctly implemented. The inactivity score optimization preserves sharing as intended. The main gap is historical_summaries not being included in rebase_on, which wastes memory and breaks the sharing contract for that field.
Automated review by Kimi (Moonshot AI) · kimi-k2.6 · custom prompt
ilitteri
left a comment
There was a problem hiding this comment.
Low: historical_summaries is not rebased onto resident states.
BeaconState::rebase_on shares the other newly tree-backed fields but not historical_summaries (crates/common/types/src/beacon/containers/mod.rs:1011-1018), even though that field becomes tree-backed in crates/common/types/src/beacon/containers/shared.rs:120-121.
On Capella or later, a state reconstructed from storage therefore cannot reuse the resident state's unchanged summaries subtrees or cached hashes. This does not change contents or roots, but it loses the intended copy-on-write sharing and adds memory retention and rehashing.
Please conditionally rebase summaries when both states are Capella or later, and extend the storage sharing test with a non-empty summaries list and a ptr_eq assertion.
…ce states are keyed by slot||root (#683) and are read back under their block's slot
2495d66 to
6e9a3ac
Compare
…ambdaclass#651) ## Motivation Beacon post-states are persisted by one background thread fed through a bounded queue. `StateWriterHandle::send` blocks while the queue is full, and `Store::insert_state` calls it from inside `fork_choice::on_block`, which the chain actor times as the `stf` phase. If blocks import faster than the writer persists them, the importer's wait for the writer is reported as state-transition time. This PR reports the wait as its own phase, so `stf` measures state-transition work only. Measured on the import-replay benchmark (128 mainnet blocks, eth-5) with this change merged into each leg: | leg | `stf` p50 (now exclusive) | `writer_wait` p50 | block wall p50 | |---|---|---|---| | base (beacon-chain-integration) | 581 ms | ~0 | 777 ms | | each single performance PR (lambdaclass#647-lambdaclass#650, lambdaclass#633) | 353-618 ms | ~0 (max 4 us) | 551-778 ms | | all five performance PRs merged | 202 ms | **326 ms** | 559 ms | With all five merged, blocks import faster than the writer persists their states, so the queue fills and the importer waits about 326 ms per block. Without this phase that wait was reported as `stf` (about 530 ms instead of 202 ms), which hid most of the combined gain and pointed at the wrong place. The state writer, not the state transition, is now the replay's bottleneck. ## Changes | Area | Change | |---|---| | `storage/state_writer.rs` | `StateWriterHandle::send` returns the `(start, end)` instants around its blocking `send` | | `storage/store.rs` | `insert_state` records the hand-off span (first start, last end if repeated) in a shared field; new `Store::take_state_handoff()` takes and clears it | | `blockchain/import_timing.rs` | `writer_wait_start`/`writer_wait_end` on `ImportTimings` and `StoreTimings`, copied in `absorb_store`; new `writer_wait` row after `stf`; header documents the nesting | | `blockchain/lib.rs` | Beacon arm takes the hand-off instants right after `on_block` returns (both `Ok` and `Err`); clears any stale span before the call | | `blockchain/metrics.rs` | `writer_wait` added to `BLOCK_IMPORT_PHASES` (the `phase` label list) | | `docs/metrics.md`, `docs/benchmarking.md` | Phase documented; beacon `stf` now excludes the wait | The replay benchmark builds its `phases` map from `BLOCK_IMPORT_PHASES` and the same histogram, so it picks up `writer_wait` and the exclusive `stf` with no code change. One beacon `on_block` makes one `insert_state` call. The store still merges repeated calls (first start, last end), the module's convention for repeated sections. ## Semantics change Beacon `stf` (the report row and the `lean_block_import_phase_seconds{phase="stf"}` series) now excludes the writer wait: it is the `on_block` span minus `writer_wait`, saturating at zero. The rows still sum to the end-to-end time. Lean blocks never set the new instants, so their `stf` and report are unchanged. A new `phase="writer_wait"` series appears on beacon only. Dashboards comparing beacon `stf` across this change will see it drop by the wait. ## Testing | Command | Result | |---|---| | `cargo fmt --all` | clean | | `cargo clippy --workspace --all-targets --profile release-fast -- -D warnings` | clean | | `cargo test -p ethlambda-storage --profile release-fast --lib` | 139 passed, 1 ignored | | `cargo test -p ethlambda-blockchain --profile release-fast --lib` | 210 passed, 0 ignored | | `cargo check -p ethlambda --profile release-fast` | clean | New tests: `stf` minus `writer_wait`, wait absent, wait longer than `stf` saturating to zero, row order and presence (existing label-list test also covers the new label), and a storage test that `insert_state` records a start <= end pair and that taking it clears it. Spec suites and the replay benchmark were not run. Clippy ran without `--locked`: `Cargo.lock` on the base (125dce1) is already stale (`zk_alloc` is listed under the wrong crate), so `--locked` fails there with or without this change. The lockfile is not part of this PR.
Motivation
Now that
validatorsandbalancesare persistent trees, every cached state still holds a private deep copy of the other large fields and rehashes them in full on each state root.inactivity_scoresrandao_mixesblock_roots,state_roots,slashingseth1_data_votes,historical_roots,historical_summariesEstimates, not measurements: ~26 MiB of flat fields per state and ~0.85M Merkle nodes rehashed per root, of which this PR moves the large majority of the nodes (participation lists and queues stay flat). The import-replay benchmark was not run for this PR, and no memory or node-count number was measured; the estimates come from the field sizes above.
Changes
perf(ssz-tree): flush tree fields through one Buffered list per forkBufferedtrait (impls forListandVector);tree_fields!lists each fork's tree fields once and generates bothapply_pending_mutationsandhas_pending_mutations.H256writesHashTreeRootby hand and forwardsis_basic_type(asH160does). New ssz-tree properties: multi-leafu8lists,u64and 32-byte-newtype vectors, decode parity, rebase of mostly-zero lists and vectors.perf(beacon): keep inactivity scores on the tree and skip no-op writesInactivityScoresis a tree list (BTreeMapupdate map).process_inactivity_updatescomputes the same value with the same checked arithmetic and writes onlyif new != score. Joins the flush list andBeaconState::rebase_on.perf(beacon): keep the roots and randao ring buffers on the treeBlockRoots,StateRoots,RandaoMixes,Slashingsbecome tree vectors;Eth1DataVotes,HistoricalRoots,HistoricalSummariesbecome tree lists (alias swap). All join the flush list andrebase_on. Block production now flushes before it hashes the post-state. Test-only sites that sliced or iterated in place use indexing.SSZ bytes, roots and the storage delta format are unchanged.
Design decisions
H256change done locally, not in libssz'stransparentderive (no dependency bump). The lean chain sharesH256, soethlambda-typeshas unit tests that anSszList/SszVectorofH256hashes like one of[u8; 32], and that a container holdingH256collections keeps its root. The lean state's pinned genesis root test passes unchanged.BTreeMapupdate map for inactivity scores. Outside a leak almost no score changes, so writes are sparse. A leak writes nearly every score, which costs about one B-tree insert per validator per epoch (~1M at the benchmark's registry size) until a bulk write path exists. A dense map (VecMap) was the alternative; it allocates 16 B per validator on the first push or write.process_inactivity_updates: read the score, compute the new one exactly as before (overflow still raisesArithmeticOverflowbefore the recovery step), write only if different. A write rebuilds its leaf even for an equal value, which would unshare the list from the parent state's.rebase_onpairs scores through the altair accessors, so it works across forks and skips phase0 (no scores).process_slotleaves its two roots writes buffered until the next flush (the next slot's, orprocess_slots' last step).process_historical_summaries_updatehashes the roots vectors with that write pending: correct, and the vectors are small. Tests pin this.produce_blockhashed the post-state withoutapply_pending_mutations, so the computed hashes were discarded; one added call.Bufferedandtree_fields!are additive in ssz-tree andcontainers/mod.rs, to keep a merge with other work in the same files easy.Not in this PR
previous_/current_epoch_participation)get_unslashed_participating_indices(#632): its per-index reads would cost ~2M tree descents per block in the pulled-up tip. Also needsList::filledfor an O(1) rotation.pending_depositson the treeArc+ cached-root newtype)previousequals the base'scurrent)ifinrebase_on, left for the participation PR.Testing
cargo test -p ethlambda-ssz-tree --profile release-fastcargo test -p ethlambda-types --profile release-fast --libcargo test -p ethlambda-storage --profile release-fast --libcargo test -p ethlambda-state-transition --profile release-fast --libcargo test -p ethlambda-rpc --profile release-fast --libcargo clippy(workspace, and state-transition withbeacon-spec-tests) with-D warnings,cargo fmt --check,cargo check -p ethlambdaThe 4 failures in each spec run are the
gossip/*trials, which report "matched no fixture cases": the gossip fixture directory is intentionally empty in the local setup. Everyssz_static,sanity,finality,random,epoch_processingandtransitiontrial passes. The leanssz_spectestsand the import-replay benchmark were not run.