Repository navigation
feat: bump test fixtures to latest version - #37
Merged
Merged
Conversation
MegaRedHand
marked this pull request as ready for review
January 19, 2026 16:50
tcoratger
pushed a commit
to tcoratger/ethlambda
that referenced
this pull request
Oct 30, 2026
…es (lambdaclass#37) * feat(ssz-tree): add the crate skeleton with packing and depth rules The beacon state keeps its large lists in flat vectors, so every state root rehashes them whole and every cached state holds its own copy. This crate will hold them in a persistent Merkle tree instead, one shaped like the SSZ merkleization so each node can cache its hash and be shared. This first step fixes the two rules the tree's shape follows: how many elements pack into a leaf, and how deep the tree is for a given limit. A basic type whose size does not divide 32 is refused, because SszList packs it across chunk boundaries and a one-chunk-per-leaf tree would produce a different root. * feat(ssz-tree): buffer pending writes in an UpdateMap Writing straight into a persistent tree copies the whole root-to-leaf path on every write, so an epoch that touches every balance would copy each path once per element. Buffering the writes and folding them in one pass copies each shared path once. The buffer is a trait so each list can pick its own: a BTreeMap for the validator registry, where a block writes a handful of large elements, and a dense VecMap for balances, where an epoch writes most of them. Which one wins per field is left to a later benchmark. * feat(ssz-tree): add a persistent Merkle tree that caches node hashes The tree has the exact shape of an SSZ list's merkleization: packed leaves for basic types, one leaf per element for composites, and shared zero subtrees for the unused capacity. Each node keeps its hash once computed, so a later root only rehashes the nodes a write replaced. Writes rebuild only the root-to-leaf paths they touch and share every other subtree with the previous version through Arc, which is what lets several cached states hold one copy of an unchanged registry. Hashes are filled with OnceLock get() then set() rather than get_or_init: hashing runs children under rayon::join, and a worker blocked in get_or_init can steal a job that needs the cell it is filling. Racing workers only duplicate work; they compute the same value. * feat(ssz-tree): iterate a tree in order, pending writes included Scans over the registry (pubkey lookups, epoch sweeps) would cost a full root-to-leaf descent per element if they went through get. The walk keeps a stack of deferred right subtrees instead, so each node is visited once and a scan is linear in the number of elements. The public iterator overlays the pending writes on the walk, so an element written since the last apply_updates reads back its new value, and pending pushes appear after the committed elements. * feat(ssz-tree): add List, a tree-backed drop-in for SszList The beacon state's registry fields are SszLists inside derived containers, so their replacement has to encode, decode and hash exactly as SszList does, and offer the element API the state transition already calls (index, get_mut, push, iter). List does both, with the tree and its pending writes held in an Interface that Vector will share. Fixed-size elements decode straight into tree leaves, so loading a state never holds a second full copy of the registry in a Vec. Every read, encode and root sees pending writes; a root taken with writes still pending is computed on a copy, correct but not cached, so the state transition is expected to apply them first. Equality shortcuts on a shared tree only when nothing is pending, since two clones of one tree can carry different writes. The tests compare encodings, decode rejections and roots against SszList as the oracle. * feat(ssz-tree): add Vector, the fixed-length counterpart of List The state's other large fields (block and state roots, randao mixes, slashings) are SszVectors, so benchmarking them on the tree later needs a Vector that stands in for SszVector the way List stands in for SszList. It shares List's core; it has no push, its length is part of the type, and its root has no length mix-in. * feat(ssz-tree): rebase a tree onto a relative to share its subtrees A state decoded from storage builds its lists from scratch, so it shares nothing with the resident states it mostly equals, and each decode would add a full registry to memory. Rebasing walks the decoded tree beside a resident one and swaps in the resident node wherever the contents match, so what stays unshared is only what actually differs, and the swapped nodes bring their cached hashes with them. Equal cached hashes end the walk only for subtrees wholly inside both lists' shared prefix. At the boundary a shorter list can hash like a longer one, since a trailing zero value in a packed chunk looks like padding, so there only a value comparison counts. * fix(ssz-tree): report too many vector elements the way SszVector does The shared decoder caps a fixed-size element count at N like a list does, so a Vector given too many elements failed with InvalidByteLength where SszVector, which decodes first and counts afterwards, fails with InvalidFixedLength. Both rejected the same inputs; checking the count up front makes the errors match too. * test(ssz-tree): check List and Vector against SszList and SszVector Unit tests pin the cases someone thought of; the tree's failure modes are in combinations of pushes, overwrites and flushes that land on packing and depth boundaries. These properties run random sequences of them and compare every read, encoding and root with libssz's own types, for packed, composite and variable-size elements, including a list at the registry's 2^40 limit. Two more properties cover what those do not reach: decoding arbitrary and mutated bytes must accept or reject exactly as libssz does, and rebasing must keep a list's contents and root while sharing everything it has in common with the base, including across the trailing-zero boundary where hashes cannot be trusted. * feat(types): keep the beacon registry and balances in persistent Merkle trees The validator registry and balances are the bulk of a mainnet state: about 2.4M entries each. As SszLists, every state root rehashes them in full and every cached state holds its own copy, which made state merkleization 11.8% of block import CPU and multiplies memory by the state cache's size. Backed by ethlambda-ssz-tree's List they keep per-node hashes, so a root after a block rehashes only the touched paths, and a state cloned from another shares every unchanged subtree with it. The registry buffers writes in a BTreeMap, since a block writes a few large records; balances use the dense VecMap, since an epoch writes all of them. apply_pending_mutations folds the buffered writes in before a root is taken, and rebase_on lets a state built apart (decoded from storage) share subtrees with a resident one. Both are wired into the state transition and the store in the next commits. * perf(stf): flush registry writes before every state-root computation A tree-backed list hashed with writes still pending computes its root on a throwaway copy: the answer is right, but none of the new hashes are kept, so the next root pays again. Folding the writes in first makes each root cache what it computes in the state's own nodes. The flushes sit where roots are taken: at the top of process_slot and after process_block. process_slots also flushes before returning, because epoch processing runs after the last process_slot of its loop; a state advanced without a block (a checkpoint state, an empty-slot pre-state) would otherwise be cloned and cached with a whole epoch of balance writes still buffered. * perf(storage): rebase a decoded beacon state onto a resident relative A state read from the database is decoded into fresh trees, so every state-cache miss would add a full private copy of the registry beside cached states it nearly equals. Rebasing it onto a resident state right after decode swaps in the resident nodes wherever the contents match, so only what differs stays private. The parent block's state is the closest relative when it is cached; otherwise the most recently used state still shares nearly all of the registry. The rebased state is the decoded one: only the allocations behind it change. * test(types): benchmark the tree-backed registry at mainnet scale Whether the tree pays off depends on numbers no unit test shows: the live heap of one state and of each state derived from it, the rehash after a block's writes, and what the fixed tree depth costs reads and epoch sweeps. This ignored test measures each against SszList at 2.4M validators, with a counting allocator for memory, and asserts on the way that the tree's roots, encoding and rebase agree with SszList. The reads it times include the access patterns the state transition actually has: a block's worth of random attester lookups, and the effective-balance sweep both by index and with zipped iterators. * refactor(ssz-tree): keep Tree private to the crate Tree was exported only so docs could link to it, but none of its methods are public, so outside code could only build arbitrary trees from its variants and name nothing it could use. The public surface is now List, Vector, Iter and the update maps. Also documents UpdateMap::get and get_mut, and stops the docs claiming the benchmark tunes PARALLEL_HASH_HEIGHT: it measures the value. * perf(storage): never cache a beacon state with buffered writes Once a state is behind the cache's Arc it can no longer be flushed, so a buffered write would send every later root taken through it down the slow path, which copies the pending writes and caches no hashes. The store now flushes a state before wrapping it for the cache, on insert and on decode, and cache_state asserts in debug builds that whatever it is handed is already flushed. * docs: describe the tree-backed beacon registry in CLAUDE.md Adds the ssz-tree crate to the codebase map, and notes under the beacon types what changes for code touching validators and balances: no slices or iter_mut, and buffered writes that the state transition flushes before each state root. * test(stf): flush the gossip fixture's registry writes before caching it `fulu_parent`, the gossip tests' shared parent state, sets every validator's pubkey through `validator_mut`. With the registry now in a persistent Merkle tree those writes are buffered, and the block and column tests cache the state through `Store::cache_state`, whose debug assertion refuses an `Arc` holding writes it can no longer flush. The fixture predates the tree (it came in with gossip validation), so nothing flushed it and 14 tests panicked on the assertion. Flushing where the fixture is built hands the tests a state shaped like every one production caches, instead of teaching each call site to flush. * perf(ssz-tree): keep a page of elements per leaf A leaf held one SSZ chunk: one Validator, or four balances. At mainnet scale that is millions of Arc'd nodes, and every lookup walked the tree down to a chunk and then through one more pointer to the value. The profile of the import replay put BeaconState::validator at 1.65 s of CPU per block, most of it that walk, and one state's registry took twice the memory of the Vec it replaced. A leaf now holds the elements of a whole subtree, about 4 KB of them stored contiguously, so a lookup stops at the leaf and indexes into it, iteration walks each leaf as a slice, and the tree has far fewer nodes to allocate. The Merkle shape is unchanged: a leaf hashes its chunks up to its own height, so every root is the same. Rebuilding a leaf now copies its run, so a composite leaf keeps each element's root and a rebuilt leaf carries over the roots of the elements it did not change: rehashing it folds 31 cached roots rather than rehashing 32 validators. * perf(ssz-tree): keep a page of child pointers per inner node Inner nodes were binary, so a lookup still walked one node per level above the leaves: 35 of them for the registry, whose limit sets the depth, most landing on a cold cache line. An inner node now holds up to a page of child pointers and stands for that many binary levels at once, so the same lookup crosses a handful of nodes. The Merkle shape is unchanged. A node folds its children's cached roots up its own levels, padding a missing child with the zero hash of the children's height rather than of a chunk, which is why this does not go through merkleize. Levels are counted up from the leaves, so only the root can span fewer levels than the others. The cost moves to writes: a rebuilt node copies its child pointers and refolds its levels from its children's cached roots, rather than hashing one pair. * docs: describe the tree-backed registry and how to loop over it The page-sized leaves and inner nodes change what a registry read costs, and the import profile showed where that bites: helpers that collect the active indices and then read each one back through validator(i). A contributor writing the next epoch pass needs to know to iterate instead. beacon_stf.md gains a "Registry and balances" section on the layout, the buffered writes and the access pattern; CLAUDE.md points at it. benchmarking.md's replay memory figure described flat states and is replaced by the measured tree-backed one. * docs(ssz-tree): compare the tree to a B+-tree With page-sized leaves and inner nodes the tree reads more like a B+-tree over indices than like the binary Merkle tree it hashes as, so say so, along with the ways it is not one: no keys, no splits, a fixed shape and copy-on-write paths. Also drop two intra-doc links from the public crate docs to private constants, which rustdoc reports as broken.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The current CI is failing due to using a newer version of the spec. This PR updates it, and also adds the changes needed to make the tests pass.
The spec changes this addresses are:
justified_slotsis now relative to the latest finalized slot