Skip to content

perf(blockchain): import blocks and envelopes before the engine answers - #773

Draft
MegaRedHand wants to merge 3 commits into
mainfrom
perf/async-new-payload
Draft

MegaRedHand wants to merge 3 commits into
mainfrom
perf/async-new-payload

Conversation

@MegaRedHand

Copy link
Copy Markdown
Collaborator

Why

The chain actor awaited every engine_newPayload inside the block import. The head moved only after the EL round trip plus the state transition, and the actor handled nothing else (gossip, ticks, other imports) while it waited. Gloas envelopes did the same.

What

A pre-gloas block that is an optimistic candidate now imports as optimistic as soon as its consensus checks pass, and a gloas envelope applies with its payload NOT_VALIDATED. The newPayload question goes out first, from a tokio task that reports back with a NewPayloadAnswered message. The answer is applied then, the way optimistic-sync.md applies a verdict. Optimism already gates attestation data (503), so validators still attest only to validated payloads.

EL answer Pre-gloas block Gloas payload
VALID validated with its optimistic ancestors; head sent again without the optimistic flag VALID, held children released
SYNCING / ACCEPTED stays optimistic stays NOT_VALIDATED, held children released
INVALID with a hash branch removed (invalidate_block_payload, shared with the FCU path) condemned (condemn_payload_chain)
INVALID with no hash as above taken back out (revert_envelope): only that copy is refused
no answer kept, asked again next slot kept, asked again next slot

Rules (module docs in crates/blockchain/src/beacon_new_payload.rs):

  • Order. A payload is sent only once the payload it builds on has been answered, at most MAX_NEW_PAYLOADS_IN_FLIGHT (4) at once. The Engine API orders only forkchoiceUpdated (common.md), the engine client's connection pool can reorder requests, and a payload whose parent the EL hasn't seen is answered SYNCING and may start a peer sync.
  • forkchoiceUpdated names the newest payload on the head's chain whose request is not queued or in flight, and is sent again when the head's answer arrives. Skipping the call instead would starve the EL of a head for the whole of a catch-up, where the newest block's request always waits behind its parent's.
  • Bound. At most MAX_PAYLOADS_AWAITING_VERDICT (128) payloads wait. Past that the import is held (a pre-gloas block is written as a pending block; an envelope waits in awaiting_engine) until answers free room.
  • Children of a FULL payload wait for its answer, not just for its envelope. That way nothing is built on an envelope that gets taken back out. Lighthouse (get_parent_import_status) and Prysm (ParentPayloadReady) hold the same blocks, since their FULL node exists only after the answer.
  • Hash mismatch. Since engine_newPayloadV2 a block hash mismatch is answered INVALID with a null latestValidHash (INVALID_BLOCK_HASH is "supplanted by INVALID", execution-apis shanghai.md). Lighthouse (unstable 25b9b75) and Prysm (develop bf0db8c) both reject only the envelope there and condemn nothing. This PR matches that by reverting the applied envelope. The code comment calling this an ethrex quirk is corrected.
  • Inline path kept. Two cases still wait for the answer inline: blocks that are not optimistic candidates (merge transition), and a server with no mailbox (offline replay).

Spec deviation

optimistic-sync.md says a block the EL gave no answer for MUST NOT be applied to fork choice. Here it already is, so it stays optimistic and is asked again each slot. This is documented in docs/spec_deviations.md ("Payloads are applied before the execution client answers"), which replaces the gloas-only entry.

Relation to other PRs

Testing

  • ethlambda-blockchain --lib --features beacon-spec-tests (with consensus-spec-tests v1.7.0-beta.2): 353 passed. New tests cover:
    • the queue: parent ordering, the cap, re-asking, forkchoice_head, the bound;
    • the pre-gloas verdicts (VALID/SYNCING/INVALID/no answer/stale);
    • the hold for room;
    • gloas: applied before the answer with its child held; the child arriving after the envelope but before the answer; the revert on a null-hash INVALID; condemnation on a real-hash INVALID; no answer then a re-ask.
  • The six existing tests that encode "no verdict means not imported" now cover the inline path, unchanged.
  • cargo test --workspace --lib --bins, make lint.
  • Not covered: the actor seam (the #[started] hook that records the actor's reference, and dispatch of NewPayloadAnswered through the real mailbox). Every test feeds answers through a channel. A follower run is the end-to-end check.

New metrics: lean_engine_new_payload_in_flight, lean_engine_payloads_awaiting_verdict, lean_engine_new_payload_answers_total{kind,outcome} (in docs/metrics.md).

The chain actor waited for every engine_newPayload inside the import,
so the head moved only after EL + state transition and the actor sat
idle for the EL round trip, handling nothing else. Gloas envelopes did
the same.

A pre-gloas block that is an optimistic candidate now imports as
optimistic as soon as its consensus checks pass, and a gloas envelope
applies with its payload NOT_VALIDATED. The newPayload question goes out
first, from a tokio task that reports back with a NewPayloadAnswered
message, and the answer is applied then as optimistic-sync.md applies a
verdict. Optimism already gates attestation data, so validators still
attest only to validated payloads.

- Payloads are sent parent-before-child, at most 4 in flight, and
  forkchoiceUpdated never names a payload whose request is queued or in
  flight: the Engine API orders only forkchoiceUpdated, the client's
  connection pool can reorder requests, and a payload whose parent the
  EL has not seen is answered SYNCING and may start a peer sync.
- At most 128 payloads wait for an answer; past that the import is held
  until answers free room, so catch-up cannot run unboundedly ahead.
- No answer after the retry ladder keeps the block optimistic and asks
  again each slot. optimistic-sync.md says such a block MUST NOT be
  applied; it already is, so this is documented as a deviation.
- An envelope answered INVALID with a null latestValidHash (how Engine
  API V2+ reports a block hash mismatch) is taken back out and only that
  copy refused, matching Lighthouse and Prysm, which reject the envelope
  without condemning the payload. Blocks built on a FULL payload wait
  for its answer, as they do in both clients, so nothing is built on an
  envelope that gets taken back out.
- A VALID answer for the current head sends `head` again without the
  optimistic flag, for validator clients that react to head events.

Non-candidate blocks (merge transition) and servers without a mailbox
(offline replay) still wait for the answer inline.
Skipping forkchoiceUpdated while the head's newPayload was queued or in
flight starved the execution client of a head for the whole of a
catch-up: parent ordering sends a linear chain one payload at a time,
so the newest imported block, the head, always waits behind its
parent's request. The call now names the newest payload on the head's
chain the client has been asked about, walking the waiting payloads'
parent hashes, and is sent again when the head's own answer arrives.

Also forgets the waiting asks of payloads a condemned gloas payload
took with it, names the limits by their constants in the docs, lists
the new metrics in docs/metrics.md, and covers a child that arrives
after its parent's envelope is applied but before the answer.
MegaRedHand added a commit that referenced this pull request Nov 5, 2026
…rs (#773)

The chain actor waited for every engine_newPayload inside the import,
so the head moved only after EL + state transition and the actor sat
idle for the EL round trip, handling nothing else. Gloas envelopes did
the same.

A pre-gloas block that is an optimistic candidate now imports as
optimistic as soon as its consensus checks pass, and a gloas envelope
applies with its payload NOT_VALIDATED. The newPayload question goes out
first, from a tokio task that reports back with a NewPayloadAnswered
message, and the answer is applied then as optimistic-sync.md applies a
verdict. Optimism already gates attestation data, so validators still
attest only to validated payloads.

- Payloads are sent parent-before-child, at most 4 in flight, and
  forkchoiceUpdated names the newest payload on the head's chain whose
  request is not queued or in flight (walking the waiting payloads'
  parent hashes), sent again when the head's own answer arrives, so a
  catch-up still feeds the EL a head.
- At most 128 payloads wait for an answer; past that the import is held
  until answers free room, so catch-up cannot run unboundedly ahead.
- No answer after the retry ladder keeps the block optimistic and asks
  again each slot. optimistic-sync.md says such a block MUST NOT be
  applied; it already is, so this is documented as a deviation.
- An envelope answered INVALID with a null latestValidHash (how Engine
  API V2+ reports a block hash mismatch) is taken back out and only that
  copy refused, matching Lighthouse and Prysm. Blocks built on a FULL
  payload wait for its answer, so nothing is built on an envelope that
  gets taken back out.
- A VALID answer for the current head sends `head` again without the
  optimistic flag, for validator clients that react to head events.

Non-candidate blocks (merge transition) and servers without a mailbox
(offline replay, the import benchmark) still wait for the answer inline.

On nightly, that inline wait is #755's: the request goes out from a task
and the block's state transition runs meanwhile, so the inline path keeps
its overlap, speculation metrics and EngineAnswer telemetry rows, and
#773's own inline helper (ask_payload_inline) is dropped. The engine
section of an import that asks without waiting is a recorded zero. An
applied envelope still publishes #748's execution_payload and
execution_payload_available events, now before its answer. Answers that arrive later write no el_sent/el_verdict
telemetry rows (#765): those come only from an import that waited.

Cherry-picked from perf/async-new-payload at 33d1969 (f23010e,
33d1969).

Claude-Session: https://claude.ai/code/session_011vghNmndKPkZinHRkVTwVB
MegaRedHand added a commit that referenced this pull request Nov 5, 2026
…rs (#773): write the engine rows and attestable when the answer is applied

fix(telemetry): time a block's attestable and engine rows by its answer

With #773 a block that may be imported optimistically no longer waits for
engine_newPayload, so #765's block_events rows lost the execution client:
el_sent and el_verdict came only from an import that waited (none on the
live node since), and attestable, written at the head recomputation after
the cascade, now marked a head the block was optimistic in, which
validators cannot attest to. On beacon-eth4 that cut received-to-attestable
from a 75 ms median to 40 ms without any block becoming attestable sooner,
so the Silver comparison read faster than it was.

- Each pre-gloas ask records the import's source and when its request went
  out; the request task stamps when its answer came back.
- Applying the answer writes el_sent and el_verdict at those instants, and
  attestable at the moment a VALID (or NOT_REQUIRED) answer marks the block
  validated. The head recomputation leaves out the attestable of a block
  whose answer is still out. The answer is handled only after the import
  that asked, head recomputation and forkchoiceUpdated included, so the
  applied moment is always the later one; attestable - el_verdict is the
  time the answer waited in the mailbox.
- No answer writes nothing (the redrive's answer does, with its own
  el_sent); an answer for a block that left fork choice writes nothing; a
  SYNCING/ACCEPTED or INVALID answer writes el_sent and el_verdict only.
  Envelopes write no block rows, as before.

docs/telemetry.md says when each row is written on the asynchronous path.

Claude-Session: https://claude.ai/code/session_011vghNmndKPkZinHRkVTwVB

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant