Skip to content

Commit c74c593

Browse files
authored
orchestrator: recover stalled ticket redemptions (#15)
Retry eligible failed submissions and replace transactions that exceed a configurable inclusion timeout. Persist signed transactions, acknowledged nonces, and replacement history so recovery can resume after a restart. Reconcile older transactions through finality, repair nonce gaps, and retain ticket liability across reorgs and unresolved attempts. Add recovery settings, a database migration, and tests for shared-wallet and failure scenarios.
1 parent 7a1f39e commit c74c593

28 files changed

Lines changed: 3567 additions & 509 deletions

‎cmd/livepeer/testdata/cli/orchestrator-help.golden‎

Lines changed: 34 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -11,38 +11,40 @@ Available Commands:
1111
redemptions Inspect recipient redemption state or explicitly retry stored transaction bytes
1212

1313
Flags:
14-
-c, --config string Configuration file; empty disables discovery (env: LIVEPEER_ORCHESTRATOR_CONFIG) (default "config.toml")
15-
-n, --network string Network name (custom networks require explicit chain settings) (env: LIVEPEER_ORCHESTRATOR_NETWORK) (default "offchain")
16-
-d, --data-dir string Component data directory; defaults to ~/.lpData/<network>/<component> (env: LIVEPEER_ORCHESTRATOR_DATA_DIR) (default "$HOME/.lpData/offchain/orchestrator")
17-
-a, --account string Keystore account address (env: LIVEPEER_ORCHESTRATOR_ACCOUNT)
18-
-o, --orchestrator-address string Registered orchestrator address receiving ticket payments; defaults to the keystore account (env: LIVEPEER_ORCHESTRATOR_ORCHESTRATOR_ADDRESS)
19-
-e, --eth-usd-feed string ETH/USD oracle address, alternative to a fixed wei-per-usd rate (env: LIVEPEER_ORCHESTRATOR_ETH_USD_FEED)
20-
-p, --price-max-age duration Maximum age of the oracle observation (env: LIVEPEER_ORCHESTRATOR_PRICE_MAX_AGE) (default 2h0m0s)
21-
-l, --listen string Public HTTP listener (env: LIVEPEER_ORCHESTRATOR_LISTEN) (default "127.0.0.1:8935")
22-
-m, --metrics-listen string Loopback metrics listener (env: LIVEPEER_ORCHESTRATOR_METRICS_LISTEN) (default "127.0.0.1:8936")
23-
-s, --service-url string Public orchestrator base URL; defaults to http://listen (env: LIVEPEER_ORCHESTRATOR_SERVICE_URL)
24-
-r, --runner-service-url string Runner-facing base URL for callbacks and trickle; defaults to service-url (env: LIVEPEER_ORCHESTRATOR_RUNNER_SERVICE_URL)
25-
--proxy-url-template string Generated proxy URL with {proxy} in a hostname label or final path segment (env: LIVEPEER_ORCHESTRATOR_PROXY_URL_TEMPLATE)
26-
--bootstrap-secret-file string File containing runner bootstrap credential (env: LIVEPEER_ORCHESTRATOR_BOOTSTRAP_SECRET_FILE)
27-
--runner-config string Static runner TOML path (env: LIVEPEER_ORCHESTRATOR_RUNNER_CONFIG)
28-
--redeemer-db string (env: LIVEPEER_ORCHESTRATOR_REDEEMER_DB)
29-
-k, --keystore-file string Encrypted geth redemption account JSON file (env: LIVEPEER_ORCHESTRATOR_KEYSTORE_FILE)
30-
--keystore-password-file string Owner-only file containing the exact keystore password bytes (env: LIVEPEER_ORCHESTRATOR_KEYSTORE_PASSWORD_FILE)
31-
--rpc-url-file string (env: LIVEPEER_ORCHESTRATOR_RPC_URL_FILE)
32-
--chain-id uint (env: LIVEPEER_ORCHESTRATOR_CHAIN_ID)
33-
--redeemer-max-fee-per-gas string Optional maximum redemption fee in wei per gas (env: LIVEPEER_ORCHESTRATOR_REDEEMER_MAX_FEE_PER_GAS)
34-
--controller-address string (env: LIVEPEER_ORCHESTRATOR_CONTROLLER_ADDRESS)
35-
-w, --wei-per-usd string (env: LIVEPEER_ORCHESTRATOR_WEI_PER_USD)
36-
-t, --ticket-face-value string (env: LIVEPEER_ORCHESTRATOR_TICKET_FACE_VALUE)
37-
--ticket-win-prob string (env: LIVEPEER_ORCHESTRATOR_TICKET_WIN_PROB)
38-
--runner-grants strings Private runner IP[:port] or CIDR grants (env: LIVEPEER_ORCHESTRATOR_RUNNER_GRANTS)
39-
--session-proxy-grants strings Private session proxy IP[:port] or CIDR grants (env: LIVEPEER_ORCHESTRATOR_SESSION_PROXY_GRANTS)
40-
--health-grants strings Private static runner health IP[:port] or CIDR grants (env: LIVEPEER_ORCHESTRATOR_HEALTH_GRANTS)
41-
--heartbeat-interval duration Runner heartbeat interval (env: LIVEPEER_ORCHESTRATOR_HEARTBEAT_INTERVAL) (default 5s)
42-
--heartbeat-ttl duration Runner heartbeat expiry (env: LIVEPEER_ORCHESTRATOR_HEARTBEAT_TTL) (default 30s)
43-
--print-config Print audited redacted TOML configuration
44-
-h, --help help for livepeer-orchestrator
45-
-v, --version version for livepeer-orchestrator
14+
-c, --config string Configuration file; empty disables discovery (env: LIVEPEER_ORCHESTRATOR_CONFIG) (default "config.toml")
15+
-n, --network string Network name (custom networks require explicit chain settings) (env: LIVEPEER_ORCHESTRATOR_NETWORK) (default "offchain")
16+
-d, --data-dir string Component data directory; defaults to ~/.lpData/<network>/<component> (env: LIVEPEER_ORCHESTRATOR_DATA_DIR) (default "$HOME/.lpData/offchain/orchestrator")
17+
-a, --account string Keystore account address (env: LIVEPEER_ORCHESTRATOR_ACCOUNT)
18+
-o, --orchestrator-address string Registered orchestrator address receiving ticket payments; defaults to the keystore account (env: LIVEPEER_ORCHESTRATOR_ORCHESTRATOR_ADDRESS)
19+
-e, --eth-usd-feed string ETH/USD oracle address, alternative to a fixed wei-per-usd rate (env: LIVEPEER_ORCHESTRATOR_ETH_USD_FEED)
20+
-p, --price-max-age duration Maximum age of the oracle observation (env: LIVEPEER_ORCHESTRATOR_PRICE_MAX_AGE) (default 2h0m0s)
21+
-l, --listen string Public HTTP listener (env: LIVEPEER_ORCHESTRATOR_LISTEN) (default "127.0.0.1:8935")
22+
-m, --metrics-listen string Loopback metrics listener (env: LIVEPEER_ORCHESTRATOR_METRICS_LISTEN) (default "127.0.0.1:8936")
23+
-s, --service-url string Public orchestrator base URL; defaults to http://listen (env: LIVEPEER_ORCHESTRATOR_SERVICE_URL)
24+
-r, --runner-service-url string Runner-facing base URL for callbacks and trickle; defaults to service-url (env: LIVEPEER_ORCHESTRATOR_RUNNER_SERVICE_URL)
25+
--proxy-url-template string Generated proxy URL with {proxy} in a hostname label or final path segment (env: LIVEPEER_ORCHESTRATOR_PROXY_URL_TEMPLATE)
26+
--bootstrap-secret-file string File containing runner bootstrap credential (env: LIVEPEER_ORCHESTRATOR_BOOTSTRAP_SECRET_FILE)
27+
--runner-config string Static runner TOML path (env: LIVEPEER_ORCHESTRATOR_RUNNER_CONFIG)
28+
--redeemer-db string (env: LIVEPEER_ORCHESTRATOR_REDEEMER_DB)
29+
-k, --keystore-file string Encrypted geth redemption account JSON file (env: LIVEPEER_ORCHESTRATOR_KEYSTORE_FILE)
30+
--keystore-password-file string Owner-only file containing the exact keystore password bytes (env: LIVEPEER_ORCHESTRATOR_KEYSTORE_PASSWORD_FILE)
31+
--rpc-url-file string (env: LIVEPEER_ORCHESTRATOR_RPC_URL_FILE)
32+
--chain-id uint (env: LIVEPEER_ORCHESTRATOR_CHAIN_ID)
33+
--redeemer-tx-timeout duration Inclusion timeout for each redemption transaction attempt (env: LIVEPEER_ORCHESTRATOR_REDEEMER_TX_TIMEOUT) (default 5m0s)
34+
--redeemer-max-tx-replacements int Maximum fee replacements per redemption attempt (env: LIVEPEER_ORCHESTRATOR_REDEEMER_MAX_TX_REPLACEMENTS) (default 1)
35+
--redeemer-max-fee-per-gas string Optional maximum redemption fee in wei per gas (env: LIVEPEER_ORCHESTRATOR_REDEEMER_MAX_FEE_PER_GAS)
36+
--controller-address string (env: LIVEPEER_ORCHESTRATOR_CONTROLLER_ADDRESS)
37+
-w, --wei-per-usd string (env: LIVEPEER_ORCHESTRATOR_WEI_PER_USD)
38+
-t, --ticket-face-value string (env: LIVEPEER_ORCHESTRATOR_TICKET_FACE_VALUE)
39+
--ticket-win-prob string (env: LIVEPEER_ORCHESTRATOR_TICKET_WIN_PROB)
40+
--runner-grants strings Private runner IP[:port] or CIDR grants (env: LIVEPEER_ORCHESTRATOR_RUNNER_GRANTS)
41+
--session-proxy-grants strings Private session proxy IP[:port] or CIDR grants (env: LIVEPEER_ORCHESTRATOR_SESSION_PROXY_GRANTS)
42+
--health-grants strings Private static runner health IP[:port] or CIDR grants (env: LIVEPEER_ORCHESTRATOR_HEALTH_GRANTS)
43+
--heartbeat-interval duration Runner heartbeat interval (env: LIVEPEER_ORCHESTRATOR_HEARTBEAT_INTERVAL) (default 5s)
44+
--heartbeat-ttl duration Runner heartbeat expiry (env: LIVEPEER_ORCHESTRATOR_HEARTBEAT_TTL) (default 30s)
45+
--print-config Print audited redacted TOML configuration
46+
-h, --help help for livepeer-orchestrator
47+
-v, --version version for livepeer-orchestrator
4648

4749
Use "livepeer-orchestrator [command] --help" for more information about a command.
4850

‎configs/orchestrator/config.example.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,8 @@ HealthGrants = []
2929
# Both files must be owner-only; the password is read as exact bytes (no newline).
3030
# Or use LIVEPEER_ORCHESTRATOR_KEYSTORE_PASSWORD and omit KeystorePasswordFile.
3131
# RPCURLFile = "/run/secrets/livepeer-payment-rpc-url"
32+
# RedeemerTxTimeout = "5m" # Inclusion timeout per transaction attempt.
33+
# RedeemerMaxTxReplacements = 1 # Same-nonce fee replacements before retrying the ticket.
3234
# RedeemerMaxFeePerGas = "1000000000" # Optional ceiling, wei per gas
3335
# ChainID = 42161
3436
# Controller = "0xD8E8328501E9645d16Cf49539efC04f734606ee4" # Arbitrum default.

‎docs/configuration.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -113,9 +113,9 @@ or a new chain invocation.
113113
The signer account is the payer, supplying the TicketBroker deposit and reserve.
114114
The orchestrator's keystore account signs redemptions and pays gas. Ticket payments
115115
credit `OrchestratorAddress`, which defaults to that account. The chain account
116-
must match the keystore account when signing. Keep a redemption key exclusive to
117-
its orchestrator worker; concurrent CLI or other use can conflict with nonce
118-
tracking.
116+
must match the keystore account when signing. Multiple orchestrator processes
117+
may share a redemption key. Failed submissions are retried automatically.
118+
See [transaction recovery](payment-recovery.md).
119119

120120
### Prepare a keystore
121121

‎docs/orchestrator.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -122,8 +122,8 @@ every hour; `PriceMaxAge` must be positive and defaults to two hours. Startup
122122
requires a valid observation. Stale rates suspend new sessions; existing
123123
sessions retain their initial price.
124124

125-
`RedeemerMaxFeePerGas` optionally caps redemption fees in wei per gas. The
126-
cap is per gas, not a total spending budget. Winning tickets wait until payment
125+
See [transaction recovery](payment-recovery.md#orchestrator-transaction-recovery)
126+
for timeouts, replacements and fee settings. Winning tickets wait until payment
127127
parameters expire before redemption exposes their randomness on-chain. Only a
128128
successful, canonical, finalized receipt settles their liability.
129129

‎docs/payment-recovery.md‎

Lines changed: 38 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -103,27 +103,48 @@ use these states:
103103
| State | Meaning and recovery |
104104
| --- | --- |
105105
| Queued (no attempt) | Wait for parameter expiry. Preparation failures leave the ticket eligible for another attempt. |
106-
| `prepared` | Signed bytes, nonce and locally computed hash are durable; the worker resumes the first send after restart. |
107-
| `broadcast` | A send may have reached RPC. Reconcile receipts automatically; an operator may explicitly resend identical bytes. |
108-
| `submitted` | RPC acknowledged the hash. Keep checking receipts; explicit identical-byte retry is available if necessary. |
109-
| `confirmed` | Receipt is successful, at or below the finalized head, and matches the canonical block hash. Liability is settled. |
110-
| `reverted` | A canonical finalized receipt failed. Retain the audit row; do not resubmit automatically. |
111-
| `expired` | Ticket creation round is outside the contract validity window. Retain the audit row. |
112-
113-
An uncertain broadcast blocks preparation of a new transaction
114-
for this worker, preventing accidental nonce reuse. The redemption key must be
115-
exclusive to this worker; do not submit concurrent CLI transactions from it.
116-
Only a canonical finalized receipt marks a ticket redeemed. A pending receipt,
117-
reorg, unavailable finalized head or RPC failure retains uncertainty and liability.
106+
| `prepared` | Signed bytes, nonce and hash are durable before sending; restart resumes the saved transaction. |
107+
| `broadcast` | Send in progress or interrupted before recording its outcome; restart resumes the saved transaction. |
108+
| `submitted` | RPC acknowledged the hash; monitor inclusion and replace on timeout. |
109+
| `retryable` | Submission, waiting or replacement failed; retry the ticket automatically. Older hashes remain under receipt reconciliation. |
110+
| `mined` | Canonical inclusion observed; wait for finalization without fee replacement. A reorg resumes recovery. |
111+
| `awaiting_finality` | Latest state says the ticket is used; pause submission until finalized state confirms use or a reorg removes it. |
112+
| `confirmed` / `used` | Successful canonical finalized receipt or finalized ticket-use observation settles liability. |
113+
| `reverted` / `invalid` | Finalized transaction failure or an invalid ticket; stop submitting and reconcile unresolved hashes. |
114+
| `expired` | Ticket creation round is outside its validity window; reconcile unresolved hashes. |
115+
| `superseded` | Finalized inclusion consumed the same sender/nonce, or finalized ticket settlement made this transaction unnecessary. |
116+
117+
Only canonical finalized state settles a ticket. RPC failures, missing receipts,
118+
unfinalized inclusions and reorgs retain uncertainty and liability. Each ticket
119+
counts once even when it has several signed transactions. Previously exposed
120+
recipient randomness stays closed to new payments throughout retries.
121+
122+
`RedeemerTxTimeout` (`--redeemer-tx-timeout`) defaults to five minutes per
123+
inclusion attempt. `RedeemerMaxTxReplacements` (`--redeemer-max-tx-replacements`)
124+
defaults to one; zero disables replacements. Replacements retain the nonce and
125+
calldata and raise both fee caps by geth's default 10% threshold, rounding up
126+
with a strict increase.
127+
Geth estimates initial fees using the RPC header's base fee. A missing base fee
128+
leaves the ticket queued with an error. `RedeemerMaxFeePerGas` optionally caps
129+
initial and replacement fees in wei per gas. Finalization waits do not consume
130+
the inclusion timeout.
131+
132+
Restart restores deadlines, replacement budgets and nonce reservations for
133+
acknowledged sends. Rejected submissions do not reserve a nonce. A rotated key can
134+
resend saved bytes and reconcile receipts, but cannot replace the previous account's transactions.
135+
136+
Migration 002 preserves existing transactions. Its rollback refuses to discard
137+
transactions or metadata that the old schema cannot represent. Retain the upgraded
138+
database for a compatible binary when rollback is refused.
118139

119140
Inspect status without a key or RPC connection:
120141

121142
```sh
122143
livepeer orchestrator redemptions
123144
```
124145

125-
After checking the hash on the configured chain, explicitly retry the stored
126-
transaction if it is absent or needs rebroadcast:
146+
Automatic recovery normally handles rebroadcast. To explicitly resend the
147+
current retryable transaction using its saved bytes and fees:
127148

128149
```sh
129150
livepeer orchestrator redemptions \
@@ -138,9 +159,8 @@ pass the same `--config`, or `--network` and `--data-dir`, as the service.
138159
`--redeemer-db` selects a different database. Recovery reads `RPCURLFile` and `ChainID`
139160
for retries; `--rpc-url-file` and `--chain-id` override them. It does not unlock
140161
the account or read unrelated service credentials. RPC endpoints use normal
141-
system TLS trust. This operation uses the exact saved signed bytes, nonce and
142-
hash; it cannot create a replacement
143-
or change fees. A retry error retains the hash and uncertain state.
162+
system TLS trust. Historical hashes remain inspectable; manual retry cannot
163+
reopen a terminal ticket or displace a newer active transaction.
144164

145165
In-memory control state is bounded: payment sessions have a 100,000-entry quota
146166
and 24-hour inactivity retention; ticket replay guards have a 1,000,000-nonce
@@ -149,11 +169,4 @@ session activity current. Expiry is monotonic within a recipient lifetime so
149169
a backward chain observation cannot revive parameters after their replay guards
150170
are removed. Auth-token refresh never resets nonce tracking. Winning tickets
151171
and uncertain transactions are retained for audit/reconciliation, so back up
152-
and monitor database disk use independently of these quotas.
153-
154-
Redemption planning uses geth's dynamic-fee selection and requires a base fee
155-
in the RPC header; a missing base fee leaves the ticket queued with an error.
156-
`RedeemerMaxFeePerGas` optionally caps the fee in wei per gas; a plan above
157-
that ceiling remains queued without signing or broadcasting. Prepared and
158-
submitted identities reserve their nonces across restarts. Explicit retry
159-
resends the stored transaction without changing its fees.
172+
and monitor database disk use independently of these control-state quotas.

‎eth/contracts.go‎

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -255,6 +255,12 @@ func (c *Contracts) PrepareAndStore(ctx context.Context, plan TransactionPlan, k
255255
}
256256

257257
func (c *Contracts) prepare(ctx context.Context, plan TransactionPlan, key *Key, chainID *big.Int, save func(SignedTransaction) error) (SignedTransaction, error) {
258+
return c.prepareAtNonce(ctx, plan, key, chainID, save, nil)
259+
}
260+
261+
// prepareAtNonce lets the ported redemption backend own nonce allocation through
262+
// submission. The CLI's prepare-and-reserve behavior remains unchanged.
263+
func (c *Contracts) prepareAtNonce(ctx context.Context, plan TransactionPlan, key *Key, chainID *big.Int, save func(SignedTransaction) error, explicit *uint64) (SignedTransaction, error) {
258264
if key == nil || key.Address() != plan.From || chainID == nil || chainID.Sign() <= 0 {
259265
return SignedTransaction{}, errors.New("transaction account does not match signing key")
260266
}
@@ -297,12 +303,15 @@ func (c *Contracts) prepare(ctx context.Context, plan TransactionPlan, key *Key,
297303
state := c.nonceState(plan.From)
298304
state.Lock()
299305
defer state.Unlock()
300-
nonce, err := c.RPC.ethereum.PendingNonceAt(ctx, plan.From)
301-
if err != nil {
302-
return SignedTransaction{}, safeRPCError(err)
303-
}
304-
if nonce < state.next {
305-
nonce = state.next
306+
var nonce uint64
307+
if explicit != nil {
308+
nonce = *explicit
309+
} else {
310+
nonce, err = c.RPC.ethereum.PendingNonceAt(ctx, plan.From)
311+
if err != nil {
312+
return SignedTransaction{}, safeRPCError(err)
313+
}
314+
nonce = max(nonce, state.next)
306315
}
307316
if nonce == math.MaxUint64 {
308317
return SignedTransaction{}, errors.New("transaction account nonce exhausted")
@@ -325,7 +334,9 @@ func (c *Contracts) prepare(ctx context.Context, plan TransactionPlan, key *Key,
325334
}
326335
// Once saved or returned to a caller, this identity can be broadcast even
327336
// if its outcome is uncertain. Never reuse its nonce in this client.
328-
state.next = nonce + 1
337+
if explicit == nil {
338+
state.next = nonce + 1
339+
}
329340
return prepared, nil
330341
}
331342

‎eth/contracts/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Contract bindings
22

33
The ten generated Go files in this directory and `chainlink/` are copied
4-
unchanged from `livepeer/go-livepeer` commit
4+
unchanged from source commit
55
`bd645a09266833fb859053445d9ac85846330756`:
66

77
- `controller.go`
@@ -17,7 +17,7 @@ unchanged from `livepeer/go-livepeer` commit
1717

1818
Their filtered source history is a parent of the binding import commit. It
1919
preserves the authors, committers, dates, and messages, with `Original-Commit`
20-
trailers linking each retained commit to `livepeer/go-livepeer`. Contributors
20+
trailers recording each retained commit's original identity. Contributors
2121
include Eric Tang, Yondon Fu, Nico Vergauwen, Rafał Leszko, Rick Staa, and
2222
Victor Elias. The original MIT notice is in
2323
[`LICENSE`](../../LICENSE).
@@ -27,7 +27,7 @@ copy these files from the selected committed revision, review its contract/API
2727
changes, and update this provenance. For example, from a checkout of this repo:
2828

2929
```sh
30-
git -C /path/to/go-livepeer show REVISION:eth/contracts/bondingManager.go > eth/contracts/bondingManager.go
30+
git -C /path/to/protocol-source show REVISION:eth/contracts/bondingManager.go > eth/contracts/bondingManager.go
3131
```
3232

3333
Application policy lives in the parent `eth` package: canonical snapshots,

0 commit comments

Comments
 (0)