You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Sep 8, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Sep 8, 2026. It is now read-only.
v0.15.5 Persistent "Send Queue full" in gossipsub despite low peer count and idle bandwidth #1063
node-1 | {"timestamp":"2026-05-11T12:28:41.216035Z","level":"INFO","fields":{"message":"Inserted new unsafe block","hash":"0x7de2d7bec1c2c3895e58b750c74566092909117ae917edadcb31b67600cd4b7a","number":45856587,"total_duration":"121.982748ms","insert_duration":"117.902539ms"},"target":"engine"}
node-1 | {"timestamp":"2026-05-11T12:28:41.216125Z","level":"INFO","fields":{"message":"Safe head unchanged","safe_head":45856557},"target":"engine"}
node-1 | {"timestamp":"2026-05-11T12:28:41.556015Z","level":"WARN","fields":{"message":"Send Queue full. Could not send IHave(IHave { topic_hash: TopicHash { hash: "/optimism/8453/3/blocks" }, message_ids: [MessageId(45e05d477cd987fadb1271a5027861c2a108c194)] }).","peer":"16Uiu2HAkyTBnw64qWhp1szF4gbdapece5TLPKF4rgkbB7Mn5tFHv"},"target":"libp2p_gossipsub::behaviour"}
node-1 | {"timestamp":"2026-05-11T12:28:41.556070Z","level":"WARN","fields":{"message":"Send Queue full. Could not send IHave(IHave { topic_hash: TopicHash { hash: "/optimism/8453/3/blocks" }, message_ids: [MessageId(45e05d477cd987fadb1271a5027861c2a108c194)] }).","peer":"16Uiu2HAm98QUzAaJEedoUM2otzeDdjHTs7cTNNkBcMjmFyT9qmMd"},"target":"libp2p_gossipsub::behaviour"}
node-1 | {"timestamp":"2026-05-11T12:28:42.056912Z","level":"WARN","fields":{"message":"Send Queue full. Could not send IHave(IHave { topic_hash: TopicHash { hash: "/optimism/8453/3/blocks" }, message_ids: [MessageId(45e05d477cd987fadb1271a5027861c2a108c194)] }).","peer":"16Uiu2HAm98QUzAaJEedoUM2otzeDdjHTs7cTNNkBcMjmFyT9qmMd"},"target":"libp2p_gossipsub::behaviour"}
node-1 | {"timestamp":"2026-05-11T12:28:42.557737Z","level":"WARN","fields":{"message":"Send Queue full. Could not send IHave(IHave { topic_hash: TopicHash { hash: "/optimism/8453/3/blocks" }, message_ids: [MessageId(45e05d477cd987fadb1271a5027861c2a108c194)] }).","peer":"16Uiu2HAm98QUzAaJEedoUM2otzeDdjHTs7cTNNkBcMjmFyT9qmMd"},"target":"libp2p_gossipsub::behaviour"}
My base-node is constantly logging WARN messages: Send Queue full. Could not send IHave....
The weird part is that my network is not congested (verified via nload, TX is far below capacity), and the number of connected peers is very low (only 1-3 active peers).
I investigated this issue. Here's a technical analysis:
Root Cause
This is a libp2p_gossipsub per-peer send queue issue, not a network congestion problem. The warning Send Queue full. Could not send IHave is emitted by the gossipsub behaviour when the outbound buffer for a specific peer is full.
Key factors:
Low peer count (1–3 active peers) — Every IHave message for every block must go to the same few peers. With more peers, the gossip load is distributed across more connections.
High block production rate — Base produces blocks roughly every 2 seconds. Each block generates at least one IHave announcement on the /optimism/8453/3/blocks topic. With 1–3 peers, each peer receives every IHave.
The per-peer send queue fills up — The default libp2p_gossipsub send queue size (~512 messages per peer) is adequate when there are many peers, but with only 1–3 peers, the queue fills faster than the peer can drain it (especially if the peer has high latency or its own backpressure).
Increase max outbound peers — The current --max-outbound-peers=100 targets 100 peers. If you're only connected to 1–3, check if your P2P port (9222 TCP/UDP) is reachable from outside. Use a port checking tool like [yougetsignal.com](https://www.yougetsignal.com). If the port is blocked, other nodes can't connect to you.
Add more bootstrap peers — The current bootnodes in .env.mainnet might not all be responsive. Consider adding more Base node bootnodes.
Restart the node — Sometimes the gossipsub mesh gets into a degraded state and a restart rebuilds the mesh with fresh connections.
Long-term Fix
The actual gossipsub send queue size is configured in the libp2p_gossipsub config, which is compiled into base-reth-node. A proper fix would be either:
Adding a CLI flag to control the gossipsub send queue size in base/base
Increasing the default queue size upstream
This is not a base/node Docker configuration issue — it's a reth/libp2p configuration parameter that needs to be exposed or increased in the base/base codebase.
NOTE: The node type that was chosen when first running a node cannot be changed after the initial sync. Turning Archive into Pruned, or Pruned into Full is not supported source.
Thank you for your reply. I have investigated and confirmed that it is not a bandwidth issue. The configuration also appears to be following the official recommended settings."
Could you please help me check if there are any deficiencies in the configuration?
about "Long-term Fix
The actual gossipsub send queue size is configured in the libp2p_gossipsub config, which is compiled into base-reth-node. A proper fix would be either:
Adding a CLI flag to control the gossipsub send queue size in base/base
Increasing the default queue size upstream
This is not a base/node Docker configuration issue — it's a reth/libp2p configuration parameter that needs to be exposed or increased in the base/base codebase."
Does this require waiting for an official fix? The new architecture seems less stable than the old geth architecture and consumes higher I/O. At this point, we have not seen any advantages of the new architecture over the old one
Thanks for the detailed report! The root cause of the Send Queue full warnings is indeed the low peer count (net_peerCount returning 0x1). With only a single peer, all IHave messages are routed through the same connection, causing the per-peer send buffer to fill up immediately.
You don't need to wait for an upstream fix. Here’s how you can resolve it right now:
1. Open P2P Ports in AWS Security Group
Port 9222 (TCP & UDP) must be open for inbound traffic so other nodes can discover and connect to you:
Go to AWS Console → EC2 → Security Groups → Inbound Rules
Add the following rules:
Custom TCP, Port 9222, Source 0.0.0.0/0
Custom UDP, Port 9222, Source 0.0.0.0/0
2. (Optional) Switch Back to Geth
If you prefer stability while troubleshooting, you can immediately fall back to geth:
CLIENT=geth docker compose up --build
3. Increase the Peer Limit
Once port 9222 is open, you can increase the maximum number of outbound peers:
RETH_MAX_OUTBOUND_PEERS=250 docker compose up --build
Note: The Send Queue full warnings are non-fatal (messages are automatically retried), but the underlying issue is simply a lack of active peers. Once your node is properly discoverable and peers connect, the warnings will disappear.
Let me know if you run into any issues with these steps or need help verifying your peer count afterward!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Activity
Kewe63 commentedon May 11, 2026
Problem Analysis: "Send Queue full" in gossipsub
I investigated this issue. Here's a technical analysis:
Root Cause
This is a
libp2p_gossipsubper-peer send queue issue, not a network congestion problem. The warningSend Queue full. Could not send IHaveis emitted by the gossipsub behaviour when the outbound buffer for a specific peer is full.Key factors:
Low peer count (1–3 active peers) — Every IHave message for every block must go to the same few peers. With more peers, the gossip load is distributed across more connections.
High block production rate — Base produces blocks roughly every 2 seconds. Each block generates at least one IHave announcement on the
/optimism/8453/3/blockstopic. With 1–3 peers, each peer receives every IHave.The per-peer send queue fills up — The default
libp2p_gossipsubsend queue size (~512 messages per peer) is adequate when there are many peers, but with only 1–3 peers, the queue fills faster than the peer can drain it (especially if the peer has high latency or its own backpressure).Verify it's NOT a bandwidth issue
Workarounds (in order of effectiveness)
Increase max outbound peers — The current
--max-outbound-peers=100targets 100 peers. If you're only connected to 1–3, check if your P2P port (9222 TCP/UDP) is reachable from outside. Use a port checking tool like [yougetsignal.com](https://www.yougetsignal.com). If the port is blocked, other nodes can't connect to you.Add more bootstrap peers — The current bootnodes in
.env.mainnetmight not all be responsive. Consider adding more Base node bootnodes.Restart the node — Sometimes the gossipsub mesh gets into a degraded state and a restart rebuilds the mesh with fresh connections.
Long-term Fix
The actual gossipsub send queue size is configured in the
libp2p_gossipsubconfig, which is compiled intobase-reth-node. A proper fix would be either:base/baseThis is not a
base/nodeDocker configuration issue — it's areth/libp2pconfiguration parameter that needs to be exposed or increased in thebase/basecodebase.553589912 commentedon May 11, 2026
netstat -s | egrep 'packets discarded|overflow'
0 incoming packets discarded
/data/app/base/node-0.15.5# curl -s http://localhost:8080 -X POST
-H "Content-Type: application/json"
-d '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}'
{"jsonrpc":"2.0","id":1,"result":"0x1"}root@ec2-jp-wallet-prd-node-base-az1-fix:/data/app/base/node-0.15.5#
data/app/base/node-0.15.5#
/data/app/base/node-0.15.5# python3
Python 3.10.12 (main, Mar 3 2026, 11:56:32) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
cat /data/app/base/node-0.15.5/.env.mainnet | grep -iE "p2p|peer|boot|port"
P2P CONFIGURATION
OP_NODE_P2P_AGENT=base
OP_NODE_P2P_LISTEN_IP=0.0.0.0
OP_NODE_P2P_LISTEN_TCP_PORT=9222
OP_NODE_P2P_LISTEN_UDP_PORT=9222
OP_NODE_P2P_BOOTNODES=enr:-J24QNz9lbrKbN4iSmmjtnr7SjUMk4zB7f1krHZcTZx-JRKZd0kA2gjufUROD6T3sOWDVDnFJRvqBBo62zuF-hYCohOGAYiOoEyEgmlkgnY0gmlwhAPniryHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQKNVFlCxh_B-716tTs-h1vMzZkSs1FTu_OYTNjgufplG4N0Y3CCJAaDdWRwgiQG,enr:-J24QH-f1wt99sfpHy4c0QJM-NfmsIfmlLAMMcgZCUEgKG_BBYFc6FwYgaMJMQN5dsRBJApIok0jFn-9CS842lGpLmqGAYiOoDRAgmlkgnY0gmlwhLhIgb2Hb3BzdGFja4OFQgCJc2VjcDI1NmsxoQJ9FTIv8B9myn1MWaC_2lJ-sMoeCDkusCsk4BYHjjCq04N0Y3CCJAaDdWRwgiQG,enr:-J24QDXyyxvQYsd0yfsN0cRr1lZ1N11zGTplMNlW4xNEc7LkPXh0NAJ9iSOVdRO95GPYAIc6xmyoCCG6_0JxdL3a0zaGAYiOoAjFgmlkgnY0gmlwhAPckbGHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQJwoS7tzwxqXSyFL7g0JM-KWVbgvjfB8JA__T7yY_cYboN0Y3CCJAaDdWRwgiQG,enr:-J24QHmGyBwUZXIcsGYMaUqGGSl4CFdx9Tozu-vQCn5bHIQbR7On7dZbU61vYvfrJr30t0iahSqhc64J46MnUO2JvQaGAYiOoCKKgmlkgnY0gmlwhAPnCzSHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQINc4fSijfbNIiGhcgvwjsjxVFJHUstK9L1T8OTKUjgloN0Y3CCJAaDdWRwgiQG,enr:-J24QG3ypT4xSu0gjb5PABCmVxZqBjVw9ca7pvsI8jl4KATYAnxBmfkaIuEqy9sKvDHKuNCsy57WwK9wTt2aQgcaDDyGAYiOoGAXgmlkgnY0gmlwhDbGmZaHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQIeAK_--tcLEiu7HvoUlbV52MspE0uCocsx1f_rYvRenIN0Y3CCJAaDdWRwgiQG
BASE_NODE_P2P_LISTEN_IP=0.0.0.0
BASE_NODE_P2P_ADVERTISE_TCP_PORT=9222
BASE_NODE_P2P_ADVERTISE_UDP_PORT=9222
BASE_NODE_P2P_BOOTNODES=enr:-J24QNz9lbrKbN4iSmmjtnr7SjUMk4zB7f1krHZcTZx-JRKZd0kA2gjufUROD6T3sOWDVDnFJRvqBBo62zuF-hYCohOGAYiOoEyEgmlkgnY0gmlwhAPniryHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQKNVFlCxh_B-716tTs-h1vMzZkSs1FTu_OYTNjgufplG4N0Y3CCJAaDdWRwgiQG,enr:-J24QH-f1wt99sfpHy4c0QJM-NfmsIfmlLAMMcgZCUEgKG_BBYFc6FwYgaMJMQN5dsRBJApIok0jFn-9CS842lGpLmqGAYiOoDRAgmlkgnY0gmlwhLhIgb2Hb3BzdGFja4OFQgCJc2VjcDI1NmsxoQJ9FTIv8B9myn1MWaC_2lJ-sMoeCDkusCsk4BYHjjCq04N0Y3CCJAaDdWRwgiQG,enr:-J24QDXyyxvQYsd0yfsN0cRr1lZ1N11zGTplMNlW4xNEc7LkPXh0NAJ9iSOVdRO95GPYAIc6xmyoCCG6_0JxdL3a0zaGAYiOoAjFgmlkgnY0gmlwhAPckbGHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQJwoS7tzwxqXSyFL7g0JM-KWVbgvjfB8JA__T7yY_cYboN0Y3CCJAaDdWRwgiQG,enr:-J24QHmGyBwUZXIcsGYMaUqGGSl4CFdx9Tozu-vQCn5bHIQbR7On7dZbU61vYvfrJr30t0iahSqhc64J46MnUO2JvQaGAYiOoCKKgmlkgnY0gmlwhAPnCzSHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQINc4fSijfbNIiGhcgvwjsjxVFJHUstK9L1T8OTKUjgloN0Y3CCJAaDdWRwgiQG,enr:-J24QG3ypT4xSu0gjb5PABCmVxZqBjVw9ca7pvsI8jl4KATYAnxBmfkaIuEqy9sKvDHKuNCsy57WwK9wTt2aQgcaDDyGAYiOoGAXgmlkgnY0gmlwhDbGmZaHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQIeAK_--tcLEiu7HvoUlbV52MspE0uCocsx1f_rYvRenIN0Y3CCJAaDdWRwgiQG
OP_NODE_RPC_PORT=8545
OP_NODE_METRICS_PORT="7300"
BASE_NODE_METRICS_PORT="7300"
OP_GETH_ETH_STATS=nodename:secret@host:port
OP_GETH_BOOTNODES=enode://87a32fd13bd596b2ffca97020e31aef4ddcc1bbd4b95bb633d16c1329f654f34049ed240a36b449fda5e5225d70fe40bc667f53c304b71f8e68fc9d448690b51@3.231.138.188:30301,enode://ca21ea8f176adb2e229ce2d700830c844af0ea941a1d8152a9513b966fe525e809c3a6c73a2c18a12b74ed6ec4380edf91662778fe0b79f6a591236e49e176f9@184.72.129.189:30301,enode://acf4507a211ba7c1e52cdf4eef62cdc3c32e7c9c47998954f7ba024026f9a6b2150cd3f0b734d9c78e507ab70d59ba61dfe5c45e1078c7ad0775fb251d7735a2@3.220.145.177:30301,enode://8a5a5006159bf079d06a04e5eceab2a1ce6e0f721875b2a9c96905336219dbe14203d38f70f3754686a6324f786c2f9852d8c0dd3adac2d080f4db35efc678c5@3.231.11.52:30301,enode://cdadbe835308ad3557f9a1de8db411da1a260a98f8421d62da90e71da66e55e98aaa8e90aa7ce01b408a54e4bd2253d701218081ded3dbe5efbbc7b41d7cef79@54.198.153.150:30301
OP_NETHERMIND_BOOTNODES=enode://87a32fd13bd596b2ffca97020e31aef4ddcc1bbd4b95bb633d16c1329f654f34049ed240a36b449fda5e5225d70fe40bc667f53c304b71f8e68fc9d448690b51@3.231.138.188:30301,enode://ca21ea8f176adb2e229ce2d700830c844af0ea941a1d8152a9513b966fe525e809c3a6c73a2c18a12b74ed6ec4380edf91662778fe0b79f6a591236e49e176f9@184.72.129.189:30301,enode://acf4507a211ba7c1e52cdf4eef62cdc3c32e7c9c47998954f7ba024026f9a6b2150cd3f0b734d9c78e507ab70d59ba61dfe5c45e1078c7ad0775fb251d7735a2@3.220.145.177:30301,enode://8a5a5006159bf079d06a04e5eceab2a1ce6e0f721875b2a9c96905336219dbe14203d38f70f3754686a6324f786c2f9852d8c0dd3adac2d080f4db35efc678c5@3.231.11.52:30301,enode://cdadbe835308ad3557f9a1de8db411da1a260a98f8421d62da90e71da66e55e98aaa8e90aa7ce01b408a54e4bd2253d701218081ded3dbe5efbbc7b41d7cef79@54.198.153.150:30301
NOTE: The node type that was chosen when first running a node cannot be changed after the initial sync. Turning Archive into Pruned, or Pruned into Full is not supported source.
root@ : ss -tlnup | grep 9222
udp UNCONN 0 0 0.0.0.0:9222 0.0.0.0:* users:(("dockerd",pid=183305,fd=41))
tcp LISTEN 0 100001 0.0.0.0:9222 0.0.0.0:* users:(("dockerd",pid=183305,fd=40))
Thank you for your reply. I have investigated and confirmed that it is not a bandwidth issue. The configuration also appears to be following the official recommended settings."
Could you please help me check if there are any deficiencies in the configuration?
553589912 commentedon May 11, 2026
about "Long-term Fix
The actual gossipsub send queue size is configured in the libp2p_gossipsub config, which is compiled into base-reth-node. A proper fix would be either:
Adding a CLI flag to control the gossipsub send queue size in base/base
Increasing the default queue size upstream
This is not a base/node Docker configuration issue — it's a reth/libp2p configuration parameter that needs to be exposed or increased in the base/base codebase."
Does this require waiting for an official fix? The new architecture seems less stable than the old geth architecture and consumes higher I/O. At this point, we have not seen any advantages of the new architecture over the old one
Kewe63 commentedon May 11, 2026
Thanks for the detailed report! The root cause of the
Send Queue fullwarnings is indeed the low peer count (net_peerCountreturning0x1). With only a single peer, allIHavemessages are routed through the same connection, causing the per-peer send buffer to fill up immediately.You don't need to wait for an upstream fix. Here’s how you can resolve it right now:
1. Open P2P Ports in AWS Security Group
Port
9222(TCP & UDP) must be open for inbound traffic so other nodes can discover and connect to you:Custom TCP, Port9222, Source0.0.0.0/0Custom UDP, Port9222, Source0.0.0.0/02. (Optional) Switch Back to Geth
If you prefer stability while troubleshooting, you can immediately fall back to
geth:3. Increase the Peer Limit
Once port
9222is open, you can increase the maximum number of outbound peers:Note: The
Send Queue fullwarnings are non-fatal (messages are automatically retried), but the underlying issue is simply a lack of active peers. Once your node is properly discoverable and peers connect, the warnings will disappear.Let me know if you run into any issues with these steps or need help verifying your peer count afterward!
feat: make max-outbound-peers configurable via MAX_OUTBOUND_PEERS env…
wordc56 commentedon May 25, 2026
Yeah, gossipsub queue keeps filling up
feat: make max-outbound-peers configurable via MAX_OUTBOUND_PEERS env…
github-actions commentedon Jun 10, 2026
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
github-actions commentedon Jun 15, 2026
This issue was closed because it has been inactive for 5 days since being marked as stale.