Skip to content

Feat[OBE-13207] - Add bsp (beam-splitter) source - #189

Open
janmejay-s1 wants to merge 4 commits into
masterfrom
OBE-13207__beam_splitter
Open

janmejay-s1 wants to merge 4 commits into
masterfrom
OBE-13207__beam_splitter

Conversation

@janmejay-s1

Copy link
Copy Markdown
Contributor

What

Adds bsp (beam-splitter), a source that works like packet capture for push sources. It listens where a push source would (TCP, UDP, Unix stream or Unix datagram) and streams the raw traffic to control-agent over a new Beam gRPC service (proto/observo/beam/v1/beam.proto). It runs in one of two modes:

  • Update mode (an upstream is configured). Traffic is proxied to the relocated old source as before. A copy goes to control-agent through a lossy queue bounded by bytes. When the queue overflows, the copy records a gap and the proxied traffic is never slowed. Sampling is controlled by upstream.capture.sampling.rate in (0, 1].
  • New mode (no upstream). Traffic is relayed to control-agent losslessly, with backpressure, and control-agent supplies the replies. The listener binds only after control-agent sends Ready. If a session's RPC fails, its client connection is reset. Existing sessions close when control-agent sends Stop.

How it works:

  • Control stream. Carries Hello/Status to control-agent and Ready/Stop back to bsp.
  • Session streams. One per connection or datagram peer. bsp sends Open/Data/HalfClose/Gap, and gets Reply/HalfClose/Stop back.
  • Admission. Sessions are limited by max_sessions, a max_bytes budget (reported through Status{budget_exhausted}) and a token-bucket rate limit.
  • Datagram sessions. Expire after an idle timeout. In update mode each peer gets its own upstream socket, so the replies come back to the right peer.
  • TLS. Terminated in bsp, which also checks permit_origin. tls.client_metadata_key is rejected at build time because bsp emits no events.

Design: https://sentinelone.atlassian.net/wiki/spaces/ENGINEERIN/pages/6677561370

Also, jwt_auth (shared with the vector sink), in two separate commits:

  • Throttled file reloads. File tokens were re-read on every request. They're now re-read only after reload_interval (default 2s, alias reload_interval_secs), measured with a monotonic clock. Failed reads are cached for the same interval.
  • Non-blocking reload. One caller re-reads the file under a try-lock. Other callers get the last read instead of waiting, unless nothing has been read yet.

Five existing jwt_auth tests now set reload_interval: 0 to keep their re-read-on-every-call behavior, and the deserialize test matches File { path, .. }.

Not in this PR: the control-agent side of Beam.

How to Test

  1. Run TMPDIR=/tmp cargo test --no-default-features --features sinks-vector,sources-bsp --lib -- jwt_auth sources::bsp sinks::vector. All 60 tests pass, including 24 new bsp tests and 7 new jwt_auth tests.
  2. cargo check passes both by default and with --features observo. The sinks-vector and sources-bsp features each build on their own too.
  3. Clippy and rustfmt report nothing new in the changed files.

Jira: OBE-13207

🤖 Generated with Claude Code

janmejay-s1 and others added 4 commits November 17, 2026 20:16
Listens where a push source would and splits the traffic: in update mode it
proxies to the relocated old source and tees a lossy copy to control-agent;
in new mode control-agent receives the traffic losslessly and supplies the
replies. Supports TCP (with TLS and permit_origin), UDP, Unix stream and Unix
datagram. Adds the observo.beam.v1 gRPC protocol (Control and Session
streams) and the sources-bsp feature, included in `observo`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
File tokens were re-read on every request. They are now re-read only when
`reload_interval` (default 2s, `reload_interval_secs` alias) has passed since
the last read, measured with a monotonic clock. Failed reads are cached for
the same interval so a missing file doesn't cost a filesystem call per
request. Existing tests set `reload_interval: 0` to keep their re-read-every-
call semantics.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only one caller re-reads the token file once `reload_interval` has passed; it
takes the reload lock with try-lock. Others serve the last read instead of
blocking a worker on the file read, and wait only when nothing has been read
yet. The cached read sits behind its own mutex, held just to copy it out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Several bsp failure paths only logged at debug, so production showed nothing.
They now emit Vector's standard error events, which count
`component_errors_total` like the other socket sources do:

- accept errors: SocketReceiveError
- TLS handshake failures: TcpSocketTlsConnectionError (bare connects from
  load-balancer health checks stay uncounted)
- UDP datagrams outside `permit_origin`: IpAllowlistDeniedError
- datagram receive errors and failed sends (forwarding upstream, relaying
  replies): SocketReceiveError / SocketSendError

Two failures have no standard event, so new bsp events cover them:

- BspControlStreamError: control stream open failures, failures, ends
  without Stop and token rejections, also counted as
  `bsp_control_stream_failures_total{reason}`
- BspCaptureSessionError: a failed Session RPC in either mode, counted where
  the RPC fails as `bsp_capture_sessions_failed_total`

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant