Repository navigation
Feat[OBE-13207] - Add bsp (beam-splitter) source - #189
Open
janmejay-s1 wants to merge 4 commits into
Open
janmejay-s1 wants to merge 4 commits into
janmejay-s1 wants to merge 4 commits into
Conversation
Listens where a push source would and splits the traffic: in update mode it proxies to the relocated old source and tees a lossy copy to control-agent; in new mode control-agent receives the traffic losslessly and supplies the replies. Supports TCP (with TLS and permit_origin), UDP, Unix stream and Unix datagram. Adds the observo.beam.v1 gRPC protocol (Control and Session streams) and the sources-bsp feature, included in `observo`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
File tokens were re-read on every request. They are now re-read only when `reload_interval` (default 2s, `reload_interval_secs` alias) has passed since the last read, measured with a monotonic clock. Failed reads are cached for the same interval so a missing file doesn't cost a filesystem call per request. Existing tests set `reload_interval: 0` to keep their re-read-every- call semantics. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only one caller re-reads the token file once `reload_interval` has passed; it takes the reload lock with try-lock. Others serve the last read instead of blocking a worker on the file read, and wait only when nothing has been read yet. The cached read sits behind its own mutex, held just to copy it out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Several bsp failure paths only logged at debug, so production showed nothing.
They now emit Vector's standard error events, which count
`component_errors_total` like the other socket sources do:
- accept errors: SocketReceiveError
- TLS handshake failures: TcpSocketTlsConnectionError (bare connects from
load-balancer health checks stay uncounted)
- UDP datagrams outside `permit_origin`: IpAllowlistDeniedError
- datagram receive errors and failed sends (forwarding upstream, relaying
replies): SocketReceiveError / SocketSendError
Two failures have no standard event, so new bsp events cover them:
- BspControlStreamError: control stream open failures, failures, ends
without Stop and token rejections, also counted as
`bsp_control_stream_failures_total{reason}`
- BspCaptureSessionError: a failed Session RPC in either mode, counted where
the RPC fails as `bsp_capture_sessions_failed_total`
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
bsp(beam-splitter), a source that works like packet capture for push sources. It listens where a push source would (TCP, UDP, Unix stream or Unix datagram) and streams the raw traffic to control-agent over a newBeamgRPC service (proto/observo/beam/v1/beam.proto). It runs in one of two modes:upstreamis configured). Traffic is proxied to the relocated old source as before. A copy goes to control-agent through a lossy queue bounded by bytes. When the queue overflows, the copy records a gap and the proxied traffic is never slowed. Sampling is controlled byupstream.capture.sampling.ratein (0, 1].Ready. If a session's RPC fails, its client connection is reset. Existing sessions close when control-agent sendsStop.How it works:
Hello/Statusto control-agent andReady/Stopback to bsp.Open/Data/HalfClose/Gap, and getsReply/HalfClose/Stopback.max_sessions, amax_bytesbudget (reported throughStatus{budget_exhausted}) and a token-bucket rate limit.permit_origin.tls.client_metadata_keyis rejected at build time because bsp emits no events.Design: https://sentinelone.atlassian.net/wiki/spaces/ENGINEERIN/pages/6677561370
Also,
jwt_auth(shared with thevectorsink), in two separate commits:reload_interval(default 2s, aliasreload_interval_secs), measured with a monotonic clock. Failed reads are cached for the same interval.Five existing
jwt_authtests now setreload_interval: 0to keep their re-read-on-every-call behavior, and the deserialize test matchesFile { path, .. }.Not in this PR: the control-agent side of
Beam.How to Test
TMPDIR=/tmp cargo test --no-default-features --features sinks-vector,sources-bsp --lib -- jwt_auth sources::bsp sinks::vector. All 60 tests pass, including 24 newbsptests and 7 newjwt_authtests.cargo checkpasses both by default and with--features observo. Thesinks-vectorandsources-bspfeatures each build on their own too.Jira: OBE-13207
🤖 Generated with Claude Code