Skip to content

Fix - Expose Splunk HEC sink sent-bytes metrics from sink start - #187

Open
nafiul-s1 wants to merge 1 commit into
masterfrom
fix_hec_sink_sent_bytes_metrics_from_start
Open

nafiul-s1 wants to merge 1 commit into
masterfrom
fix_hec_sink_sent_bytes_metrics_from_start

Conversation

@nafiul-s1

Copy link
Copy Markdown

A sink showed no component_sent_* series until its first successful request, and component_sent_bytes_total was deleted again after expire_metrics_secs of idle time, because emit!(EndpointBytesSent) creates the counter and drops the handle on every call. Sources do not have this problem: their EventsSent handle is registered at build and held. The manager's sink "no data" alert reads
vector_component_sent_bytes_total, so it could not fire for a sink that had never sent anything.

  • BytesSent gets an optional endpoint label (like EventsSent's optional output). Existing BytesSent::from(Protocol::X) users get endpoint: None and produce the same series as before.
  • Driver gets .endpoint(..) next to .protocol(..); it already registers and holds BytesSent at start when a protocol is given.
  • The Splunk HEC logs and metrics sinks compute (protocol, endpoint) once at build (bytes_sent_labels, built the same way a request builds them) and pass them to the driver, so the driver holds the exact series that emit!(EndpointBytesSent) counts into.
  • Driver also registers and holds the untagged component_sent_events_total and component_sent_event_bytes_total series at start.

Result: the HEC sink exposes component_sent_bytes_total{protocol,endpoint} and the events counters at 0 from startup, and they are not expired while the sink runs. emit!(EndpointBytesSent) still does the counting.

Known limits: a config reload while the sink was idle past expire_metrics_secs still resets the counters (the old component is shut down before the new one is built); other sinks are unchanged.

Tested: cargo test --lib sinks::splunk_hec (105 passed), and manual runs with a fake HEC endpoint (nothing sent, 5 events then idle past expiry).

A sink showed no `component_sent_*` series until its first successful
request, and `component_sent_bytes_total` was deleted again after
`expire_metrics_secs` of idle time, because `emit!(EndpointBytesSent)`
creates the counter and drops the handle on every call. Sources do not
have this problem: their `EventsSent` handle is registered at build and
held. The manager's sink "no data" alert reads
`vector_component_sent_bytes_total`, so it could not fire for a sink that
had never sent anything.

- `BytesSent` gets an optional `endpoint` label (like `EventsSent`'s
  optional `output`). Existing `BytesSent::from(Protocol::X)` users get
  `endpoint: None` and produce the same series as before.
- `Driver` gets `.endpoint(..)` next to `.protocol(..)`; it already
  registers and holds `BytesSent` at start when a protocol is given.
- The Splunk HEC logs and metrics sinks compute `(protocol, endpoint)`
  once at build (`bytes_sent_labels`, built the same way a request builds
  them) and pass them to the driver, so the driver holds the exact series
  that `emit!(EndpointBytesSent)` counts into.
- `Driver` also registers and holds the untagged `component_sent_events_total`
  and `component_sent_event_bytes_total` series at start.

Result: the HEC sink exposes `component_sent_bytes_total{protocol,endpoint}`
and the events counters at 0 from startup, and they are not expired while
the sink runs. `emit!(EndpointBytesSent)` still does the counting.

Known limits: a config reload while the sink was idle past
`expire_metrics_secs` still resets the counters (the old component is shut
down before the new one is built); other sinks are unchanged.

Tested: `cargo test --lib sinks::splunk_hec` (105 passed), and manual runs
with a fake HEC endpoint (nothing sent, 5 events then idle past expiry).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants