Skip to content

Query Frontend: propagate request metadata to pass-through endpoints - #7956

Draft
yeya24 wants to merge 2 commits into
cortexproject:masterfrom
yeya24:fix-query-frontend-request-metadata-passthrough
Draft

yeya24 wants to merge 2 commits into
cortexproject:masterfrom
yeya24:fix-query-frontend-request-metadata-passthrough

Conversation

@yeya24

@yeya24 yeya24 commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

What this PR does:

NewQueryTripperware routes /api/v1/query and /api/v1/query_range through a
roundTripper whose Do rebuilds the outgoing request with the codec and calls
requestmeta.InjectMetadataIntoHTTPRequestHeaders. Every other endpoint it handles
(/api/v1/series, /api/v1/labels, /api/v1/label/{name}/values, /api/v1/metadata,
/api/v1/read, /api/v1/query_exemplars, /api/v1/format_query, /api/v1/parse_query)
falls through to next.RoundTrip(r) with the request as it was received, and nothing
injects the metadata.

api.HTTPHeaderMiddleware.injectRequestContext puts the request ID (generated, or taken
from -api.request-id-header) into the request context only, never into the headers. So
for the pass-through endpoints the querier's ContextWithRequestMetadataMapFromHeaders
finds no x-cortex-request-id, and the querier's own HTTPHeaderMiddleware does not
generate a replacement because a metadata map is already in the context. The gRPC calls
the querier then makes to ingesters and store-gateways carry an empty request ID in
x-request-metadata-propagation-string, so a request to one of these endpoints cannot be
correlated across components. I noticed this on a cluster where a handful of
/api/v1/series requests reached the store-gateways with an empty request ID while every
/api/v1/query and /api/v1/query_range request carried one.

The fix injects the metadata on the pass-through branch, into a clone of the request so
the request owned by the HTTP server is not modified (a RoundTripper should not mutate
the request it is given, and the inbound Header map is shared with it).

It also changes InjectMetadataIntoHTTPRequestHeaders from Header.Add to Header.Set.
frontend_processor.go and scheduler_processor.go build their header map from
h.Values[0], so with Add a value the client sent for the same header shadows the one
the query frontend put in the context. The metadata map is a map[string]string, so no
key it holds is meant to carry more than one value, and roundTripper.Do was the
function's only non-test caller.

One case I deliberately left alone: when both middleware lists are empty
NewQueryTripperware returns next unchanged, so nothing is injected for any endpoint.
That path is unreachable in Cortex's own wiring, because queryrange.Middlewares always
returns at least NewLimitsMiddleware. I can wrap next there too if you would rather
have the guard.

Which issue(s) this PR fixes:
None, found while tracing a request through the read path.

Checklist

  • Tests updated
  • Documentation added — not applicable, no config or user-facing surface changes
  • CHANGELOG.md updated
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags — no new flags

This change was drafted with the help of an AI coding assistant. I have reviewed and
verified every line, the reasoning about the header semantics, and the tests.

NewQueryTripperware sends /api/v1/query and /api/v1/query_range through a
roundTripper that rebuilds the outgoing request with the codec and injects the
request metadata into the headers of the request it builds. Every other
endpoint it handles, /api/v1/series, /api/v1/labels,
/api/v1/label/{name}/values, /api/v1/metadata, /api/v1/read,
/api/v1/query_exemplars, /api/v1/format_query and /api/v1/parse_query, is
forwarded with the request as it was received, and nothing injects the metadata
into its headers.

HTTPHeaderMiddleware puts the request ID in the request context only, so for
those endpoints the querier found no request ID header to recover it from, and
its own HTTPHeaderMiddleware generated no replacement because a metadata map
was already in the context. The gRPC calls the querier made to ingesters and
store-gateways then carried an empty request ID, so a request to one of those
endpoints could not be followed across components.

Inject the metadata on the pass-through branch, into a clone of the request so
that the request owned by the server is left unmodified.

InjectMetadataIntoHTTPRequestHeaders now sets each header instead of appending
to it. The receiving side builds its map from the first value of each header,
in frontend_processor.go and scheduler_processor.go, so a value the client sent
for the same header would otherwise shadow the one the query frontend put in
the request context. The metadata map is a map[string]string, so no key it
holds carries more than one value, and the function has no other caller.

Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant