Skip to content

loadbalancers: register health checks on LB creation - #1787

Open
deanrobertcook wants to merge 2 commits into
mainfrom
dean/6573/register-healthchecks
Open

deanrobertcook wants to merge 2 commits into
mainfrom
dean/6573/register-healthchecks

Conversation

@deanrobertcook

@deanrobertcook deanrobertcook commented Oct 13, 2026 •

Copy link
Copy Markdown

How to categorize this PR?
/kind enhancement

What this PR does / why we need it:
Now that the STACKIT LB supports active health checks, we can register them on LB creation.

Which issue(s) this PR fixes:
Fixes #6573

Special notes for your reviewer:
Cannot be merged before stackitcloud/gardener-extension-provider-stackit#339.

Breaking changes:
Health checks now default to true if not explicitly disabled. This is expected behaviour in K8s, but for customers that depended upon the old behaviour this may lead to lost traffic. These users will be advised then to disable the health checks via release notes

@ske-prow ske-prow Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 13, 2026
@ske-prow

ske-prow Bot commented Oct 13, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ske-prow ske-prow Bot added kind/enhancement Enhancement, improvement, extension size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 13, 2026
@deanrobertcook
deanrobertcook force-pushed the dean/6573/register-healthchecks branch from d2238eb to d41eb2c Compare October 21, 2026 14:02
@ske-prow ske-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 21, 2026
@deanrobertcook
deanrobertcook force-pushed the dean/6573/register-healthchecks branch from d41eb2c to 4144ced Compare October 22, 2026 12:06
@ske-prow ske-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 22, 2026
@deanrobertcook
deanrobertcook force-pushed the dean/6573/register-healthchecks branch 2 times, most recently from 8d98c48 to dd19b28 Compare October 22, 2026 12:11
@deanrobertcook
deanrobertcook marked this pull request as ready for review October 22, 2026 12:12
@ske-prow ske-prow Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 22, 2026
@deanrobertcook
deanrobertcook force-pushed the dean/6573/register-healthchecks branch from dd19b28 to a401c4f Compare October 22, 2026 12:13
@deanrobertcook deanrobertcook changed the title lbs: register health checks on LB creation loadbalancers: register health checks on LB creation Oct 22, 2026
@dergeberl
dergeberl requested review from a team and stackit-ske-bot October 22, 2026 12:23
@deanrobertcook
deanrobertcook force-pushed the dean/6573/register-healthchecks branch from a401c4f to 28e13ba Compare October 22, 2026 12:27
@dergeberl

Copy link
Copy Markdown
Member

/cherry-pick release-v1.34
/cherry-pick release-v1.35
/cherry-pick release-v1.36
/cherry-pick release-v1.37

@stackit-ske

Copy link
Copy Markdown

@dergeberl: once the present PR merges, I will cherry-pick it on top of release-v1.34, release-v1.35, release-v1.36, release-v1.37 in new PRs and assign them to you.

Details

In response to this:

/cherry-pick release-v1.34
/cherry-pick release-v1.35
/cherry-pick release-v1.36
/cherry-pick release-v1.37

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@stackit-ske-bot stackit-ske-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Architectural Feedback

Reconciler handling for externalTrafficPolicy: Local when HealthCheckNodePort is unassigned:

In activeHealthCheckForService, when service.Spec.ExternalTrafficPolicy == corev1.ServiceExternalTrafficPolicyLocal and service.Spec.HealthCheckNodePort <= 0, the probe port falls back to kubeProxyHealthCheckPort (10256).

In Kubernetes, kube-proxy's /healthz endpoint on port 10256 indicates only whether the node-level kube-proxy daemon is alive, returning HTTP 200 on every healthy node. For services with externalTrafficPolicy: Local, kube-proxy intentionally does not forward traffic across nodes and drops packets arriving on nodes without local endpoints. Probing port 10256 causes the load balancer to treat all nodes as healthy, routing client traffic to nodes without backing pods and resulting in connection drops.

To prevent this reconciler flaw, activeHealthCheckForService / lbSpecFromService should return an error if ExternalTrafficPolicy == Local and HealthCheckNodePort <= 0 (or not yet allocated). Returning an error from lbSpecFromService allows controller reconciliation to safely requeue and retry once the Kubernetes service controller has assigned HealthCheckNodePort.

Findings & Feedback

All findings and recommendations have been provided as inline code suggestions above.

Comment thread docs/load-balancer.md Resolved
Comment thread pkg/stackit/config/config.go Resolved
Comment thread pkg/ccm/loadbalancer_spec.go Resolved
Comment thread pkg/ccm/loadbalancer_spec_test.go Resolved
@deanrobertcook
deanrobertcook force-pushed the dean/6573/register-healthchecks branch from 28e13ba to 3057037 Compare October 22, 2026 13:57

@stackit-ske-bot stackit-ske-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Architectural Feedback

Reconciler handling for externalTrafficPolicy: Local when HealthCheckNodePort is unassigned:

In activeHealthCheckForService, when service.Spec.ExternalTrafficPolicy == corev1.ServiceExternalTrafficPolicyLocal and service.Spec.HealthCheckNodePort <= 0, the probe port falls back to kubeProxyHealthCheckPort (10256).

In Kubernetes, kube-proxy's /healthz endpoint on port 10256 indicates only whether the node-level kube-proxy daemon is alive, returning HTTP 200 on every healthy node. For services with externalTrafficPolicy: Local, kube-proxy intentionally does not forward traffic across nodes and drops packets arriving on nodes without local endpoints. Probing port 10256 causes the load balancer to treat all nodes as healthy, routing client traffic to nodes without backing pods and resulting in connection drops.

To prevent this reconciler flaw, activeHealthCheckForService / lbSpecFromService should return an error if ExternalTrafficPolicy == Local and HealthCheckNodePort <= 0 (or not yet allocated). Returning an error from lbSpecFromService allows controller reconciliation to safely requeue and retry once the Kubernetes service controller has assigned HealthCheckNodePort.

Findings & Feedback

All findings and recommendations have been provided as inline code suggestions above.

Comment thread docs/load-balancer.md Resolved
Comment thread pkg/stackit/config/config.go Resolved
Comment thread pkg/ccm/loadbalancer_spec.go Resolved
Comment thread pkg/ccm/loadbalancer_spec_test.go Resolved

@dergeberl dergeberl left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure about the default to false for backwards comparability. The "normal" way should be health checks on kubeproxy port. To not have a immediate breaking change we also could think about feature gates. WDYT?

Comment thread docs/load-balancer.md Resolved
Comment thread pkg/stackit/config/config.go Resolved
@hown3d

hown3d commented Oct 29, 2026

Copy link
Copy Markdown
Member

I would not use a dedicated feature gate here because the config itself is already a "feature gate".
I'd prefer an opt-out instead of opt-in and mark that clearly in the change log of the new release.
If people do not want to use the new feature they have to disable the config option.

@deanrobertcook
deanrobertcook force-pushed the dean/6573/register-healthchecks branch from 3057037 to f7fc1ee Compare October 29, 2026 15:00
@deanrobertcook

deanrobertcook commented Oct 29, 2026 •

Copy link
Copy Markdown
Author

@dergeberl @hown3d I think that should be about right now: opt in with a pointer bool.

I might need to just revisit docs/load-balancer.md again. Is there any other documentation I should update?

@deanrobertcook
deanrobertcook force-pushed the dean/6573/register-healthchecks branch from f7fc1ee to 9dd4d2b Compare November 2, 2026 09:39
Comment thread pkg/stackit/config/config.go Resolved
Comment thread docs/load-balancer.md Resolved
Comment thread docs/load-balancer.md Resolved
Comment thread pkg/stackit/config/config.go Resolved
Comment thread pkg/stackit/config/config.go Resolved
@deanrobertcook
deanrobertcook force-pushed the dean/6573/register-healthchecks branch 3 times, most recently from 65ebc14 to 43364a9 Compare November 3, 2026 06:37

@stackit-ske-bot stackit-ske-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings & Feedback

All findings and recommendations have been provided as inline code suggestions above.

Comment thread docs/load-balancer.md Resolved
Comment thread docs/load-balancer.md Resolved
Comment thread pkg/ccm/loadbalancer_spec.go Resolved
@dergeberl dergeberl added the breaking This PR contains a breaking change and should be listed accordingly in release notes. label Nov 3, 2026
@deanrobertcook
deanrobertcook force-pushed the dean/6573/register-healthchecks branch 2 times, most recently from 9735ba7 to 71085fa Compare November 3, 2026 09:50
@hown3d

hown3d commented Nov 3, 2026

Copy link
Copy Markdown
Member

/lgtm

@ske-prow ske-prow Bot added the lgtm Indicates that a PR is ready to be merged. label Nov 3, 2026
@ske-prow

ske-prow Bot commented Nov 3, 2026

Copy link
Copy Markdown

LGTM label has been added.

DetailsGit tree hash: 308b05e3cb6c3c4682efad85b59ea177ab638e03

@deanrobertcook

Copy link
Copy Markdown
Author

/hold

@ske-prow ske-prow Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 3, 2026
@dergeberl

Copy link
Copy Markdown
Member

/lgtm

@deanrobertcook
deanrobertcook force-pushed the dean/6573/register-healthchecks branch from 71085fa to cfa02e5 Compare November 17, 2026 06:06
@ske-prow ske-prow Bot removed the lgtm Indicates that a PR is ready to be merged. label Nov 17, 2026
@hown3d

hown3d commented Nov 20, 2026

Copy link
Copy Markdown
Member

/lgtm
/approve

@ske-prow ske-prow Bot added the lgtm Indicates that a PR is ready to be merged. label Nov 20, 2026
@ske-prow

ske-prow Bot commented Nov 20, 2026

Copy link
Copy Markdown

LGTM label has been added.

DetailsGit tree hash: 1ee11ef973ca7d7ea60f39a4f7f53ed2040b16d4

@ske-prow

ske-prow Bot commented Nov 20, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hown3d

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ske-prow ske-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. breaking This PR contains a breaking change and should be listed accordingly in release notes. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. kind/enhancement Enhancement, improvement, extension lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants