Repository navigation
loadbalancers: register health checks on LB creation - #1787
deanrobertcook wants to merge 2 commits into
Conversation
|
Skipping CI for Draft Pull Request. |
d2238eb to
d41eb2c
Compare
d41eb2c to
4144ced
Compare
8d98c48 to
dd19b28
Compare
dd19b28 to
a401c4f
Compare
a401c4f to
28e13ba
Compare
|
/cherry-pick release-v1.34 |
|
@dergeberl: once the present PR merges, I will cherry-pick it on top of DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
stackit-ske-bot
left a comment
There was a problem hiding this comment.
Architectural Feedback
Reconciler handling for externalTrafficPolicy: Local when HealthCheckNodePort is unassigned:
In activeHealthCheckForService, when service.Spec.ExternalTrafficPolicy == corev1.ServiceExternalTrafficPolicyLocal and service.Spec.HealthCheckNodePort <= 0, the probe port falls back to kubeProxyHealthCheckPort (10256).
In Kubernetes, kube-proxy's /healthz endpoint on port 10256 indicates only whether the node-level kube-proxy daemon is alive, returning HTTP 200 on every healthy node. For services with externalTrafficPolicy: Local, kube-proxy intentionally does not forward traffic across nodes and drops packets arriving on nodes without local endpoints. Probing port 10256 causes the load balancer to treat all nodes as healthy, routing client traffic to nodes without backing pods and resulting in connection drops.
To prevent this reconciler flaw, activeHealthCheckForService / lbSpecFromService should return an error if ExternalTrafficPolicy == Local and HealthCheckNodePort <= 0 (or not yet allocated). Returning an error from lbSpecFromService allows controller reconciliation to safely requeue and retry once the Kubernetes service controller has assigned HealthCheckNodePort.
Findings & Feedback
All findings and recommendations have been provided as inline code suggestions above.
28e13ba to
3057037
Compare
stackit-ske-bot
left a comment
There was a problem hiding this comment.
Architectural Feedback
Reconciler handling for externalTrafficPolicy: Local when HealthCheckNodePort is unassigned:
In activeHealthCheckForService, when service.Spec.ExternalTrafficPolicy == corev1.ServiceExternalTrafficPolicyLocal and service.Spec.HealthCheckNodePort <= 0, the probe port falls back to kubeProxyHealthCheckPort (10256).
In Kubernetes, kube-proxy's /healthz endpoint on port 10256 indicates only whether the node-level kube-proxy daemon is alive, returning HTTP 200 on every healthy node. For services with externalTrafficPolicy: Local, kube-proxy intentionally does not forward traffic across nodes and drops packets arriving on nodes without local endpoints. Probing port 10256 causes the load balancer to treat all nodes as healthy, routing client traffic to nodes without backing pods and resulting in connection drops.
To prevent this reconciler flaw, activeHealthCheckForService / lbSpecFromService should return an error if ExternalTrafficPolicy == Local and HealthCheckNodePort <= 0 (or not yet allocated). Returning an error from lbSpecFromService allows controller reconciliation to safely requeue and retry once the Kubernetes service controller has assigned HealthCheckNodePort.
Findings & Feedback
All findings and recommendations have been provided as inline code suggestions above.
dergeberl
left a comment
There was a problem hiding this comment.
I am not sure about the default to false for backwards comparability. The "normal" way should be health checks on kubeproxy port. To not have a immediate breaking change we also could think about feature gates. WDYT?
|
I would not use a dedicated feature gate here because the config itself is already a "feature gate". |
3057037 to
f7fc1ee
Compare
|
@dergeberl @hown3d I think that should be about right now: opt in with a pointer bool. I might need to just revisit docs/load-balancer.md again. Is there any other documentation I should update? |
f7fc1ee to
9dd4d2b
Compare
65ebc14 to
43364a9
Compare
stackit-ske-bot
left a comment
There was a problem hiding this comment.
Findings & Feedback
All findings and recommendations have been provided as inline code suggestions above.
9735ba7 to
71085fa
Compare
|
/lgtm |
|
LGTM label has been added. DetailsGit tree hash: 308b05e3cb6c3c4682efad85b59ea177ab638e03 |
|
/hold |
|
/lgtm |
71085fa to
cfa02e5
Compare
|
/lgtm |
|
LGTM label has been added. DetailsGit tree hash: 1ee11ef973ca7d7ea60f39a4f7f53ed2040b16d4 |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hown3d The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
How to categorize this PR?
/kind enhancement
What this PR does / why we need it:
Now that the STACKIT LB supports active health checks, we can register them on LB creation.
Which issue(s) this PR fixes:
Fixes #6573
Special notes for your reviewer:
Cannot be merged before stackitcloud/gardener-extension-provider-stackit#339.
Breaking changes:
Health checks now default to true if not explicitly disabled. This is expected behaviour in K8s, but for customers that depended upon the old behaviour this may lead to lost traffic. These users will be advised then to disable the health checks via release notes