Skip to content

feat: Add the flag-overrides capability to the contract test service - #225

Open
kinyoklion wants to merge 1 commit into
rlamb/overrides-java-file-override-sourcefrom
rlamb/overrides-java-contract-tests
Open

kinyoklion wants to merge 1 commit into
rlamb/overrides-java-file-override-sourcefrom
rlamb/overrides-java-contract-tests

Conversation

@kinyoklion

@kinyoklion kinyoklion commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Update, 2026-11-13

  • Rebased onto feat/overrides, which now includes feat: Add the overrideAffected indicator to EvaluationReason #218 and feat: Count override-affected evaluations separately and send no individual events for them #219 and has main merged into it. From main it gets the 7.18.0 release, CI on the latest v3 contract tests, flag change listener support in the contract test service, and the java-sdk-internal 1.12.1 and 1.12.2 fixes. A common 2.6.0 and internal 1.13.0 release from this branch needs those fixes.
  • No changes of its own. With main merged into feat/overrides, the contract test service also has main's flag change listener support. The flag overrides suite, both flag listener suites (including the override change notifications) and the full v3 suite pass against it at the top of the stack. See the verification below.
  • Verified at the top of the stack, with common and internal published locally from feat/overrides:

Summary

The contract test service now advertises the flag-overrides capability and accepts the overrides object in the SDK configuration, as the test harness service specification describes: filePaths (required), duplicateKeysHandling (fail or ignore), changeDetection (polling or watching), and pollIntervalMs. The block configures FileOverrides.source() on the data system builder. An overrides block without a dataSystem block is rejected, because the override source is an option of the FDv2 data system.

The service also reads its listening port from the PORT environment variable when set, which the Makefile already passes, so several test services can run side by side on one machine. The default stays 8000.

This PR depends on the file-based override source (rlamb/overrides-java-file-override-source) and is based on that branch; retarget it to feat/overrides once that branch merges.

The existing file data source keeps its current behavior; this change does not touch its code paths.

SDK-3246


Note

Overview
The contract test service now supports the harness flag-overrides capability so conformance can drive file-based flag overrides on FDv2 data systems.

Configuration parsing adds an optional overrides block on SDK config (filePaths, duplicateKeysHandling, changeDetection, pollIntervalMs), mapped through FileOverrides.source() onto the data system builder. overrides without dataSystem is rejected with a clear error, matching the requirement that overrides are an FDv2 data-system option.

The service advertises flag-overrides in its status capabilities list. It also honors the PORT environment variable for its listen address (default 8000), so multiple instances can run in parallel during local or CI runs.

Reviewed by Cursor Bugbot for commit 2a32216. Bugbot is set up for automated code reviews on this repo. Configure here.

Opened from termag

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from bc2e4a9 to 657a598 Compare September 28, 2026 21:04
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-contract-tests branch from 5069a69 to dd3942a Compare September 28, 2026 21:04
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 657a598 to 5be8948 Compare October 1, 2026 23:44
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-contract-tests branch from dd3942a to 56efaf2 Compare October 1, 2026 23:44
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 5be8948 to 9992774 Compare October 3, 2026 00:58
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-contract-tests branch from 56efaf2 to c355fd9 Compare October 3, 2026 00:58
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 9992774 to 1728c29 Compare October 3, 2026 01:43
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-contract-tests branch from c355fd9 to 310a4a1 Compare October 3, 2026 01:43
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 1728c29 to c134bcc Compare October 13, 2026 21:02
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-contract-tests branch 2 times, most recently from 712cb2f to 048b678 Compare October 15, 2026 22:24
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from a0f4c55 to 21f7364 Compare October 16, 2026 21:18
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-contract-tests branch from 048b678 to 948f0bc Compare October 16, 2026 21:18
kinyoklion added a commit that referenced this pull request Oct 23, 2026
…240)

## Summary

The Java contract test service did not declare the
`flag-change-listeners` and `flag-value-change-listeners` capabilities,
so the test harness skipped its flag change listener suite for Java. The
suite now also covers dependent-flag notifications (sdk-test-harness
#514) and override change notifications (#515), and Java should be
checked by all of it.

The service now declares both capabilities and handles the three
listener commands. `ListenerRegistry` keeps the listeners the harness
has registered with one SDK client, keyed by the harness's listener ID.
A general flag change listener is registered with
`FlagTracker.addFlagChangeListener` and posts the flag key of every
change event to its callback URI. A flag value change listener uses
`FlagTracker.addFlagValueChangeListener` for the given flag and context
and posts the old and new values when the SDK reports a value change.
`unregisterListener` removes the listener and returns 400 for an unknown
ID; closing the client removes them all; re-registering an ID replaces
the earlier listener. Notifications are posted through the existing
`HookCallbackService`, and a failed post is logged rather than thrown,
since the SDK calls listeners from its worker thread.

The Java API has no application default value for a value change
listener, so the `defaultValue` the harness sends is accepted and not
used; the harness tests only compare the values of flags that exist.

## Verification

Run locally: the released harness listener suite passes against this
service, and so does the dependent-flag suite from sdk-test-harness #514
(prerequisite chains, segment and nested segment references, a segment
behind a prerequisite, deletions, and full data sets). With this commit
cherry-picked onto the flag overrides stack (#225), the override change
notification suite from sdk-test-harness #515 passes as well. CI will
run the released suite once the capabilities are declared.

[SDK-3355](https://launchdarkly.atlassian.net/browse/SDK-3355)


[SDK-3355]:
https://launchdarkly.atlassian.net/browse/SDK-3355?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 21f7364 to 68fe0d8 Compare October 28, 2026 22:19
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-contract-tests branch from 948f0bc to 4b626ae Compare October 28, 2026 22:19
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 68fe0d8 to 7f0a3eb Compare October 28, 2026 22:39
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-contract-tests branch 3 times, most recently from 2bdbcd3 to c7d26c1 Compare October 28, 2026 23:05
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 8ccc74b to af78d3a Compare October 29, 2026 16:49
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-contract-tests branch from c7d26c1 to 3fb7c29 Compare October 29, 2026 16:49
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from af78d3a to 2ab2fe7 Compare October 29, 2026 17:01
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-contract-tests branch from 3fb7c29 to d9ec2af Compare October 29, 2026 17:01
@kinyoklion
kinyoklion marked this pull request as ready for review November 2, 2026 23:46
@kinyoklion
kinyoklion requested a review from a team as a code owner November 2, 2026 23:46
kinyoklion added a commit to launchdarkly/sdk-test-harness that referenced this pull request Nov 3, 2026
…thout the init key (#539)

<!-- TERMAG_PR_BODY_BEGIN -->
## Summary

Overrides only add to what the SDK reports: a flag that the override
layer does not hold evaluates as it does without an override source, and
the all-flags state holds every flag it holds without one, plus the
overridden flags. A client that has not initialized is where SDKs broke
this, and only when the store holds flags without being initialized, for
example a persistent store that holds data but no init key. The flag
overrides suite uses the SDK's in-memory store, which is empty until a
data source provides data, so it could not tell a read of the override
layer apart from a read of everything the store returns.

A new `flag overrides` group in the persistent data store suite
(`server_side_persistence_overrides.go`) runs when the test service has
the `flag-overrides` capability. It writes flags to the store without
the init key and gives the client a data source that never provides
data. Each test compares a client with an override source to a client
without one against the same store, so it holds whatever an SDK reports
for an uninitialized store:

- **a flag without an override evaluates as it does without an override
source**: an override file that holds a different flag leaves the
evaluation of a store flag unchanged (OVERRIDE Requirements 1.5.1 and
1.5.3).
- **an empty override source leaves the all-flags state unchanged**
(Requirement 1.2.2).
- **overrides add to the all-flags state and make it valid**: the state
holds exactly the flags of the client without a source, with override
values for the overridden keys, plus the flags that exist only in the
override layer, and it is valid (launchdarkly/sdk-specs#280,
Requirements 1.4.1.2 and 1.4.1.3).

The flag overrides suite gains **evaluate all flags returns only the
override flags as a valid state** in its uninitialized client group,
where the store is empty and every SDK reports only the override flags.

## Results

Run with `-enable-persistence-tests` against Redis.

| Test service | flag without an override | empty override source |
overrides add |
|---|---|---|---|
| Go `v7` | fails: served from the store | fails: valid state of the
store's flags | fails: the store's flags join the overrides |
| Go launchdarkly/go-server-sdk#491 | passes | passes | passes |
| Node `main` | passes | fails: valid state of the store's flags |
passes |
| Node launchdarkly/js-core#2132 first commit (`57498a9e`) | passes |
passes | fails: drops a store flag the state holds without overrides |
| Node launchdarkly/js-core#2132 | passes | passes | passes |
| Python (top of the launchdarkly/python-server-sdk#530 stack, sync and
async) | passes | passes | passes |
| Ruby (top of the launchdarkly/ruby-server-sdk#454 stack) | passes |
passes | passes |

The Java (launchdarkly/java-core#225) and .NET (`dotnet-server-sdk-v9`)
test services do not support persistent stores, so the group skips
there; the new uninitialized client test passes on both. The full Redis
persistent data store suite and the flag overrides suite pass against Go
#491 at `30d4f216` (131 tests run). `golangci-lint` reports no issues.

Go and Node fail these tests until launchdarkly/go-server-sdk#491 and
launchdarkly/js-core#2132 are merged, for test services whose CI runs
the persistent data store suite with this harness version.

[SDK-3419](https://launchdarkly.atlassian.net/browse/SDK-3419)
<!-- TERMAG_PR_BODY_END -->

<!-- termag: box=i-0575d1c54853058d7 project=sdk-test-harness -->
<sub>Opened from termag</sub>


[SDK-3419]:
https://launchdarkly.atlassian.net/browse/SDK-3419?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Adds harness coverage for **file flag overrides** when the SDK never
initializes but a **persistent store already holds feature data without
the `$inited` key**—a case the existing in-memory override suite could
not distinguish from “read everything from the store.”
> 
> A new **`flag overrides`** group under the server-side persistent data
store suite (`server_side_persistence_overrides.go`, gated on
`flag-overrides`) seeds Redis/Consul/Dynamo with flags, uses a blocking
unavailable data source, and compares clients with and without override
files via `uninitializedClient`. Scenarios assert: store flags not in
the override file behave like no override source; an empty override file
leaves **evaluate all flags** unchanged; overrides merge into all-flags
state, mark it **valid**, and apply override values while retaining
other store flags.
> 
> The standalone flag-overrides **`uninitialized client`** group gains
**evaluate all flags returns only the override flags as a valid state**
(empty store, overrides-only valid state). The persistent suite
registers the new group from `server_side_persistence_base.go`.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
09a8e02. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 2ab2fe7 to 43d6c3d Compare November 3, 2026 23:55
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-contract-tests branch from d9ec2af to a2dbdfc Compare November 3, 2026 23:55

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit a2dbdfc. Configure here.

/** "fail" (the default) or "ignore". */
String duplicateKeysHandling;
/** "polling" or "watching"; null means the SDK default. */
String changeDetection;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overrides config field name mismatch

High Severity

The overrides block is deserialized with a changeDetection string, but the test harness protocol documents watch and poll booleans instead. Those properties are ignored, so the source stays on the SDK polling default and watch-mode contract tests never enable watching.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit a2dbdfc. Configure here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reply was written by an AI agent (Claude Code) working on behalf of @kinyoklion.

This one does not apply. The test harness sends the overrides block with these properties: filePaths (array of strings), duplicateKeysHandling (string, fail or ignore), changeDetection (string, polling or watching), and pollIntervalMs (number). There are no watch or poll booleans. See SDKConfigOverridesParams in servicedef/sdk_config.go and the overrides section of docs/service_spec.md in launchdarkly/sdk-test-harness (branch v3). The service's changeDetection string field matches that protocol, and the harness's watch-mode tests send changeDetection: "watching".

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-contract-tests branch from a2dbdfc to 67fff40 Compare November 4, 2026 22:45
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 43d6c3d to b20c3b6 Compare November 4, 2026 22:45
The contract test service advertises the flag-overrides capability and
accepts the overrides object in the SDK configuration: filePaths,
duplicateKeysHandling, changeDetection, and pollIntervalMs. The block
configures FileOverrides.source() on the data system builder. An
overrides block without a dataSystem block is rejected, because the
override source is an option of the FDv2 data system.

The service reads its listening port from the PORT environment variable
when set, which the Makefile already passes, so several test services
can run side by side. The default stays 8000.
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from b20c3b6 to fa9fbbb Compare November 13, 2026 00:14
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-contract-tests branch from 67fff40 to 2a32216 Compare November 13, 2026 00:14
@cursor cursor Bot added the risk:medium label Nov 13, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: medium.

Automated risk review (dry run): Medium

This PR would not be automatically approved. Human approval required.

Why: The contract test service now wires FileOverrides from an overrides config block in SdkClientEntity and Representations, and advertises flag-overrides plus a PORT env listener in TestService. An unresolved Bugbot high-severity finding remains on the changeDetection mapping, so this is not Low.

If you disagree with this review, share this PR in #feedback-agent-approvals or submit a changes-requested review explaining why.

Open in Web View Automation 

Sent by Cursor Approval Agent: Dry-Run PR Approver

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Risk: medium.

Automated risk review (dry run): Medium

This PR would not be automatically approved. Human approval required.

Why: The change wires a new overrides config onto FileOverrides.source() in the contract-test service (SdkClientEntity, Representations, TestService) and leaves an unresolved Bugbot finding on that mapping, so it is above the Low auto-approve threshold.

If you disagree with this review, share this PR in #feedback-agent-approvals or submit a changes-requested review explaining why.

Human review is needed; no reviewers were assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Dry-Run PR Approver

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant