Repository navigation
feat: replace permanent event-delivery disable with a bounded cooldown - #256
Conversation
…sifier DefaultEventSender now classifies HTTP failures with HttpErrors.classifyHttpFailure instead of the deprecated isHttpErrorRecoverable/checkIfErrorIsRecoverableAndLog helpers. Behavior and log output are unchanged: the set of statuses treated as unexpected is identical, unexpected failures still skip the retry and still log at error level with the same wording, and transport-level exceptions keep their warn-and-retry-once handling.
…n (SDK-3375) On an unexpected delivery failure (a non-recoverable HTTP status such as 401 or 403), the event processor now enters a bounded, self-recovering cooldown window instead of permanently disabling delivery for the lifetime of the SDK instance, per the EVTPROC spec's Section 1.8. Windows double from 5 minutes to a 1 hour ceiling, one step per failed probe. In-window flushes of any kind are no-ops that dispose of the batch exactly as a sent-and-refused payload would be, without the network or serialization cost. Recording is never gated, retry state resets after two consecutive successful analytics deliveries, and no cooldown state persists across restarts. Diagnostic sends are not gated by the window; an unexpected diagnostic failure opens one, as it previously disabled delivery, but diagnostic successes do not drive the delivery schedule. EventSender.Result gains the unexpected classification with factory methods and the isUnexpectedFailure accessor; the legacy constructor and isMustShutDown are deprecated and still dual-written for consumers that have not migrated.
Diagnostics obtain their delivery token through the same atomic cooldown-check-plus-issue call that analytics flushes use, so a delivery dispatched inside a window carries no token at all and every pre-window token goes stale when the window opens. All three in-cooldown guards are therefore uniformly defensive; one of them previously claimed to be reachable from in-window diagnostic failures.
MockEventSender enqueues the captured payload before it reads the result field, so a test that awaits the startup diagnostic and then immediately changes the sender's result can hand the new result to the still-running diagnostic task. The startup diagnostic then fails unexpectedly and opens the cooldown window itself, the test's own window-opening flush is discarded in-window, and the test times out waiting for it - the fake clock never advances, so the accidental window never elapses. Reproduced deterministically by widening the descheduling gap; seen once in CI. The three tests that change the result right after awaiting the startup diagnostic now wait for the diagnostic task to finish first, using the single-threaded executor barrier the file already uses elsewhere.
beekld
left a comment
There was a problem hiding this comment.
AI review: The description says the Android SDK's own DirectEventProcessor reads the legacy flag and keeps its permanent disable unchanged. Android's main branch has no DirectEventProcessor; it exists only on unmerged branches. Android main builds this module's DefaultEventSender and DefaultEventProcessor directly (ComponentsImpl.java), pinned at internal 1.9.0. So Android's next bump of this dependency picks up the cooldown and the ungated diagnostics without a coordinated change.
Relatedly, nothing in either repo reads the dual-written legacy flag. EventSenderWrapper only writes it, so it isn't a consumer that has not migrated.
|
AI review comment: Mirror work pending from the Go PR's human review (launchdarkly/go-sdk-events#68, threads by beekld). The Go side landed these in ff661e0 and fcf7d6f; this PR needs the equivalents before merge. Parking the checklist here rather than pushing now.
|
…fixes (SDK-3375) Diagnostics are now fully out of scope for the event-delivery cooldown, matching the Go implementation. A diagnostic delivery no longer mints a window token or feeds the tracker; it only calibrates the clock from the Date header as before. This removes a blackout mode a reviewer surfaced: a proxy that accepts /bulk but rejects /diagnostic would otherwise open a new window on every periodic diagnostic and discard analytics the service would accept. Backpressure on the diagnostic endpoint itself belongs to the DIAG spec (SDK-2777), not here. Mirrors the fixes made on the Go PR's human review (launchdarkly/go-sdk-events#68): - When a window opens, forget every context key so post-window payloads re-introduce contexts whose index/identify events were lost with the refused batch. The window opens on a flush worker thread while the deduplicator is only safe to touch from the dispatcher thread, so the clear is routed through the inbox as a FLUSH_USERS message. - A delivery failure now breaks the consecutive-success streak even when its token is stale, so success/failure/success no longer resets the ladder across a failure (RETRY 1.8.3). Stale outcomes still cannot open, step, or end a window; unstamped (null) tokens remain fully inert. - Comment trims, and the EventSender.failure() javadoc no longer claims it does not affect the delivery schedule. Tests updated for the window-open wipe: the three diagnostic-coupling tests become decoupling tests (the straddle case is gone with the token), and the dedup tests assert via index-event presence rather than exact flush counts.
…wedge close() (SDK-3375) A flush worker caught only Exception around its send and response handling, so an Error (StackOverflowError, NoClassDefFoundError, a test AssertionError, etc.) escaping the loop body skipped the busy-worker-count decrement. The count stayed elevated and close() blocked forever in waitUntilAllFlushWorkersInactive. Moving the decrement into a finally guarantees it runs even when an Error ends the worker thread. Pre-existing on main; surfaced in review of this change. Adds errorFromEventSenderDoesNotWedgeClose, which times out without the fix.
… failure (SDK-3375) Two review findings from Cursor Bugbot: - The transport-failure warning in DefaultEventSender logged e.toString(), which can include hostnames or URLs. It now logs LogValues.exceptionSummary(e), matching HttpErrors' own transport-failure helper and this module's other exception logging. - A custom EventSender that returns null reported no delivery, but the null was silently ignored, leaving the consecutive-success streak intact. It is now treated as an ordinary failure, the same as an exception from the sender, so it breaks the streak without opening a window. This matches the Go implementation, whose value-typed result has an ordinary-failure zero value and so never had this gap. Adds nullResultBreaksTheSuccessStreak.
… (SDK-3513) A 413 Payload Too Large is locally unrecoverable: DefaultEventSender now drops the payload at the sender call site without retrying it and returns an ordinary failure, so delivery is neither disabled nor put into a cooldown window and a subsequent smaller payload can still succeed (EVTPROC 1.7.3.1). This matches the Go server SDK, which drops immediately (the mobile SDKs retry once before dropping; the server SDKs follow the spec's no-retry reading). The shared HttpErrors classifier is deliberately untouched, since data sources share it. Adds a DefaultEventSenderTest case asserting a 413 is not retried, not unexpected, and not a shutdown.
There was a problem hiding this comment.
Risk: medium.
Automated risk review (dry run): Medium
This PR would not be automatically approved. Human approval required.
Why: This replaces permanent event-delivery shutdown with a concurrent cooldown state machine (and 413 payload drop) in DefaultEventProcessor, DeliveryCooldownTracker, DefaultEventSender, and EventSender, which is a non-trivial shared-SDK behavior change even with extensive tests.
If you disagree with this review, share this PR in #feedback-agent-approvals or submit a changes-requested review explaining why.
Sent by Cursor Approval Agent: Dry-Run PR Approver
| } finally { | ||
| // The busy-worker count must always come back down, even if an Error (not an Exception) | ||
| // escaped the work above, or close() would wait forever in | ||
| // waitUntilAllFlushWorkersInactive. The Error still ends this worker thread. | ||
| synchronized (activeFlushWorkersCount) { | ||
| activeFlushWorkersCount.decrementAndGet(); | ||
| activeFlushWorkersCount.notifyAll(); | ||
| } |
There was a problem hiding this comment.
AI review: The finally keeps the count right, but the Error still ends this worker, and nothing replaces it. Once every worker is gone, close() can still hang: its flush increments the busy count in triggerFlush and hands the payload to payloadQueue, where no worker will take it, so waitUntilAllFlushWorkersInactive waits forever. Android runs a single worker, so one Error is enough if any events are pending at close(). errorFromEventSenderDoesNotWedgeClose passes only because its outbox is already empty when it calls close().
This test times out on 6a9a1b0, stuck in close():
@Test(timeout = 15000)
public void errorOnTheOnlyFlushWorkerDoesNotWedgeClose() throws Exception {
EventSender throwingSender = new EventSender() {
@Override
public Result sendAnalyticsEvents(byte[] data, int eventCount, URI eventsBaseUri) {
throw new AssertionError("boom");
}
@Override
public Result sendDiagnosticEvent(byte[] data, URI eventsBaseUri) {
return Result.success(null);
}
@Override
public void close() throws IOException {}
};
// An Error from the sender ends the only flush worker.
DefaultEventProcessor ep = makeEventProcessor(baseConfig(throwingSender).eventSendingThreadPoolSize(1));
ep.sendEvent(identifyEvent(user));
ep.flushAsync();
ep.waitUntilInactive();
// close() must still return when it flushes a pending event that no worker is left to deliver.
ep.sendEvent(identifyEvent(user));
ep.close();
}There was a problem hiding this comment.
AI review comment: Fixed in 5075d56. Both flush-worker work sites now catch Throwable rather than Exception, so an Error (for example an OutOfMemoryError while serializing) is handled as an ordinary failure and the worker survives instead of dying; the finally comment is corrected to match. I also strengthened errorFromEventSenderDoesNotWedgeClose to reproduce exactly this case: a single flush worker (as on Android), a real Error on the first delivery, then an assertion that a second delivery still lands (proving the worker lived) in addition to close() returning. I confirmed it fails when the fix is reverted and passes with it. Leaving this open for you to confirm the approach (catch Throwable to keep the worker alive, rather than replacing a dead worker).
Addresses review feedback on the cooldown PR: - The flush worker now catches Throwable (not only Exception) at both work sites, so an Error such as an OutOfMemoryError during serialization is handled as an ordinary failure instead of ending the worker thread. On Android, which runs a single worker, a lost worker would leave a later close() flush with nobody to drain it and wedge waitUntilAllFlushWorkersInactive forever. Adds a single-worker regression test (proven to fail without the fix). - Replaces spec section numbers in comments with behavioral prose, in the processor's diagnostic comment and two cooldown test comments. - Corrects the DeliveryCooldownTracker class doc: a stale failure still breaks the consecutive-success streak, while tokens confine only window and ladder movement to actual probes.
|
bugbot review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit f601290. Configure here.
🤖 I have created a release *beep* *boop* --- ## [1.14.0](launchdarkly-java-sdk-internal-1.13.1...launchdarkly-java-sdk-internal-1.14.0) (2026-11-20) ### Features * replace permanent event-delivery disable with a bounded cooldown ([#256](#256)) ([f77bf78](f77bf78)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > **Release-only PR** that cuts **launchdarkly-java-sdk-internal 1.14.0**: bumps `lib/shared/internal/gradle.properties`, `.release-please-manifest.json`, and adds the **1.14.0** section to `CHANGELOG.md`. > > The new changelog entry documents the headline feature already merged for this release: **event delivery no longer stays permanently disabled** after failures; it uses a **bounded cooldown** instead (`DeliveryCooldownTracker` in the event processor). > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit bb67741. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What Adopts the event-delivery cooldown (EVTPROC Section 1.8, launchdarkly/sdk-specs#278) in the server SDK by picking up launchdarkly-java-sdk-internal 1.14.0, which replaces the permanent disable-on-unexpected-delivery-failure with bounded, self-recovering cooldown windows (see the internal module PR #256 for the behavior itself). - Bumps `launchdarklyJavaSdkInternal` to 1.14.0. - Migrates the internal `EventSenderWrapper` off the deprecated internal `Result` constructor onto the new factories (`SUCCESS` to `success(null)`, `FAILURE` to `failure()`, `STOP` to `unexpectedFailure()`), and adds the previously missing unit coverage for that mapping. This is the one behavioral change in the server module: a custom `EventSender` returning `STOP` now drives a bounded cooldown rather than a permanent stop. - Leaves the public `subsystems.EventSender.Result.STOP` enum and its javadoc unchanged. `STOP` is still the public signal for an unexpected delivery failure; the cooldown is the SDK's internal response to it, documented in the internal module and the release notes rather than in this interface doc. - Advertises the `event-delivery-cooldown` capability in the contract test service, activating the `events/delivery cooldown` tests. - The 413 contract test runs unsuppressed: the locally-unrecoverable 413 carve-out landed in 1.14.0 (#256), so a 413 drops only the oversized payload without entering cooldown, and the test passes. - The data sources' uses of the deprecated boolean HTTP-error helpers are deliberately untouched; the FDv2 implementations' migration to the classifier is a separate follow-up. ## Dependencies (both gates cleared) 1. launchdarkly-java-sdk-internal 1.14.0 is released (#280 merged; on Maven Central). 2. sdk-test-harness 2.44.0 ships the cooldown tests (#526) with the configurer-ordering fix. ## Verification Built and tested against the released 1.14.0: the server module compiles, checkstyle is clean, and the `EventSenderWrapper` mapping tests pass (STOP to unexpectedFailure, FAILURE to ordinary failure, SUCCESS to delivered, and the diagnostic path using the same mapping). The full `events/` contract suite runs in CI against the released harness. ## Release notes - Minor (feat). Behavior change worth calling out: event delivery no longer shuts down permanently on an unexpected error such as an invalid SDK key; it backs off in windows from 5 minutes up to 1 hour, probing again after each window, and returns to the normal delivery schedule as soon as a probe succeeds. A custom `EventSender` returning `STOP` now causes the same bounded cooldown. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > This PR **adopts event-delivery cooldown** in the Java server SDK by upgrading **`launchdarkly-java-sdk-internal` to 1.14.0**. Unexpected delivery failures (e.g. custom `EventSender` returning `STOP`) now trigger **bounded backoff and recovery** instead of permanently shutting down event delivery. > > The server’s **`EventSenderWrapper`** is updated to use the internal **`Result` factory APIs** (`failure()`, `unexpectedFailure()`, `success(null)`) so `STOP` maps to unexpected failure and plugs into the new cooldown behavior in the internal event processor. > > The contract test service **advertises `event-delivery-cooldown`**, enabling harness cooldown tests. **Unit tests** cover success, ordinary failure, and `STOP` mapping for both analytics and diagnostic sends. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 2598dcf. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->


What
Implements the EVTPROC event processor spec's Section 1.8 delivery cooldown (launchdarkly/sdk-specs#278) in the shared internal events code: on an unexpected delivery failure (a non-recoverable HTTP status such as 401 or 403), the event processor enters a bounded, self-recovering cooldown window instead of permanently disabling event delivery for the lifetime of the SDK instance.
flushAsync()/flushBlocking(), or the shutdown flush) is a no-op: it makes no network attempt and disposes of the batch exactly as a sent-and-refused payload would be disposed of. Two deliberate departures from pure refusal emulation: the diagnostic events-in-last-batch stat reports 0 (no batch was serialized), and the context deduplicator is flushed when the discarded batch contained an index or identify event, so a later payload can introduce those contexts again.DefaultEventSender(1.7.2) is unchanged.This is a port of the reviewed Go implementation (launchdarkly/go-sdk-events#68), adapted to this module's threading model: delivery outcomes arrive on flush worker threads and the shared executor rather than on the dispatcher thread, so the cooldown tracker is synchronized pure state, and outcomes are correlated to the window era they were dispatched in via an opaque
WindowToken(an outcome from a delivery in flight across a window boundary updates nothing).API (internal module surface)
EventSender.Resultgains theunexpectedclassification with static factoriessuccess(Date),failure(), andunexpectedFailure(), and the accessorisUnexpectedFailure(), which also accepts the legacy flag so results from senders written against the old API still get cooldown behavior.Resultconstructor andisMustShutDown()are deprecated.unexpectedFailure()keeps writing the legacy flag for consumers that have not migrated (the server SDK'sEventSenderWrapperand the Android SDK).DefaultEventSendermigrates off the deprecatedisHttpErrorRecoverable/checkIfErrorIsRecoverableAndLoghelpers toclassifyHttpFailure(first commit, behavior-identical) and then returns the new classification (second commit). Transport-levelIOExceptions keep their always-retry behavior.EventsConfigurationgains a package-privatecooldownNanoTimeFntest seam (deliberately not part of any constructor); the tracker documents that the source must haveSystem.nanoTime()semantics, and all window arithmetic uses overflow-safe difference comparisons.Design notes
Dateheader remains unconditional, exactly as on main.disabledflag remains as a shutdown-only latch; nothing sets it from delivery results anymore.finally, so even anError(not anException) escaping the loop body cannot leaveclose()waiting forever; theErrorstill ends that worker thread. (This last part fixes a pre-existing hazard on main, surfaced in review.)NanoTimeSourceinterface rather thanjava.util.function.LongSupplier, which does not exist on Android below API 24 (this module ships in the Android SDK at minSdk 21); the fix is verified with ad8 --min-api 21dex sweep showing zerojava.util.functionreferences.Testing
DeliveryCooldownTrackerTest(24 tests): window ladder with every rung asserted past the shift-guard bound, reset semantics, boundary-round staleness on both probe success and recoverable probe failure, stale-outcome and in-window-outcome guards, null token, default clock path, boundary instant, the atomic gate, and the once-per-window discard notice.DefaultEventProcessorCooldownTest(17 tests): cooldown entry and resume across a fake clock, legacy shutdown-flag results, contradictory success+shutdown results, in-window disposal with context-introduction recovery through both the index and identify arms, a window open forgetting even already-delivered introductions, empty in-window flushes doing nothing, shutdown during a window, warn-once-then-debug and resume logging, diagnostics being fully decoupled from the cooldown (a diagnostic failure opens no window; diagnostic outcomes do not affect an open window), a null sender result neither stopping delivery nor wedgingclose(), a sender exception breaking the success streak, simultaneous failures across the full worker pool stepping the ladder exactly once, and the batch-size stat zeroing on discards.DeliveryCooldownTrackerTest(24 tests): the window ladder and ceiling clamp, reset semantics, boundary-round staleness on probe success and recoverable probe failure, stale failures breaking the streak without moving a window, null-token inertness, the atomic gate, the once-per-window discard notice, overflow and wrap safety.diagnosticEventsStopAfter401Erroris rewritten asdiagnosticEventsContinueAfter401Error; the permanent-disable test is renamed to describe the cooldown contract; a transport-failure sender test pins the deliberate never-unexpected classification of connection errors.Out of scope (follow-ups)
classifyHttpFailurehas no locally-unrecoverable classification, so a 413 opens a window rather than dropping only the oversized payload. That is a pre-existing classifier gap with its own Jira bug (it also requires unsuppressing the corresponding contract test), deliberately not fixed here.EventSender.Result.STOPdoc reinterpretation, wrapper migration,event-delivery-cooldowncontract-test capability, and the 413 test suppression) ship in a follow-up PR once this module releases.DirectEventProcessorreads the dual-written legacy flag and keeps its current permanent-disable behavior byte-identical before and after this release; migrating it to the cooldown is its own coordinated change.Release notes
Note
Overview
Replaces permanent analytics event shutdown on unrecoverable errors (e.g. 401/403) with a bounded delivery cooldown backed by new
DeliveryCooldownTracker: doubling windows (5→60 min), resume after successful probes, and in-window flushes that discard batches without network I/O while preserving recording semantics.DefaultEventProcessorgates flushes with atomic cooldown tokens, correlates worker outcomes viaWindowToken, clears context deduplication on unexpected failures and in-window discards, and hardens flush workers (Throwablehandling,finallyworker accounting). Diagnostic sends use separate response handling and do not consume cooldown tokens on dispatch.EventSender.Resultaddssuccess/failure/unexpectedFailurefactories andisUnexpectedFailure()(legacymustShutDownstill maps).DefaultEventSenderusesclassifyHttpFailure, treats 413 as drop-without-cooldown, and classifies transport errors as ordinary failures with retry.Extensive new tests for the tracker and end-to-end cooldown behavior; existing tests updated for the new result API.
Reviewed by Cursor Bugbot for commit f601290. Bugbot is set up for automated code reviews on this repo. Configure here.