Repository navigation
fix: Keep deleted flags and segments deleted in the persistent store after an FDv2 store outage - #588
Draft
kinyoklion wants to merge 1 commit into
Draft
kinyoklion wants to merge 1 commit into
kinyoklion wants to merge 1 commit into
Conversation
…after an FDv2 store outage After a persistent store outage, the FDv2 store writes its in-memory data back to the persistent store. Store.commit() and AsyncStore.commit() read that data with InMemoryFeatureStore.all(), which leaves out deleted items. The write-back replaces the whole store, so it removed the tombstone of each item deleted during the outage, and an update with an older version could then add the item again. The write-back now includes deleted items. The contract test services now advertise persistent-data-store-recovery, so that the store recovery tests run. Generated-by: termag
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After a persistent store outage, the FDv2 data system writes its in-memory data back to the store.
Store.commitandAsyncStore.commitread that data withInMemoryFeatureStore.all, which leaves out deleted items, and the write-back replaces the whole store. So a flag or segment deleted during the outage lost its tombstone, and an update with an older version from another writer could add it again. The write-back now includes deleted items. The contract test services now advertisepersistent-data-store-recovery; without it the store recovery tests do not run.Still failing:
persistent data store/dynamodb/store recovery/retries write back after another failureon the sync service. boto3 retries the cut write-back request itself and reports success, so the SDK never starts a fresh write-back. Whether client retries should count as a failed write-back is a separate question.Testing: the new unit tests fail without the change; the v3 store recovery contract tests go from 3/18 to 17/18 passing on the sync service and from 1/6 to 6/6 on the async service.
SDK-3705
Opened from termag