Skip to content

fix: Keep deleted flags and segments deleted in the persistent store after an FDv2 store outage - #588

Draft
kinyoklion wants to merge 1 commit into
mainfrom
rlamb/sdk-3705/write-back-tombstones
Draft

kinyoklion wants to merge 1 commit into
mainfrom
rlamb/sdk-3705/write-back-tombstones

Conversation

@kinyoklion

Copy link
Copy Markdown
Member

After a persistent store outage, the FDv2 data system writes its in-memory data back to the store. Store.commit and AsyncStore.commit read that data with InMemoryFeatureStore.all, which leaves out deleted items, and the write-back replaces the whole store. So a flag or segment deleted during the outage lost its tombstone, and an update with an older version from another writer could add it again. The write-back now includes deleted items. The contract test services now advertise persistent-data-store-recovery; without it the store recovery tests do not run.

Still failing: persistent data store/dynamodb/store recovery/retries write back after another failure on the sync service. boto3 retries the cut write-back request itself and reports success, so the SDK never starts a fresh write-back. Whether client retries should count as a failed write-back is a separate question.

Testing: the new unit tests fail without the change; the v3 store recovery contract tests go from 3/18 to 17/18 passing on the sync service and from 1/6 to 6/6 on the async service.

SDK-3705

Opened from termag

…after an FDv2 store outage

After a persistent store outage, the FDv2 store writes its in-memory data
back to the persistent store. Store.commit() and AsyncStore.commit() read
that data with InMemoryFeatureStore.all(), which leaves out deleted items.
The write-back replaces the whole store, so it removed the tombstone of each
item deleted during the outage, and an update with an older version could
then add the item again. The write-back now includes deleted items.

The contract test services now advertise persistent-data-store-recovery,
so that the store recovery tests run.

Generated-by: termag
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant