Skip to content

Fail a retried upsert instead of inserting a concurrently inserted key - #4392

Open
Anurag-S1ngh wants to merge 1 commit into
apache:mainfrom
Anurag-S1ngh:fix/4194-upsert-retry-concurrent-insert
Open

Anurag-S1ngh wants to merge 1 commit into
apache:mainfrom
Anurag-S1ngh:fix/4194-upsert-retry-concurrent-insert

Conversation

@Anurag-S1ngh

Copy link
Copy Markdown
Contributor

Summary

A commit retry replays the append planned before the concurrent commit. The fast append producer skips concurrency validation, so a key another writer inserted after the upsert read was inserted a second time.

Changes

  • Register the filter for the rows an upsert inserts on the append producer.
  • On retry, validate the data files added since the starting snapshot against that filter and raise CommitFailedException instead of inserting a duplicate key.
  • Plain appends keep skipping validation.

Testing

  • Added a test with two table handles where one upserts a key after the other has read the table, asserting the second upsert fails and the key is stored once.
  • make test passes (4697 passed, 5 skipped).

Closes #4194

A commit retry replays the append that was planned before the concurrent
commit, and the fast append producer skips concurrency validation, so a key
another writer inserted after the upsert read ends up inserted twice.

Record the filter for the rows an upsert inserts on the append producer and
validate the files added since the starting snapshot against it when the
commit is rebuilt for a retry. The upsert now fails with a
CommitFailedException instead of leaving a duplicate key.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A retried upsert inserts a key that a concurrent writer already inserted

1 participant