Skip to content

[Bug] Sequence field doesn't respect late arriving inserts following a delete event #11019

Open

Description

@junmuz

Search before asking

  • I searched in the issues and found nothing similar.

Paimon version

2.1

Compute Engine

Spark / Paimon

Minimal reproduce step

>>> spark.sql("create table if not exists paimon.test_db.seq_tab (id INT, data STRING, event_ts INT, op_type STRING) TBLPROPERTIES ('primary-key'='id', 'rowkind.field'='op_type', 'changelog-producer' = 'lookup', 'sequence.field' = 'event_ts',  'changelog-producer.ignore-update-before' = 'true', 'changelog-producer.row-deduplicate' = 'true')");
DataFrame[]


>>> spark.sql("INSERT INTO paimon.test_db.seq_tab (id, data, event_ts, op_type) VALUES (1, 'one', 1, '+I')")
DataFrame[]


>>> spark.sql("INSERT INTO paimon.test_db.seq_tab (id, data, event_ts, op_type) VALUES (1, 'one+', 4, '-D')")
DataFrame[]


# After delete a late arriving insert isn't sequenced properly
>>> spark.sql("INSERT INTO paimon.test_db.seq_tab (id, data, event_ts, op_type) VALUES (1, 'one+++', 2, '+I')")
DataFrame[]

>>> spark.sql("select * from paimon.test_db.seq_tab LIMIT 10").show(truncate=False)
+---+------+--------+-------+
|id |data  |event_ts|op_type|
+---+------+--------+-------+
|1  |one+++|2       |+I     |

What doesn't meet your expectations?

A late arriving insert shouldn't be inserted in such cases

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions