Repository navigation
[fix](paimon) Keep CHAR/VARCHAR lengths and DATETIME precision when creating Paimon tables #69045
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -209,13 +209,10 @@ private static ConnectorType toStructType(RowType rowType, Options options) { | |
| * Convert a Doris {@link ConnectorType} (as produced by the CREATE TABLE request path) | ||
| * to a Paimon {@link DataType}. | ||
| * | ||
| * <p>This is the faithful reverse of the legacy fe-core | ||
| * {@code DorisToPaimonTypeVisitor}: the scalar set is intentionally narrow (it mirrors | ||
| * the visitor's {@code atomic} branches and NOT MaxCompute's richer set), CHAR/VARCHAR/STRING | ||
| * all collapse to {@code VarChar(MAX)} (declared length dropped), DATETIME/DATETIMEV2 map to a | ||
| * plain {@code TimestampType()} (scale dropped), and the MAP key is forced non-null. Types the | ||
| * legacy visitor did not handle (TINYINT, SMALLINT, LARGEINT, TIME, IPV4/6, JSON, ...) throw, | ||
| * preserving the legacy gap.</p> | ||
| * <p>This is the connector-SPI equivalent of the former fe-core | ||
| * {@code DorisToPaimonTypeVisitor}. It preserves declared CHAR/VARCHAR bounds and timestamp | ||
| * precision because Paimon uses them when applying casts and interpreting numeric timestamp | ||
| * values. The MAP key remains non-null, matching Paimon's type contract.</p> | ||
| * | ||
| * <p>The returned type carries Paimon's default (nullable) flag; column-level nullability is | ||
| * applied by the caller via {@code .copy(nullable)} (mirroring legacy | ||
|
|
@@ -243,10 +240,11 @@ public static DataType toPaimonType(ConnectorType type) { | |
| case "DOUBLE": | ||
| return new DoubleType(); | ||
| case "CHAR": | ||
| // DDL must retain bounds: widening to STRING changes historical narrowing casts. | ||
| return new CharType(type.getPrecision()); | ||
| case "VARCHAR": | ||
| return new VarCharType(type.getPrecision()); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P1] Apply narrowing casts on the native Paimon read path This mapping is also used by MODIFY COLUMN. For an append-only Parquet table
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The difference is real, but it is how Doris's native readers already treat the CHAR/VARCHAR bounds of external tables, not something this PR adds. They apply the table's length only when |
||
| case "STRING": | ||
| // Legacy parity: all char-family types collapse to VarChar(MAX); declared | ||
| // length is intentionally dropped (DorisToPaimonTypeVisitor.atomic isCharFamily). | ||
| return new VarCharType(VarCharType.MAX_LENGTH); | ||
| case "DATE": | ||
| case "DATEV2": | ||
|
|
@@ -259,9 +257,9 @@ public static DataType toPaimonType(ConnectorType type) { | |
| case "DECIMAL256": | ||
| return new DecimalType(type.getPrecision(), type.getScale()); | ||
| case "DATETIME": | ||
| return new TimestampType(); | ||
| case "DATETIMEV2": | ||
| return new TimestampType(type.getPrecision()); | ||
| // Paimon interprets numeric timestamp strings in units derived from this precision. | ||
| return new TimestampType("DATETIMEV2".equals(name) ? type.getPrecision() : 0); | ||
| case "TIMESTAMPTZ": | ||
| // Preserve the instant contract when deriving an external schema. | ||
| return new LocalZonedTimestampType(type.getPrecision()); | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P2] Normalize CHAR values before writing the new CHAR schema
Returning
CharType(n)also changes newly Doris-created tables from VARCHAR(MAX) to real Paimon CHAR, but the Doris sink does not implement Paimon's CHAR padding. Withk CHAR(3) NOT NULLas a primary key, inserting'a'passes the one-character string throughBindSink, Arrow, andPaimonWriteSchema.tableRow; the SDK extracts/stores that key without casting. Paimon's CHAR cast produces'a ', and its CHAR comparator compares the raw strings, so another writer using the typed CHAR value creates a separate key instead of updating the Doris-written record. Please normalize CHAR values before key/bucket extraction and storage (including nested CHAR), and add a write/cast parity test with a short value. The former VARCHAR(MAX) schema did not require this padding.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Confirmed: neither write backend pads CHAR.
PaimonWriteSchema.tableRow(JNI) and the FFI backend pass the string through as Doris holds it. That is already the case on master for CHAR(n) columns of tables created by Spark or Flink, which Doris reads as CHAR(n) and writes to unpadded, and branch-4.1/4.2 behave the same with this mapping. The fix belongs in the write path (both backends, nested fields included), where it also covers those existing tables; mapping CHAR(n) to a wider type here would only hide it for tables created by Doris. I'll do the padding as a separate change, and the description now states the limitation.