Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -209,13 +209,10 @@ private static ConnectorType toStructType(RowType rowType, Options options) {
* Convert a Doris {@link ConnectorType} (as produced by the CREATE TABLE request path)
* to a Paimon {@link DataType}.
*
* <p>This is the faithful reverse of the legacy fe-core
* {@code DorisToPaimonTypeVisitor}: the scalar set is intentionally narrow (it mirrors
* the visitor's {@code atomic} branches and NOT MaxCompute's richer set), CHAR/VARCHAR/STRING
* all collapse to {@code VarChar(MAX)} (declared length dropped), DATETIME/DATETIMEV2 map to a
* plain {@code TimestampType()} (scale dropped), and the MAP key is forced non-null. Types the
* legacy visitor did not handle (TINYINT, SMALLINT, LARGEINT, TIME, IPV4/6, JSON, ...) throw,
* preserving the legacy gap.</p>
* <p>This is the connector-SPI equivalent of the former fe-core
* {@code DorisToPaimonTypeVisitor}. It preserves declared CHAR/VARCHAR bounds and timestamp
* precision because Paimon uses them when applying casts and interpreting numeric timestamp
* values. The MAP key remains non-null, matching Paimon's type contract.</p>
*
* <p>The returned type carries Paimon's default (nullable) flag; column-level nullability is
* applied by the caller via {@code .copy(nullable)} (mirroring legacy
Expand Down Expand Up @@ -243,10 +240,11 @@ public static DataType toPaimonType(ConnectorType type) {
case "DOUBLE":
return new DoubleType();
case "CHAR":
// DDL must retain bounds: widening to STRING changes historical narrowing casts.
return new CharType(type.getPrecision());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Normalize CHAR values before writing the new CHAR schema

Returning CharType(n) also changes newly Doris-created tables from VARCHAR(MAX) to real Paimon CHAR, but the Doris sink does not implement Paimon's CHAR padding. With k CHAR(3) NOT NULL as a primary key, inserting 'a' passes the one-character string through BindSink, Arrow, and PaimonWriteSchema.tableRow; the SDK extracts/stores that key without casting. Paimon's CHAR cast produces 'a ', and its CHAR comparator compares the raw strings, so another writer using the typed CHAR value creates a separate key instead of updating the Doris-written record. Please normalize CHAR values before key/bucket extraction and storage (including nested CHAR), and add a write/cast parity test with a short value. The former VARCHAR(MAX) schema did not require this padding.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed: neither write backend pads CHAR. PaimonWriteSchema.tableRow (JNI) and the FFI backend pass the string through as Doris holds it. That is already the case on master for CHAR(n) columns of tables created by Spark or Flink, which Doris reads as CHAR(n) and writes to unpadded, and branch-4.1/4.2 behave the same with this mapping. The fix belongs in the write path (both backends, nested fields included), where it also covers those existing tables; mapping CHAR(n) to a wider type here would only hide it for tables created by Doris. I'll do the padding as a separate change, and the description now states the limitation.

case "VARCHAR":
return new VarCharType(type.getPrecision());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Apply narrowing casts on the native Paimon read path

This mapping is also used by MODIFY COLUMN. For an append-only Parquet table t(s STRING) containing abcdef, ALTER TABLE t MODIFY COLUMN s VARCHAR(3) now persists the bound, but the default native read still returns abcdef: PaimonScanPlanProvider routes raw files to native without checking character casts, the v1 string converter is ConsistentConverter, and v2 treats string types as equal regardless of length. Paimon's historical cast instead returns abc (also with force_jni_scanner=true). Previously this ALTER mapped to VARCHAR(MAX), so the change exposes inconsistent results after Doris DDL. Please apply Paimon character casts in native reads or route affected splits through JNI, and cover ALTER/read parity with an end-to-end test; the new direct CastExecutors assertion bypasses this path.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The difference is real, but it is how Doris's native readers already treat the CHAR/VARCHAR bounds of external tables, not something this PR adds. They apply the table's length only when truncate_char_or_varchar_columns is on: the format_v2 TableReader handles exactly this case (a table VARCHAR(n) over a wider or STRING file column, _should_truncate_char_or_varchar_column), and Hive and Iceberg tables behave the same. Master already reads Paimon VARCHAR(n) as VARCHAR(n), so a column narrowed by Spark or Flink gives the same native/JNI difference today, and with the session variable on the native read returns abc. This PR only makes Doris DDL store the declared type, as branch-4.1 and branch-4.2 do. Making native reads follow Paimon's casts by default would change those existing tables as well, so I'd keep it out of this PR. I updated the description, which said older values are truncated without mentioning the native default.

case "STRING":
// Legacy parity: all char-family types collapse to VarChar(MAX); declared
// length is intentionally dropped (DorisToPaimonTypeVisitor.atomic isCharFamily).
return new VarCharType(VarCharType.MAX_LENGTH);
case "DATE":
case "DATEV2":
Expand All @@ -259,9 +257,9 @@ public static DataType toPaimonType(ConnectorType type) {
case "DECIMAL256":
return new DecimalType(type.getPrecision(), type.getScale());
case "DATETIME":
return new TimestampType();
case "DATETIMEV2":
return new TimestampType(type.getPrecision());
// Paimon interprets numeric timestamp strings in units derived from this precision.
return new TimestampType("DATETIMEV2".equals(name) ? type.getPrecision() : 0);
case "TIMESTAMPTZ":
// Preserve the instant contract when deriving an external schema.
return new LocalZonedTimestampType(type.getPrecision());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,13 @@
import org.apache.doris.connector.spi.ConnectorType;
import org.apache.doris.connector.spi.DorisConnectorException;

import org.apache.paimon.casting.CastExecutor;
import org.apache.paimon.casting.CastExecutors;
import org.apache.paimon.data.BinaryString;
import org.apache.paimon.types.ArrayType;
import org.apache.paimon.types.BigIntType;
import org.apache.paimon.types.BooleanType;
import org.apache.paimon.types.CharType;
import org.apache.paimon.types.DataField;
import org.apache.paimon.types.DataType;
import org.apache.paimon.types.DateType;
Expand All @@ -46,7 +50,7 @@

/**
* P5-T11 — pins the Doris-&gt;Paimon reverse type mapping in
* {@link PaimonTypeMapping#toPaimonType} to byte-parity with the legacy fe-core
* {@link PaimonTypeMapping#toPaimonType}, the connector equivalent of the former fe-core
* {@code DorisToPaimonTypeVisitor}.
*
* <p>The CREATE TABLE path produces these {@link ConnectorType} descriptors; this mapping is
Expand Down Expand Up @@ -74,26 +78,56 @@ public void scalarPrimitivesMapExactly() {
}

@Test
public void charFamilyCollapsesToVarcharMaxDroppingLength() {
// WHY: legacy isCharFamily -> VarCharType(MAX_LENGTH) unconditionally; the declared length
// is intentionally dropped and CHAR is NOT mapped to paimon CharType. MUTATION: honoring
// the declared length (e.g. new VarCharType(10)) or mapping CHAR -> CharType makes these red.
DataType expected = new VarCharType(VarCharType.MAX_LENGTH);
Assertions.assertEquals(expected, PaimonTypeMapping.toPaimonType(ConnectorType.of("CHAR", 10, 0)));
Assertions.assertEquals(expected, PaimonTypeMapping.toPaimonType(ConnectorType.of("VARCHAR", 20, 0)));
Assertions.assertEquals(expected, PaimonTypeMapping.toPaimonType(ConnectorType.of("STRING")));
public void characterTypesKeepDeclaredBounds() {
// WHY: the declared length is part of the Paimon schema; Paimon truncates to it when it casts
// a historical data file to the current column type. MUTATION: collapsing CHAR/VARCHAR to
// VarChar(MAX), or mapping CHAR to VarCharType, makes these red.
Assertions.assertEquals(new CharType(3), PaimonTypeMapping.toPaimonType(ConnectorType.of("CHAR", 3, 0)));
Assertions.assertEquals(new VarCharType(3),
PaimonTypeMapping.toPaimonType(ConnectorType.of("VARCHAR", 3, 0)));
Assertions.assertEquals(new VarCharType(VarCharType.MAX_LENGTH),
PaimonTypeMapping.toPaimonType(ConnectorType.of("STRING")));
}

@Test
public void datetimePreservesPrecisionAndTimestampKind() {
Assertions.assertEquals(new TimestampType(3),
PaimonTypeMapping.toPaimonType(ConnectorType.of("DATETIMEV2", 3, 0)));
Assertions.assertEquals(new TimestampType(),
for (int precision : new int[] {0, 3, 6}) {
Assertions.assertEquals(new TimestampType(precision),
PaimonTypeMapping.toPaimonType(ConnectorType.of("DATETIMEV2", precision, 0)));
}
// DATETIME (v1) has no fractional seconds.
Assertions.assertEquals(new TimestampType(0),
PaimonTypeMapping.toPaimonType(ConnectorType.of("DATETIME")));
Assertions.assertEquals(new org.apache.paimon.types.LocalZonedTimestampType(6),
PaimonTypeMapping.toPaimonType(ConnectorType.of("TIMESTAMPTZ", 6, 0)));
}

@Test
@SuppressWarnings("unchecked")
public void mappedTypesDriveHistoricalCasts() {
// WHY: Paimon reads a data file written under an older schema by casting the file type to the
// current column type, so the mapped bounds decide the values a reader sees.
CastExecutor<BinaryString, ?> varcharCast = (CastExecutor<BinaryString, ?>) CastExecutors.resolve(
new VarCharType(10), PaimonTypeMapping.toPaimonType(ConnectorType.of("VARCHAR", 3, 0)));
Assertions.assertNotNull(varcharCast);
Assertions.assertEquals("abc", varcharCast.cast(BinaryString.fromString("abcdef")).toString());
CastExecutor<BinaryString, ?> timestampCast = (CastExecutor<BinaryString, ?>) CastExecutors.resolve(
new VarCharType(VarCharType.MAX_LENGTH),
PaimonTypeMapping.toPaimonType(ConnectorType.of("DATETIMEV2", 3, 0)));
Assertions.assertNotNull(timestampCast);
Assertions.assertEquals("1970-01-01T00:00:00.042",
timestampCast.cast(BinaryString.fromString("42")).toString());
}

@Test
public void nestedTypesKeepBoundsAndPrecision() {
Assertions.assertEquals(new ArrayType(new VarCharType(3)),
PaimonTypeMapping.toPaimonType(ConnectorType.arrayOf(ConnectorType.of("VARCHAR", 3, 0))));
Assertions.assertEquals(new MapType(new CharType(false, 2), new TimestampType(3)),
PaimonTypeMapping.toPaimonType(ConnectorType.mapOf(
ConnectorType.of("CHAR", 2, 0), ConnectorType.of("DATETIMEV2", 3, 0))));
}

@Test
public void decimalCarriesPrecisionAndScale() {
// WHY: every decimal family member carries precision/scale through verbatim. MUTATION:
Expand Down
Loading