Skip to content

Commit 41e1103

Browse files
committed
fix(doris): address CodeRabbit review feedback and extend read-only lakehouse format guards
1 parent 34ba79a commit 41e1103

6 files changed

Lines changed: 97 additions & 41 deletions

File tree

‎docs/concepts/models/model_kinds.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -763,7 +763,7 @@ Depending on the target engine, models of the `FULL` kind are materialized using
763763
| Redshift | DROP TABLE, CREATE TABLE, INSERT |
764764
| Postgres | DROP TABLE, CREATE TABLE, INSERT |
765765
| DuckDB | CREATE OR REPLACE TABLE |
766-
| Doris | TRUNCATE TABLE, INSERT (OLAP) / INSERT OVERWRITE (Iceberg) |
766+
| Doris | CREATE TABLE ... AS SELECT (new OLAP), DELETE + INSERT (existing OLAP) / INSERT OVERWRITE (Iceberg) |
767767

768768
## VIEW
769769
The model kinds described so far cause the output of a model query to be materialized and stored in a physical table.

‎docs/integrations/engines/doris.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -746,17 +746,17 @@ MODEL (
746746

747747
## Supported Model Kinds & Catalog Matrix
748748

749-
| SQLMesh Model Kind | Doris Internal OLAP Catalog (`internal`) | External Apache Iceberg Catalog (`table_format iceberg`) | External Lance Catalog (`type = 'lance'`) |
749+
| SQLMesh Model Kind | Doris Internal OLAP Catalog (`internal` on Local Disk or S3/GCS Storage Vault) | External Apache Iceberg Catalog (`table_format iceberg` on S3/GCS/HDFS) | External Read-Only Lakehouse Catalogs (`lance`, `paimon`, `hudi`, `deltalake`) |
750750
| --- | :---: | :---: | :---: |
751-
| Source / Upstream `SELECT` | Supported | Supported (incl. `FOR TIME/VERSION AS OF`, `@branch`, `@tag`) | Supported (Read-Only via Catalog or TVF) |
752-
| `FULL` | Supported (`CREATE TABLE ... AS SELECT` / `DELETE` + `INSERT`) | Supported (`CREATE TABLE ... ENGINE=iceberg` / `INSERT OVERWRITE TABLE`) | Read-Only in Doris 4.1 (Write Guarded) |
753-
| `INCREMENTAL_BY_TIME_RANGE` | Supported (`DELETE WHERE time_col >= low AND time_col <= high` + `INSERT`) | Supported (`DELETE` + `INSERT`) | Read-Only in Doris 4.1 (Write Guarded) |
754-
| `INCREMENTAL_BY_UNIQUE_KEY` | Supported (Auto-promotes `unique_key` to `UNIQUE KEY`, `DELETE ... USING` + `INSERT` or native `MERGE INTO` when `when_matched` / `merge_filter` is specified) | Supported (Batched key `DELETE` + `INSERT` or native `MERGE INTO`) | Read-Only in Doris 4.1 (Write Guarded) |
755-
| `INCREMENTAL_BY_PARTITION` | Supported (Column & time-truncation range `DELETE` + `INSERT`) | Supported (`INSERT OVERWRITE` / partition `DELETE` + `INSERT`) | Read-Only in Doris 4.1 (Write Guarded) |
756-
| `SCD_TYPE_2_BY_TIME` / `SCD_TYPE_2_BY_COLUMN` | Supported | Supported | Read-Only in Doris 4.1 (Write Guarded) |
751+
| Source / Upstream `SELECT` | Supported | Supported (incl. `FOR TIME/VERSION AS OF`, `@branch`, `@tag`) | Supported (Read-Only via Catalog or S3/GCS TVF) |
752+
| `FULL` | Supported (`CREATE TABLE ... AS SELECT` / `DELETE` + `INSERT`) | Supported (`CREATE TABLE ... ENGINE=iceberg` / `INSERT OVERWRITE TABLE`) | Read-Only in Doris (Write Guarded) |
753+
| `INCREMENTAL_BY_TIME_RANGE` | Supported (`DELETE WHERE time_col >= low AND time_col <= high` + `INSERT`) | Supported (`DELETE` + `INSERT`; requires Doris 4.1.0+ & Iceberg v2) | Read-Only in Doris (Write Guarded) |
754+
| `INCREMENTAL_BY_UNIQUE_KEY` | Supported (Auto-promotes `unique_key` to `UNIQUE KEY`, `DELETE ... USING` + `INSERT` or native `MERGE INTO` when `when_matched` / `merge_filter` is specified) | Supported (Batched key `DELETE` + `INSERT` or native `MERGE INTO`; requires Doris 4.1.0+ & Iceberg v2) | Read-Only in Doris (Write Guarded) |
755+
| `INCREMENTAL_BY_PARTITION` | Supported (Column & time-truncation range `DELETE` + `INSERT`) | Supported (`INSERT OVERWRITE` on Doris 2.1+; partition `DELETE` + `INSERT` requires Doris 4.1.0+ & Iceberg v2) | Read-Only in Doris (Write Guarded) |
756+
| `SCD_TYPE_2_BY_TIME` / `SCD_TYPE_2_BY_COLUMN` | Supported | Supported (Requires Doris 4.1.0+ & Iceberg v2 for row-level `DELETE`) | Read-Only in Doris (Write Guarded) |
757757
| `VIEW` | Supported (`CREATE OR REPLACE VIEW`) | Requires `internal` catalog (`environment_catalog_mapping`) | Requires `internal` catalog (`environment_catalog_mapping`) |
758758
| `VIEW (materialized true)` | Supported (Async MTMV & Sync MV with `materialized_type = 'SYNC'`) | Not supported by Doris inside external catalogs | Not supported by Doris inside external catalogs |
759-
| `SEED` / Python DataFrame Models | Supported (Automatic `VARCHAR(255)` OLAP key coercion) | Supported | Read-Only in Doris 4.1 (Write Guarded) |
759+
| `SEED` / Python DataFrame Models | Supported (Automatic `VARCHAR(255)` OLAP key coercion) | Supported | Read-Only in Doris (Write Guarded) |
760760
| `CUSTOM ('doris_ddl_only')` | Supported (DDL + `ALTER TABLE` schema evolution only) | Supported | Not supported (Read-Only) |
761761

762762
---

‎sqlmesh/core/dialect.py‎

Lines changed: 20 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -883,14 +883,22 @@ def _doris_parse_build(self: Parser) -> t.Optional[exp.Expr]:
883883
index_name = self._parse_id_var(any_token=False)
884884
if self._match(TokenType.ON):
885885
table = self._parse_table_parts()
886-
partitions = self._parse_wrapped_id_vars() if self._match_text_seq("PARTITIONS") else []
887-
return self.expression(
886+
partition_kw: t.Optional[str] = None
887+
if self._match_text_seq("PARTITIONS"):
888+
partition_kw = "PARTITIONS"
889+
elif self._match(TokenType.PARTITION) or self._match_text_seq("PARTITION"):
890+
partition_kw = "PARTITION"
891+
partitions = self._parse_wrapped_id_vars() if partition_kw else []
892+
alter_expr = self.expression(
888893
exp.Alter(
889894
this=table,
890895
kind="BUILD_INDEX",
891896
actions=[index_name, *partitions],
892897
)
893898
)
899+
if partition_kw:
900+
alter_expr.meta["partition_keyword"] = partition_kw
901+
return alter_expr
894902
return self._parse_as_command(start)
895903

896904

@@ -911,8 +919,9 @@ def _doris_alter_sql(self: Generator, expression: exp.Alter) -> str:
911919
if expression.text("kind") == "BUILD_INDEX":
912920
idx_sql = self.sql(actions[0]) if actions else ""
913921
tbl_sql = self.sql(expression, "this")
922+
partition_kw = expression.meta.get("partition_keyword") or "PARTITIONS"
914923
parts_sql = (
915-
f" PARTITIONS ({', '.join(self.sql(p) for p in actions[1:])})"
924+
f" {partition_kw} ({', '.join(self.sql(p) for p in actions[1:])})"
916925
if len(actions) > 1
917926
else ""
918927
)
@@ -1755,15 +1764,14 @@ class _DorisTokenizer(Doris.tokenizer_class): # type: ignore
17551764
"SEARCH": lambda args: exp.Anonymous(this="SEARCH", expressions=args),
17561765
}
17571766
)
1758-
Doris.Parser.ALTER_PARSERS.update(
1759-
{
1760-
"ADD": _doris_parse_alter_add,
1761-
"CREATE": lambda self: self._parse_as_command(self._prev),
1762-
"DROP": _doris_parse_alter_drop,
1763-
"EXECUTE": lambda self: self._parse_as_command(self._prev),
1764-
"REPLACE": lambda self: self._parse_as_command(self._prev),
1765-
}
1766-
)
1767+
Doris.Parser.ALTER_PARSERS = {
1768+
**Doris.Parser.ALTER_PARSERS,
1769+
"ADD": _doris_parse_alter_add,
1770+
"CREATE": lambda self: self._parse_as_command(self._prev),
1771+
"DROP": _doris_parse_alter_drop,
1772+
"EXECUTE": lambda self: self._parse_as_command(self._prev),
1773+
"REPLACE": lambda self: self._parse_as_command(self._prev),
1774+
}
17671775
Doris.Generator.TYPE_MAPPING.update(
17681776
{
17691777
exp.DataType.Type.HLLSKETCH: "HLL",

‎sqlmesh/core/engine_adapter/base.py‎

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -526,20 +526,23 @@ def replace_query(
526526
target_columns_to_types = self.columns(target_table)
527527
query = source_queries[0].query_factory()
528528

529+
default_catalog = self.default_catalog
530+
531+
def _qualified(table: exp.Table) -> exp.Table:
532+
table = table.copy()
533+
if default_catalog and not table.catalog and table.db:
534+
table.set("catalog", exp.parse_identifier(default_catalog))
535+
return quote_identifiers(table)
536+
537+
quoted_target = quote_identifiers(target_table.copy())
538+
qualified_target = _qualified(target_table)
539+
529540
def _is_target_table(node: exp.Expr) -> bool:
530541
if not isinstance(node, exp.Table):
531542
return False
532-
if quote_identifiers(node.copy()) == quote_identifiers(target_table.copy()):
543+
if quote_identifiers(node.copy()) == quoted_target:
533544
return True
534-
if self.default_catalog:
535-
norm_node = node.copy()
536-
if not norm_node.catalog and norm_node.db:
537-
norm_node.set("catalog", exp.parse_identifier(self.default_catalog))
538-
norm_target = target_table.copy()
539-
if not norm_target.catalog and norm_target.db:
540-
norm_target.set("catalog", exp.parse_identifier(self.default_catalog))
541-
return quote_identifiers(norm_node) == quote_identifiers(norm_target)
542-
return False
545+
return bool(default_catalog) and _qualified(node) == qualified_target
543546

544547
self_referencing = any(_is_target_table(table) for table in query.find_all(exp.Table))
545548
# If a query references itself then it must have a table created regardless of approach used.

‎sqlmesh/core/engine_adapter/doris.py‎

Lines changed: 39 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -351,28 +351,57 @@ def _restore_active_catalog_on_reconnect(self, sql: t.Optional[str] = None) -> N
351351
self.cursor.execute(f"USE {db_sql}")
352352
self._needs_catalog_restore = False
353353

354-
def _is_lance_table(
354+
_READ_ONLY_EXTERNAL_FORMATS: t.ClassVar[t.Dict[str, str]] = {
355+
"lance": "Lance",
356+
"paimon": "Paimon",
357+
"hudi": "Hudi",
358+
"deltalake": "Delta Lake",
359+
"delta": "Delta Lake",
360+
"lakesoul": "LakeSoul",
361+
}
362+
363+
def _get_read_only_external_format(
355364
self,
356365
table_format: t.Optional[str] = None,
357366
table_properties: t.Optional[t.Dict[str, t.Any]] = None,
358-
) -> bool:
359-
if table_format and str(table_format).strip().lower() == "lance":
360-
return True
367+
) -> t.Optional[str]:
368+
if table_format:
369+
fmt_lower = str(table_format).strip().lower()
370+
if fmt_lower in self._READ_ONLY_EXTERNAL_FORMATS:
371+
return fmt_lower
361372
if table_properties:
362373
engine_val = self._extract_str_value(table_properties.get("engine"))
363-
if engine_val and engine_val.strip().lower() == "lance":
364-
return True
365-
return False
374+
if engine_val:
375+
eng_lower = engine_val.strip().lower()
376+
if eng_lower in self._READ_ONLY_EXTERNAL_FORMATS:
377+
return eng_lower
378+
return None
379+
380+
def _is_lance_table(
381+
self,
382+
table_format: t.Optional[str] = None,
383+
table_properties: t.Optional[t.Dict[str, t.Any]] = None,
384+
) -> bool:
385+
return (
386+
self._get_read_only_external_format(
387+
table_format=table_format, table_properties=table_properties
388+
)
389+
is not None
390+
)
366391

367392
def _ensure_not_lance_table(
368393
self,
369394
table_format: t.Optional[str] = None,
370395
table_properties: t.Optional[t.Dict[str, t.Any]] = None,
371396
) -> None:
372-
if self._is_lance_table(table_format=table_format, table_properties=table_properties):
397+
ro_fmt = self._get_read_only_external_format(
398+
table_format=table_format, table_properties=table_properties
399+
)
400+
if ro_fmt is not None:
401+
fmt_label = self._READ_ONLY_EXTERNAL_FORMATS[ro_fmt]
373402
raise SQLMeshError(
374-
"Doris Lance tables are read-only; reference Lance tables as upstream sources or EXTERNAL models "
375-
"using a Doris Lance catalog ('type' = 'lance') or file table-valued function ('format' = 'lance')."
403+
f"Doris {fmt_label} tables are read-only; reference {fmt_label} tables as upstream sources or EXTERNAL models "
404+
f"using a Doris {fmt_label} catalog ('type' = '{ro_fmt}') or file table-valued function ('format' = '{ro_fmt}')."
376405
)
377406

378407
def _is_iceberg_table(

‎tests/core/engine_adapter/test_doris.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4875,6 +4875,13 @@ def test_lance_table_reading_and_write_guard(
48754875
table_properties={"engine": exp.Literal.string("lance")},
48764876
)
48774877

4878+
with pytest.raises(SQLMeshError, match="Doris Paimon tables are read-only"):
4879+
adapter.create_table(
4880+
"paimon_cat.ml_db.out_tbl",
4881+
target_columns_to_types={"id": exp.DataType.build("INT", dialect="doris")},
4882+
table_format="paimon",
4883+
)
4884+
48784885

48794886
def test_iceberg_reading_time_travel_branch_and_compaction_ddl(
48804887
make_mocked_engine_adapter: t.Callable[..., DorisEngineAdapter],
@@ -4969,6 +4976,15 @@ def test_doris_4_1_search_indexes_compaction_and_sequence_col(
49694976
assert build_idx.this.sql("doris") == "db.docs"
49704977
assert build_idx.sql("doris") == "BUILD INDEX idx_title_ft ON db.docs PARTITIONS (p1, p2)"
49714978

4979+
build_idx_singular = parse_one(
4980+
"BUILD INDEX idx_title_ft ON db.docs PARTITION (p1, p2)", read="doris"
4981+
)
4982+
assert isinstance(build_idx_singular, exp.Alter)
4983+
assert build_idx_singular.this.sql("doris") == "db.docs"
4984+
assert (
4985+
build_idx_singular.sql("doris") == "BUILD INDEX idx_title_ft ON db.docs PARTITION (p1, p2)"
4986+
)
4987+
49724988
alter_add_idx = parse_one(
49734989
"ALTER TABLE db.docs ADD INDEX idx_title (title) USING INVERTED PROPERTIES ('parser' = 'english') COMMENT 'ft'",
49744990
read="doris",

0 commit comments

Comments
 (0)