Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
f844c90
[feature](be) Add the native Delta Lake reader foundation
Oct 30, 2026
1663b99
[feature](fe) Add the Delta Lake connector metadata foundation
Oct 30, 2026
1a43112
[fix](fe) Resolve Spark Delta table locations from SerDe properties
Oct 30, 2026
2ba9ec1
[feature](fe) Plan native Delta Lake data scans
Oct 30, 2026
0734719
[feature](fe) Prune Delta files and push down exact row counts
Oct 30, 2026
2cfb7d8
[feature](fe) Enable Delta schema evolution and deletion vectors
Oct 30, 2026
4258bd8
[feature](fe) Cache Delta snapshots and stream native scan ranges
Oct 30, 2026
96e48f3
[feature](fe) Read Delta tables through Hive catalogs
Oct 30, 2026
65ec123
[test](fe) Verify Delta planning with one million file actions
Oct 30, 2026
cb2b3c1
[improvement](fe) Avoid redundant work when producing Delta splits
Oct 30, 2026
1b89d88
[test](regression) Verify Parquet pruning for Delta column mapping
Oct 30, 2026
5ae7e2b
[feature](fe) Expose Delta scan diagnostics in EXPLAIN and profiles
Oct 30, 2026
bec3535
[test](regression) Add a repeatable Delta Lake Docker profile
Oct 30, 2026
2dca787
[test](regression) Verify Delta reads across storage backends
Oct 30, 2026
189e3f7
[test](regression) Preserve Delta fixtures across Spark restarts
Oct 30, 2026
4b9a550
[test](regression) Add retained Delta and Hive benchmark fixtures
Oct 30, 2026
85772d8
[test](regression) Reuse retained Delta golden objects on restart
Oct 30, 2026
27bbb8c
[test](regression) Stabilize Delta benchmark comparisons
Oct 30, 2026
96c8ec1
[test](regression) Control the Hive TPC-DS Q85 baseline join order
Oct 30, 2026
5fd10b5
[fix](fe) Read connector statistics at the query snapshot
Nov 2, 2026
7e5c0f6
[feature](fe) Cache Delta Lake table statistics by snapshot version
Nov 2, 2026
6b30d84
[test](regression) Measure Delta benchmarks with prepared table stati…
Nov 2, 2026
7d0e34b
[test](regression) Trim duplicate Delta scanner compatibility cases
Nov 3, 2026
3567a9b
[fix](fe) Restore standard row-count caching for Delta catalog reads
Nov 3, 2026
dae5c8c
[fix](deltalake) Adapt the Delta reader to current master
xylaaaaa Nov 3, 2026
000a321
[refactor](be) Share one portable deletion vector magic check
xylaaaaa Nov 3, 2026
0d9f5c4
[refactor](be) Drop the redundant Delta Parquet check from shared sca…
xylaaaaa Nov 3, 2026
708d32e
[refactor](fe) Keep fe-core split profile error handling unchanged in…
xylaaaaa Nov 3, 2026
235fbee
[test](regression) Drop the deprecated scanner option from Delta suites
xylaaaaa Nov 3, 2026
bd080cb
[fix](regression) Avoid the Hive 3 HMS port in the Delta Docker profile
xylaaaaa Nov 5, 2026
13fe069
[test](regression) Cover how Delta tables are registered in HMS
xylaaaaa Nov 5, 2026
b45bced
[fix](be) Include the cache definition for Delta deletion vector reads
Nov 5, 2026
47dbebd
[test](regression) Record Delta HMS registration results
Nov 5, 2026
ad8d234
[test](regression) Match the Delta HMS INSERT refusal
Nov 5, 2026
eb512b9
[test](regression) Cover Delta deletion vectors with Parquet pruning
xylaaaaa Nov 5, 2026
7e31fe7
[test](regression) Record Delta deletion vector pruning results
Nov 5, 2026
a8954bd
[test](regression) Cover Delta deletion vector pruning boundaries
Nov 5, 2026
a20f534
[fix](fe) Keep cast predicates out of Delta file pruning
Nov 5, 2026
0b9c149
[test](regression) Cover Delta and Iceberg CAST isolation in one HMS …
Nov 6, 2026
661bcbd
[test](deltalake) Guard CAST rejection without pinning Iceberg policy
Nov 6, 2026
3f893ca
Merge upstream master into codex/delta-m1-reader
xylaaaaa Nov 6, 2026
fb4e695
[opt](deltalake) Reuse complete file counts for small-table scheduling
Nov 6, 2026
8b5d68c
[opt](deltalake) Reuse identical scans during statement planning
Nov 6, 2026
e321ede
Merge upstream master into codex/delta-m1-reader
xylaaaaa Nov 12, 2026
8ba8e67
[test](deltalake) Check actual splits in Delta reader regressions
Nov 12, 2026
71897a6
[feature](deltalake) Read Delta tables at a numeric version
Nov 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions be/src/exec/scan/file_scan_range_utils.h
Original file line number Diff line number Diff line change
Expand Up @@ -91,10 +91,11 @@ inline Status validate_file_scan_support(const TFileScanRangeParams& params,
}

inline bool can_ignore_not_found_file(const TFileRangeDesc& range, bool ignore_not_found) {
// Iceberg ranges belong to a committed snapshot, not a stale directory listing. Skipping a
// missing data or delete file would silently return an incomplete or incorrect snapshot.
return ignore_not_found && !(range.__isset.table_format_params &&
range.table_format_params.table_format_type == "iceberg");
// Iceberg and Delta ranges belong to a committed snapshot. Skipping a missing data or
// delete file would silently return a partial snapshot.
const auto& table = range.table_format_params.table_format_type;
return ignore_not_found &&
!(range.__isset.table_format_params && (table == "iceberg" || table == "deltalake"));
}

} // namespace doris
3 changes: 3 additions & 0 deletions be/src/exec/scan/file_scanner_v2.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@
#include "format_v2/jni/max_compute_jni_reader.h"
#include "format_v2/jni/trino_connector_jni_reader.h"
#include "format_v2/table/adbc_reader.h"
#include "format_v2/table/deltalake_reader.h"
#include "format_v2/table/es_reader.h"
#include "format_v2/table/fluss_hybrid_reader.h"
#include "format_v2/table/hive_reader.h"
Expand Down Expand Up @@ -668,6 +669,8 @@ Status FileScannerV2::_create_table_reader_for_format(
*reader = std::make_unique<format::paimon::PaimonHybridReader>();
} else if (table_format == "hudi") {
*reader = std::make_unique<format::hudi::HudiHybridReader>();
} else if (table_format == "deltalake") {
*reader = std::make_unique<format::deltalake::DeltaReader>();
} else if (table_format == "fluss") {
*reader = std::make_unique<format::fluss::FlussHybridReader>();
} else if (table_format == "jdbc") {
Expand Down
10 changes: 10 additions & 0 deletions be/src/format/table/deletion_vector_reader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,16 @@ Status validate_paimon_deletion_vector_read_range(int64_t offset, int64_t length
DeletionVectorSizeLimitStatus::DATA_QUALITY, bytes_read);
}

Status validate_delta_deletion_vector_read_range(int64_t offset, int32_t size_in_bytes,
size_t& bytes_read) {
// Delta and Iceberg share the portable bitmap framing. Delta's descriptor omits the
// four-byte length and four-byte CRC; promote before adding them to avoid i32 overflow.
return validate_deletion_vector_read_range(
"Delta deletion vector", offset, static_cast<int64_t>(size_in_bytes) + 8,
ICEBERG_DELETION_VECTOR_MIN_BYTES, MAX_ICEBERG_DELETION_VECTOR_BYTES,
DeletionVectorSizeLimitStatus::NOT_SUPPORTED, bytes_read);
}

DeletionVectorReader::~DeletionVectorReader() {
// The file reader may retain the child IOContext. Destroy it before merging and before the
// child statistics storage goes away.
Expand Down
7 changes: 7 additions & 0 deletions be/src/format/table/deletion_vector_reader.h
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
#include <cstddef>
#include <cstdint>
#include <memory>
#include <optional>
#include <string>
#include <vector>

Expand All @@ -44,10 +45,14 @@ Status validate_iceberg_deletion_vector_read_range(int64_t offset, int64_t size,
Status validate_paimon_deletion_vector_read_range(int64_t offset, int64_t length,
size_t& bytes_read);

Status validate_delta_deletion_vector_read_range(int64_t offset, int32_t size_in_bytes,
size_t& bytes_read);

struct DeleteFileDesc {
enum class Format {
PAIMON,
ICEBERG,
DELTA,
};

std::string key = "";
Expand All @@ -58,6 +63,8 @@ struct DeleteFileDesc {
int64_t file_size = -1;
int64_t modification_time = 0;
Format format = Format::PAIMON;
std::optional<std::string> inline_data = std::nullopt;
std::optional<uint64_t> expected_cardinality = std::nullopt;
};

class DeletionVectorReader {
Expand Down
44 changes: 33 additions & 11 deletions be/src/format/table/iceberg_delete_file_reader_helper.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,24 @@ namespace {
constexpr const char* ICEBERG_FILE_PATH = "file_path";
constexpr const char* ICEBERG_ROW_POS = "pos";
constexpr size_t ICEBERG_DELETION_VECTOR_MIN_BYTES = 12;
constexpr char PORTABLE_DELETION_VECTOR_MAGIC[] = {'\xD1', '\xD3', '\x39', '\x64'};

Status validate_portable_deletion_vector_magic(const char* buf) {
if (memcmp(buf, PORTABLE_DELETION_VECTOR_MAGIC, sizeof(PORTABLE_DELETION_VECTOR_MAGIC)) != 0) {
return Status::DataQualityError("Deletion vector magic number mismatch");
}
return Status::OK();
}

Status decode_portable_deletion_vector_bitmap(const char* buf, size_t size,
DeletionVector* rows_to_delete) {
try {
*rows_to_delete |= roaring::Roaring64Map::readSafe(buf, size);
} catch (const std::runtime_error& e) {
return Status::DataQualityError("Decode roaring bitmap failed, {}", e.what());
}
return Status::OK();
}

Status validate_position_delete_file_format(const TIcebergDeleteFileDesc& delete_file,
TFileFormatType::type* file_format) {
Expand Down Expand Up @@ -144,10 +162,7 @@ Status decode_deletion_vector_buffer(const char* buf, size_t buffer_size,
static_cast<uint64_t>(total_length) + 8, buffer_size);
}

constexpr static char MAGIC_NUMBER[] = {'\xD1', '\xD3', '\x39', '\x64'};
if (memcmp(buf + sizeof(total_length), MAGIC_NUMBER, 4) != 0) {
return Status::DataQualityError("Deletion vector magic number mismatch");
}
RETURN_IF_ERROR(validate_portable_deletion_vector_magic(buf + sizeof(total_length)));

const uint32_t expected_crc = BigEndian::Load32(buf + sizeof(total_length) + total_length);
const uint32_t actual_crc =
Expand All @@ -157,13 +172,8 @@ Status decode_deletion_vector_buffer(const char* buf, size_t buffer_size,
expected_crc, actual_crc);
}

try {
*rows_to_delete |= roaring::Roaring64Map::readSafe(
buf + 8, buffer_size - ICEBERG_DELETION_VECTOR_MIN_BYTES);
} catch (const std::runtime_error& e) {
return Status::DataQualityError("Decode roaring bitmap failed, {}", e.what());
}
return Status::OK();
return decode_portable_deletion_vector_bitmap(
buf + 8, buffer_size - ICEBERG_DELETION_VECTOR_MIN_BYTES, rows_to_delete);
}

} // namespace
Expand Down Expand Up @@ -324,4 +334,16 @@ Status decode_iceberg_deletion_vector_buffer(const char* buf, size_t buffer_size
return decode_deletion_vector_buffer(buf, buffer_size, rows_to_delete);
}

Status decode_portable_deletion_vector_payload(const char* buf, size_t buffer_size,
DeletionVector* rows_to_delete) {
DORIS_CHECK(buf != nullptr);
DORIS_CHECK(rows_to_delete != nullptr);
// Even an empty portable Roaring64Map has an eight-byte map count after the magic.
if (buffer_size < 12) {
return Status::DataQualityError("Deletion vector payload size too small: {}", buffer_size);
}
RETURN_IF_ERROR(validate_portable_deletion_vector_magic(buf));
return decode_portable_deletion_vector_bitmap(buf + 4, buffer_size - 4, rows_to_delete);
}

} // namespace doris
5 changes: 5 additions & 0 deletions be/src/format/table/iceberg_delete_file_reader_helper.h
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,11 @@ Status validate_iceberg_deletion_vector_descriptor(const TIcebergDeleteFileDesc&
Status decode_iceberg_deletion_vector_buffer(const char* buf, size_t buffer_size,
DeletionVector* rows_to_delete);

// Portable DV magic and bitmap, without the on-disk length prefix and CRC, as carried by
// inline Delta DVs. On-disk Iceberg/Delta DVs use decode_iceberg_deletion_vector_buffer.
Status decode_portable_deletion_vector_payload(const char* buf, size_t buffer_size,
DeletionVector* rows_to_delete);

Status read_iceberg_position_delete_file(const TIcebergDeleteFileDesc& delete_file,
const IcebergDeleteFileReaderOptions& options,
IcebergPositionDeleteVisitor* visitor);
Expand Down
130 changes: 130 additions & 0 deletions be/src/format_v2/table/deltalake_reader.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

#include "format_v2/table/deltalake_reader.h"

#include "format/table/deletion_vector_reader.h"
#include "format_v2/column_mapper.h"
#include "format_v2/parquet/parquet_reader.h"
#include "format_v2/table/schema_history_util.h"
#include "gen_cpp/PlanNodes_types.h"
#include "runtime/runtime_state.h"

namespace doris::format::deltalake {

Status DeltaReader::prepare_split(const SplitReadOptions& options) {
{
SCOPED_TIMER(_profile.total_timer);
SCOPED_TIMER(_profile.prepare_split_timer);
_split_schema_id = -1;
const auto& params = options.current_range.table_format_params;
if (params.__isset.deltalake_params && params.deltalake_params.__isset.schema_id) {
_split_schema_id = params.deltalake_params.schema_id;
DORIS_CHECK(can_map_by_history_schema(_scan_params, _split_schema_id));
}
}
RETURN_IF_ERROR(TableReader::prepare_split(options));
SCOPED_TIMER(_profile.total_timer);
SCOPED_TIMER(_profile.prepare_split_timer);
if (current_split_pruned()) {
return Status::OK();
}
// Delta commits replace data files instead of mutating them. A file may acquire a new DV,
// but its Parquet bytes (and therefore its page-cache identity) remain unchanged.
mark_current_data_file_immutable();
return Status::OK();
}

TableColumnMappingMode DeltaReader::mapping_mode() const {
return can_map_by_history_schema(_scan_params, _split_schema_id)
? TableColumnMappingMode::BY_FIELD_ID
: TableColumnMappingMode::BY_NAME;
}

Status DeltaReader::create_file_reader(std::unique_ptr<FileReader>* reader) {
DORIS_CHECK(reader != nullptr);
DORIS_CHECK(_format == FileFormat::PARQUET);
DORIS_CHECK(_runtime_state != nullptr);
const bool mapping_timestamp_tz = _scan_params != nullptr &&
_scan_params->__isset.enable_mapping_timestamp_tz &&
_scan_params->enable_mapping_timestamp_tz;
const bool mapping_varbinary = _scan_params != nullptr &&
_scan_params->__isset.enable_mapping_varbinary &&
_scan_params->enable_mapping_varbinary;
// Spark's legacy INT96 timestamps are instants. Use the query timezone, independently
// of Hive's INT96 compatibility setting; timestamp_ntz keeps its Parquet wall-clock flag.
*reader = std::make_unique<parquet::ParquetReader>(
_system_properties, _current_task->data_file, _io_ctx, _scanner_profile,
_global_rowid_context, mapping_timestamp_tz, mapping_varbinary,
_runtime_state->timezone(), false, _parquet_profile_cache);
return Status::OK();
}

Status DeltaReader::annotate_file_schema(std::vector<ColumnDefinition>* file_schema) {
DORIS_CHECK(file_schema != nullptr);
if (mapping_mode() == TableColumnMappingMode::BY_FIELD_ID) {
return annotate_file_schema_from_history(_scan_params, _split_schema_id, file_schema);
}
return Status::OK();
}

Status DeltaReader::_parse_deletion_vector_file(const TTableFormatFileDesc& t_desc,
DeleteFileDesc* desc, bool* has_delete_file) {
DORIS_CHECK(desc != nullptr);
DORIS_CHECK(has_delete_file != nullptr);
*has_delete_file = false;
const auto& params = t_desc.deltalake_params;
if (!params.__isset.deletion_vector) {
return Status::OK();
}
const auto& dv = params.deletion_vector;
DORIS_CHECK(dv.__isset.size_in_bytes);
DORIS_CHECK(dv.__isset.cardinality);
if (dv.cardinality < 0) {
return Status::DataQualityError("Delta deletion vector cardinality is negative: {}",
dv.cardinality);
}
size_t bytes_read = 0;
RETURN_IF_ERROR(validate_delta_deletion_vector_read_range(dv.__isset.offset ? dv.offset : 0,
dv.size_in_bytes, bytes_read));
desc->format = DeleteFileDesc::Format::DELTA;
desc->expected_cardinality = static_cast<uint64_t>(dv.cardinality);
if (dv.__isset.inline_data) {
DORIS_CHECK(!dv.__isset.path);
if (dv.inline_data.size() != static_cast<size_t>(dv.size_in_bytes)) {
return Status::DataQualityError(
"Inline Delta deletion vector size mismatch, expected: {}, actual: {}",
dv.size_in_bytes, dv.inline_data.size());
}
desc->inline_data = dv.inline_data;
desc->size = dv.size_in_bytes;
// The exact bytes avoid hash collisions and distinguish different inline DVs without
// performing I/O. Binary strings, including embedded NULs, are valid KV-cache keys.
desc->key = "deltalake_dv_inline_" + dv.inline_data;
} else {
DORIS_CHECK(dv.__isset.path);
desc->path = dv.path;
desc->start_offset = dv.__isset.offset ? dv.offset : 0;
desc->size = static_cast<int64_t>(bytes_read);
desc->key =
fmt::format("deltalake_dv_{}#{}#{}", desc->path, desc->start_offset, desc->size);
}
*has_delete_file = true;
return Status::OK();
}

} // namespace doris::format::deltalake
40 changes: 40 additions & 0 deletions be/src/format_v2/table/deltalake_reader.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

#pragma once

#include "format_v2/table_reader.h"

namespace doris::format::deltalake {

class DeltaReader final : public TableReader {
public:
~DeltaReader() override = default;
Status prepare_split(const SplitReadOptions& options) override;

protected:
Status create_file_reader(std::unique_ptr<FileReader>* reader) override;
TableColumnMappingMode mapping_mode() const override;
Status annotate_file_schema(std::vector<ColumnDefinition>* file_schema) override;
Status _parse_deletion_vector_file(const TTableFormatFileDesc& t_desc, DeleteFileDesc* desc,
bool* has_delete_file) override;

private:
int64_t _split_schema_id = -1;
};

} // namespace doris::format::deltalake
Loading
Loading