Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docker/thirdparties/docker-compose/iceberg/iceberg.yaml.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ services:
- ./spark-defaults.conf:/opt/spark/conf/spark-defaults.conf
- ./data/input/jars/iceberg-aws-bundle-1.10.1.jar:/opt/spark/jars/iceberg-aws-bundle-1.10.1.jar
- ./data/input/jars/iceberg-spark-runtime-4.0_2.13-1.10.1.jar:/opt/spark/jars/iceberg-spark-runtime-4.0_2.13-1.10.1.jar
- ./data/input/jars/paimon-s3-1.3.1.jar:/opt/spark/jars/paimon-s3-1.3.1.jar
- ./data/input/jars/paimon-spark-4.0-1.3.1.jar:/opt/spark/jars/paimon-spark-4.0-1.3.1.jar
- ./data/input/jars/paimon-s3-1.4.2.jar:/opt/spark/jars/paimon-s3-1.4.2.jar
- ./data/input/jars/paimon-spark-4.0_2.13-1.4.2.jar:/opt/spark/jars/paimon-spark-4.0_2.13-1.4.2.jar

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Refresh cached jars before switching these mounts

After an existing Iceberg environment is updated to this revision, running run-thirdparties-docker.sh -c iceberg retains its data/ directory and skips the new archive download (start_iceberg, lines 1440–1457). The previous archive contains only the two Paimon 1.3.1 jars, so neither new 1.4.2 source path exists. Compose creates directories for missing short-syntax bind sources, leaving Spark without the Paimon catalog classes. Paimon fixture initialization and the suite's spark_paimon queries then fail to load the configured catalog. Please refresh the required jars or detect and invalidate an incompatible cached fixture before starting Compose.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 10327556968. Agreed on the mechanism: start_iceberg downloads the archive only when data/ is absent, so an environment extracted from the previous archive kept only the Paimon 1.3.1 jars, and Compose created empty directories for the new 1.4.2 mount sources. CI starts from a clean environment on every run, which is why the External pipeline passed; local environments stayed broken until data/ was removed by hand. branch-4.x and the earlier archive switch in #61149 have the same gap.

start_iceberg now checks every jar that iceberg.yaml mounts from data/input/jars before it decides whether to download. If one is missing, or is a directory Compose created, the stale data/ is removed and the current archive is extracted again, the same as a fresh install. A later jar change in iceberg.yaml is picked up the same way.

Checked on four layouts: all mounted jars present (kept), a data/ from the old archive (re-extracted), a jar path that Compose had created as a directory (re-extracted), and no data/ (downloaded as before). run-thirdparties-docker.sh -c iceberg against an up-to-date data/ keeps it and starts the stack as before.

environment:
- AWS_ACCESS_KEY_ID=admin
- AWS_SECRET_ACCESS_KEY=password
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,10 @@ tblproperties (

insert into variant_mixed_su values
(1, date '2026-06-01', parse_json('{"name":"alice","age":18,"layout":"shredded"}'));
alter table variant_mixed_su set tblproperties (
'parquet.variant.shreddingSchema' = ''
-- Paimon 1.4.2 parses every present shredding schema as JSON, so remove the property
-- before writing unshredded files.
alter table variant_mixed_su unset tblproperties (
'parquet.variant.shreddingSchema'
);
insert into variant_mixed_su values
(2, date '2026-07-01', parse_json('{"name":"bob","age":30,"layout":"unshredded"}'));
23 changes: 18 additions & 5 deletions docker/thirdparties/run-thirdparties-docker.sh
Original file line number Diff line number Diff line change
Expand Up @@ -1437,21 +1437,34 @@ start_iceberg() {
register_stack_metadata "iceberg" "${ROOT}/docker-compose/iceberg/iceberg.yaml" "${ROOT}/docker-compose/iceberg/iceberg.env"
compose_down_stack "${ROOT}/docker-compose/iceberg/iceberg.yaml" "${ROOT}/docker-compose/iceberg/iceberg.env" --remove-orphans
if [[ "${STOP}" -ne 1 ]]; then
# data/ is extracted from the fixture archive once and then reused. A data/ from an older archive lacks
# jars that iceberg.yaml now mounts, and Compose would create empty directories in their place, so
# start over from the current archive.
if [[ -d "${ICEBERG_DIR}/data" ]]; then
local jar
for jar in $(grep -o '\./data/input/jars/[^:]*\.jar' "${ICEBERG_DIR}/iceberg.yaml"); do
if [[ ! -f "${ICEBERG_DIR}/${jar#./}" ]]; then
echo "${ICEBERG_DIR}/${jar#./} is missing, ${ICEBERG_DIR}/data is from an older archive"
host_admin_cmd rm -rf "${ICEBERG_DIR:?}/data"
break
fi
done
fi
if [[ ! -d "${ICEBERG_DIR}/data" ]]; then
echo "${ICEBERG_DIR}/data does not exist"
(
cd "${ICEBERG_DIR}" || exit 1
rm -f iceberg_data*.zip
if [[ "${HOST_OS}" == "Darwin" ]]; then
curl -fL -o iceberg_data_spark40.zip \
"https://${s3BucketName}.${s3Endpoint}/regression/datalake/pipeline_data/iceberg_data_spark40.zip"
curl -fL -o iceberg_data_spark40_paimon142.zip \
"https://${s3BucketName}.${s3Endpoint}/regression/datalake/pipeline_data/iceberg_data_spark40_paimon142.zip"
else
wget -P "${ROOT}/docker-compose/iceberg" \
"https://${s3BucketName}.${s3Endpoint}/regression/datalake/pipeline_data/iceberg_data_spark40.zip"
"https://${s3BucketName}.${s3Endpoint}/regression/datalake/pipeline_data/iceberg_data_spark40_paimon142.zip"
fi
host_admin_cmd unzip iceberg_data_spark40.zip
host_admin_cmd unzip iceberg_data_spark40_paimon142.zip
host_admin_cmd mv iceberg_data data
host_admin_cmd rm -rf iceberg_data_spark40.zip
host_admin_cmd rm -rf iceberg_data_spark40_paimon142.zip
)
else
echo "${ICEBERG_DIR}/data exist, continue !"
Expand Down
Original file line number Diff line number Diff line change
@@ -1,30 +1,30 @@
-- This file is automatically generated. You should know what you did if you want to edit this
-- !show_partition_sales_by_date --
sale_date=2024-01-15 sale_date 2 2216 1
sale_date=2024-01-16 sale_date 2 4217 2
sale_date=2024-01-17 sale_date 1 2118 1
sale_date=2024-01-15 sale_date 2 2100 1
sale_date=2024-01-16 sale_date 2 3985 2
sale_date=2024-01-17 sale_date 1 2002 1

-- !show_partition_sales_by_region --
region=China-Beijing region 1 2425 1
region=Japan-Tokyo region 1 2420 1
region=USA-California region 1 2454 1
region=China-Beijing region 1 2295 1
region=Japan-Tokyo region 1 2290 1
region=USA-California region 1 2324 1

-- !show_partition_sales_by_date_region --
sale_date=2024-01-15/region=China-Beijing sale_date,region 1 2627 1
sale_date=2024-01-15/region=Japan-Tokyo sale_date,region 1 2614 1
sale_date=2024-01-15/region=USA-California sale_date,region 1 2655 1
sale_date=2024-01-16/region=China-Shanghai sale_date,region 1 2636 1
sale_date=2024-01-16/region=Japan-Osaka sale_date,region 1 2636 1
sale_date=2024-01-16/region=USA-New York sale_date,region 1 2643 1
sale_date=2024-01-15/region=China-Beijing sale_date,region 1 2485 1
sale_date=2024-01-15/region=Japan-Tokyo sale_date,region 1 2472 1
sale_date=2024-01-15/region=USA-California sale_date,region 1 2513 1
sale_date=2024-01-16/region=China-Shanghai sale_date,region 1 2494 1
sale_date=2024-01-16/region=Japan-Osaka sale_date,region 1 2494 1
sale_date=2024-01-16/region=USA-New York sale_date,region 1 2501 1

-- !show_partition_events_by_hour --
hour_partition=2024-01-15-10 hour_partition 2 2361 1
hour_partition=2024-01-15-11 hour_partition 2 4515 2
hour_partition=2024-01-15-14 hour_partition 2 2371 1
hour_partition=2024-01-15-10 hour_partition 2 2245 1
hour_partition=2024-01-15-11 hour_partition 2 4283 2
hour_partition=2024-01-15-14 hour_partition 2 2255 1

-- !show_partition_logs_by_date_hierarchy --
year_val=2024/month_val=1/day_val=15 year_val,month_val,day_val 2 2841 1
year_val=2024/month_val=1/day_val=16 year_val,month_val,day_val 2 5323 2
year_val=2024/month_val=1/day_val=17 year_val,month_val,day_val 1 2658 1
year_val=2024/month_val=2/day_val=1 year_val,month_val,day_val 1 2686 1
year_val=2024/month_val=1/day_val=15 year_val,month_val,day_val 2 2701 1
year_val=2024/month_val=1/day_val=16 year_val,month_val,day_val 2 5042 2
year_val=2024/month_val=1/day_val=17 year_val,month_val,day_val 1 2517 1
year_val=2024/month_val=2/day_val=1 year_val,month_val,day_val 1 2546 1

Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,18 @@ p2 4 4
p3 5 4
p3 6 3

-- !external_weight_robin --
1 weight-1
2 weight-2
3 weight-3
4 weight-4
5 weight-5
6 weight-6

-- !external_specific_fs --
1 specific-1
2 specific-2

-- !external_default_path --
1 default-path

Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,19 @@ suite("test_paimon_write_external_paths", "p0,external,paimon") {
'data-file.external-paths.strategy' = 'round-robin'
);

-- The Spark fixture uses Paimon 1.3.1, which cannot create a table with the
-- weight-robin strategy introduced in Paimon 1.4. Re-enable that strategy
-- after the fixture has a compatible Spark connector.
DROP TABLE IF EXISTS paimon.${dbName}.t_weight_robin;
CREATE TABLE paimon.${dbName}.t_weight_robin (
id INT, payload STRING
) USING paimon
TBLPROPERTIES (
'primary-key' = 'id',
'bucket' = '1',
'write-only' = 'true',
'target-file-size' = '1 kb',
'data-file.external-paths' = '${pathRoot}/weight-a,${pathRoot}/weight-b',
'data-file.external-paths.strategy' = 'weight-robin',
'data-file.external-paths.weights' = '1,1'
);

DROP TABLE IF EXISTS paimon.${dbName}.t_specific_fs;
CREATE TABLE paimon.${dbName}.t_specific_fs (
Expand Down Expand Up @@ -197,6 +207,18 @@ suite("test_paimon_write_external_paths", "p0,external,paimon") {
assertDorisSparkRows("external_round_robin_changed", "t_round_robin",
"pt, id, length(payload)", "ORDER BY pt, id")

(1..6).each { id ->
sql """INSERT INTO t_weight_robin VALUES (${id}, 'weight-${id}')"""
}
def weightedFiles = dataFiles("t_weight_robin")
assertFalse(weightedFiles.isEmpty())
assertTrue(weightedFiles.every {
it.startsWith("${pathRoot}/weight-a/") ||
it.startsWith("${pathRoot}/weight-b/")
})
assertDorisSparkRows("external_weight_robin", "t_weight_robin",
"id, payload", "ORDER BY id")

sql """INSERT INTO t_specific_fs VALUES (1, 'specific-1')"""
sql """INSERT INTO t_specific_fs VALUES (2, 'specific-2')"""
def specificFiles = dataFiles("t_specific_fs")
Expand Down
Loading