From 3bd9251175959a7b59d4f0cfc5ad877229356c34 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 15 Sep 2026 14:42:31 +0800 Subject: [PATCH] [fix](build) Run copy-dependencies on a build-cache hit so a cache-restored BE plugin still deploys its dependencies Since #66729 every BE Java extension deploys as a thin module jar plus the runtime closure that maven-dependency-plugin:copy-dependencies writes to target/lib at the package phase; build.sh copies that directory verbatim into plugins/jni/. The Maven build cache restores a module's jar on a hit and nothing else - the log says so outright, "Skipping plugin execution (cached): dependency:copy-dependencies" - so a cache-restored plugin has no target/lib at all. On a working copy that is harmless: target/lib is still there from the module's last real build, which is what the comment in be-java-extensions/pom.xml relied on. CI is the opposite case. TeamCity runs `git clean -f -d -x` before every build, so target/ is always fresh, while /home/work/.m2 - build-cache included - is mounted from the host and survives. The first build on an agent after #66729 built the plugins for real and seeded the cache; every later build on that agent restored the thin jars, deployed each plugin as a single jar, and failed check_plugin_layout.py's closure-self-contained rule with a few hundred "is in no jar of this plugin directory" problems (Compile builds 121702, 121701, 121700, 121695, 121689 on 2026-09-15, one PR after another). Only a module whose inputs the PR happened to touch, such as java-udf behind a Config.java change, was rebuilt and came out whole. Force the goal under runAlways in fe/.mvn/maven-build-cache-config.xml, the same mechanism that already keeps checkstyle running on a cache hit. It is matched by goal rather than execution id so the plugins' copy-plugin-dependencies and hadoop-deps' copy-dependencies are covered alike, and so is any module that adds the goal later. The wipe-plugin-lib clean execution that precedes it is forced too: the two exist as a pair, and a copy without the wipe would deploy the union of every dependency set the directory has ever held. Forcing the goal costs a copy out of the local repository; the alternative, keeping the closure in the cache as attached outputs, would store hundreds of jars per plugin per checksum. build.sh had a fallback for exactly this for hadoop-deps only (#60964), re-running the goal from the command line when target/lib was missing. It goes: the forced execution covers hadoop-deps as well, and the fallback ran the goal without the pom's runtime-scope filter, so the be/lib/hadoop_hdfs/lib it produced carried the test-scope closure - the JUnit 5 stack, hamcrest, awaitility, opentest4j - in every CI build where hadoop-deps hit the cache (167 jars where the pom's execution yields 149). Verified locally on master@846d9b2ebfc with jdbc-scanner and hadoop-deps: a real build (12 and 149 jars in target/lib), then rm -rf target and a rebuild with the old config restores the module from the cache and leaves target/ holding only the jar; the same rebuild with this config logs "Mojo execution is forced by project property: dependency:copy-dependencies" and refills target/lib with the identical 12 and 149 jars. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Ja986K4PEm2LD8u44L9jd2 --- build.sh | 13 ++++++------ fe/.mvn/maven-build-cache-config.xml | 30 +++++++++++++++++++++++++++- fe/be-java-extensions/pom.xml | 7 +++++-- 3 files changed, 40 insertions(+), 10 deletions(-) diff --git a/build.sh b/build.sh index d2f701bb8088bb..55c5bacb919001 100755 --- a/build.sh +++ b/build.sh @@ -1591,13 +1591,12 @@ if [[ "${OUTPUT_BE_BINARY}" -eq 1 ]]; then mkdir "${BE_HADOOP_HDFS_DIR}" HADOOP_DEPS_JAR_DIR="${DORIS_HOME}/fe/be-java-extensions/${HADOOP_DEPS_NAME}/target" echo "HADOOP_DEPS_JAR_DIR: ${HADOOP_DEPS_JAR_DIR}" - if [[ "${BUILD_BE_JAVA_EXTENSIONS}" -eq 1 && ! -d "${HADOOP_DEPS_JAR_DIR}/lib" ]]; then - echo "WARN: lib directory missing (likely due to Maven cache). Regenerating..." - pushd "${DORIS_HOME}/fe/be-java-extensions/${HADOOP_DEPS_NAME}" - "${MVN_CMD}" dependency:copy-dependencies -DskipTests -Dcheckstyle.skip=true - mv target/dependency target/lib - popd - fi + # target/lib is present even when the Maven build cache restored this module instead of + # building it: fe/.mvn/maven-build-cache-config.xml forces copy-dependencies to run on a + # cache hit, for hadoop-deps and every plugin alike. There used to be a fallback here + # that re-ran the goal from the command line when the directory was missing; besides + # covering only this one module, it ran the goal without the pom's runtime-scope + # filter and so put the test-scope closure (JUnit and friends) into lib/hadoop_hdfs. if [[ -f "${HADOOP_DEPS_JAR_DIR}/${HADOOP_DEPS_NAME}.jar" ]]; then echo "Copy Be Extensions hadoop deps jar to ${BE_HADOOP_HDFS_DIR}" cp "${HADOOP_DEPS_JAR_DIR}/${HADOOP_DEPS_NAME}.jar" "${BE_HADOOP_HDFS_DIR}" diff --git a/fe/.mvn/maven-build-cache-config.xml b/fe/.mvn/maven-build-cache-config.xml index 59c73b6dafbce0..3643d9a32b1846 100644 --- a/fe/.mvn/maven-build-cache-config.xml +++ b/fe/.mvn/maven-build-cache-config.xml @@ -63,14 +63,42 @@ under the License. - + validate + + + + wipe-plugin-lib + + + + + + + copy-dependencies + + + diff --git a/fe/be-java-extensions/pom.xml b/fe/be-java-extensions/pom.xml index 6758f919a8a299..5b61be24269ec0 100644 --- a/fe/be-java-extensions/pom.xml +++ b/fe/be-java-extensions/pom.xml @@ -106,8 +106,11 @@ under the License. copy-dependencies refills it makes an incremental build's plugin directory equal to a clean one's. - A cache-restored module skips both executions and keeps the target/lib its own last - real build wrote, which is consistent for the same reason. + Both executions are also forced under runAlways in fe/.mvn/maven-build-cache-config.xml. + A build-cache hit restores the module jar and nothing else, so without that a cache- + restored module has no target/lib at all: harmless on a working copy, which still + holds the directory its last real build wrote, but CI wipes target/ before every + build while the cache survives, and deployed every plugin empty. --> org.apache.maven.plugins