From 6c25f24c34bdd12f59266785a9cf518f1e73b04d Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 10 Sep 2026 13:52:25 +0800 Subject: [PATCH 01/19] [improvement](fe) Stop fe-common depending on Hadoop and pin fe-core's hidden Hadoop passengers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Groundwork for an FE kernel that ships no Hadoop classes at all — the end state `ConnectorPluginManager` already names in its parent-first comment. Nothing is removed from `fe/lib` here; this PR only makes the remaining coupling honest, so that the step that does remove it cannot silently take unrelated libraries with it. fe-common declared `hadoop-common` (provided) and `hadoop-aws` while not naming a single Hadoop class — the module has zero `org.apache.hadoop` references, only two configuration-key strings in `Config`. Dropping them exposed `Hll`, which uses `org.apache.commons.codec.binary.StringUtils` and had been riding on hadoop-common's commons-codec; that is now declared where it is used. `hadoop-aws` was compile scope, so it also stops being re-exported to fe-type, fe-catalog, hive-udf and the BE's java-udf plugin, none of which reference `org.apache.hadoop.fs.s3a` either. fe-core is unaffected: it declares hadoop-aws itself. The BE's java-udf plugin is the one consumer whose closure is checked by name (tools/be-java-plugins/check_plugin_layout.py), and it excluded `hadoop-aws` from fe-common explicitly; that exclusion is now dead and goes. In its place the plugin excludes `commons-codec`: the layout check records that jar as deliberately absent from java-udf — fe-common's only user of it, `Hll`, is on no BE path — and it used to be absent for free, because hadoop-common's provided scope never re-exported it. With fe-common declaring it, the plugin has to say so, or the jar walks in. Its plugin directory is unchanged as a result. fe-core likewise names no Hadoop class in main, but two libraries it compiles against reach it only as Hadoop transitives: - metrics-core (`com.codahale.metrics`) via hadoop-auth — the entire metric layer is built on it: MetricRepo, every MetricVisitor, HistogramMetric, CloudMetrics, SqlBlockRule. - bcprov-jdk18on (`org.bouncycastle.util`) via hadoop-common — used by TableScanParams. Both are now declared. They are already on the classpath at these versions, so no jar is added. Finally, the three property names fe-core read from `AuthenticationConfig` — a fe-kerberos class that imports `org.apache.hadoop.conf.Configuration` — move to a new hadoop-free holder in fe-foundation, `HadoopAuthConfigKeys`. `AuthenticationConfig` re-exports every one of them, so both spellings name the same property. The old copies were plain `public static String`, not compile-time constants, so `HdfsStorageVault.PropertyKey` and `HdfsResource` loaded `AuthenticationConfig` — and with it Hadoop's `Configuration` — to read a string; they no longer do. fe-core keeps its fe-kerberos dependency for `ExecutionAuthenticator`, which is hadoop-free. Verified: `build.sh --fe` builds all 83 modules, be-java-extensions included, with fe-core's 4476 main sources compiling and 0 Checkstyle violations anywhere. `dependency:list -DincludeScope=runtime` for fe-core is byte-for-byte identical before and after — 395 artifacts — so `fe/lib` does not change. fe-common, fe-type and fe-catalog lose hadoop-aws and its two transitives (wildfly-openssl, analyticsaccelerator-s3) and gain commons-codec. The java-udf plugin directory keeps exactly its 27 jars, and check_plugin_layout.py passes on all eight BE plugins. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XYs8EpL1D6bjufhYSt3mC4 --- fe/be-java-extensions/java-udf/pom.xml | 22 ++++++---- fe/fe-common/pom.xml | 22 +++------- fe/fe-core/pom.xml | 14 +++++++ .../doris/analysis/TableScanParams.java | 6 ++- .../apache/doris/catalog/HdfsResource.java | 8 ++-- .../doris/catalog/HdfsStorageVault.java | 8 ++-- .../security/HadoopAuthConfigKeys.java | 41 +++++++++++++++++++ .../doris/kerberos/AuthenticationConfig.java | 15 ++++--- 8 files changed, 95 insertions(+), 41 deletions(-) create mode 100644 fe/fe-foundation/src/main/java/org/apache/doris/foundation/security/HadoopAuthConfigKeys.java diff --git a/fe/be-java-extensions/java-udf/pom.xml b/fe/be-java-extensions/java-udf/pom.xml index 60a98854ba2ad3..6d323f350e0d19 100644 --- a/fe/be-java-extensions/java-udf/pom.xml +++ b/fe/be-java-extensions/java-udf/pom.xml @@ -68,12 +68,16 @@ under the License. dependency that is only reached transitively is one refactor away from disappearing. The exclusions are what fe-common carries for the FE that no Java function executor can - reach: netty, the S3 filesystem, protobuf, jackson, RoaringBitmap. They were found by - walking jdeps' class-level edges out of this module and taking the jars with nothing - reachable in them, then dropping the ones that are reached by ServiceLoader instead - (the logging providers) or that exist to be extended rather than called + reach: netty, protobuf, jackson, RoaringBitmap, commons-io, commons-codec. They were + found by walking jdeps' class-level edges out of this module and taking the jars with + nothing reachable in them, then dropping the ones that are reached by ServiceLoader + instead (the logging providers) or that exist to be extended rather than called (hive-udf-shade). Verified the only way it can be verified, by loading the plugin out - of a real deployment directory with them gone. + of a real deployment directory with them gone. commons-codec is the one that used to be + excluded for free: fe-common reached it only through hadoop-common, at provided scope, + so it never arrived here; now that fe-common declares it for its own Hll, this list has + to say so or the jar walks in. The S3 filesystem (hadoop-aws) went the other way - it + left fe-common altogether - so it no longer needs naming here. --> org.apache.doris @@ -84,10 +88,6 @@ under the License. io.netty netty-all - - org.apache.hadoop - hadoop-aws - com.google.protobuf protobuf-java @@ -108,6 +108,10 @@ under the License. commons-io commons-io + + commons-codec + commons-codec + org.apache.logging.log4j log4j-iostreams diff --git a/fe/fe-common/pom.xml b/fe/fe-common/pom.xml index aa6e4631666ad7..83550350dae8e4 100644 --- a/fe/fe-common/pom.xml +++ b/fe/fe-common/pom.xml @@ -104,20 +104,12 @@ under the License. org.projectlombok lombok + - org.apache.hadoop - hadoop-common - - - commons-collections - commons-collections - - - org.apache.commons - commons-compress - - - provided + commons-codec + commons-codec + + io.dropwizard.metrics + metrics-core + + + broke CREATE FUNCTION for every hive-derived UDF at runtime. + + (2) It is also the provider of org.apache.hadoop.hive.ql.* for four connector plugins. + Three of them (paimon, hive, iceberg) bundle no hive-exec class at all; the fourth, + hudi, bundles one subset: fe-connector-hudi unpacks this artifact's ql/io/parquet/** + (156 classes) into its own jar, for the mapred/mapreduce hierarchy its pom explains. + Every other hive-exec reference they carry resolves to this jar through their + classloader - parent-first today via + ConnectorPluginManager.CONNECTOR_PARENT_FIRST_PREFIXES, which also makes the kernel's + copy of that hudi subset the one that loads; plain child-first fallback if that prefix + ever goes away, since dropping it changes the delegation order but not the + reachability. Measured over the built plugin zips (referenced minus self-carried, cut + against this artifact's class list): paimon 15 - paimon-hive's PaimonStorageHandler + implements ql.metadata.HiveStorageHandler, HiveCatalog.createView calls + ql.metadata.Table.getEmptyTable, HiveUtils uses ql.io.sarg.ConvertAstToSearchArg; hudi + 46 - hudi-hadoop-mr reaches 25 (HoodieCombineHiveInputFormat's ql.io.HiveInputFormat, + CombineHiveRecordReader and IOContextMap, the ORC branch of HoodieInputFormatUtils, + the ql.plan descriptors), and the unpacked parquet subset itself reaches 33 + (ql.exec.Utilities, ql.io.HiveFileFormatUtils, ql.exec.vector.*, + llap.LlapCacheAwareFs, the same descriptors), so carrying that subset made the plugin + depend on MORE of this jar, not less; hive and iceberg 3 each - the bundled Aliyun DLF + metastore client calls ql.session.SessionState.get(). Swapping this for the 10-class + be-java-extensions/hive-udf-shade compiles green and breaks those paths at runtime, the + same shape of failure as the hive-catalog-shade removal in (1). Making the plugins + self-sufficient is the prerequisite for shrinking this, not a follow-up to it. --> org.apache.hive hive-exec diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java index 4eddb0563460d3..bd77adbe70a099 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java @@ -98,9 +98,22 @@ public class ConnectorPluginManager { // // Parent-first is a delegation ORDER, not an exclusive claim: ChildFirstClassLoader falls back // to the plugin's own jars for anything the parent lacks. So org.apache.hadoop.hbase.* (hudi) - // and org.apache.hadoop.hive.* still come from the plugin -- FE carries hive-exec:core, the - // plugins carry hive-metastore, and the class names do not intersect. Everything else the - // plugins bundle under this namespace does change provider: hadoop-common/auth/annotations/ + // still comes from the plugin. + // + // org.apache.hadoop.hive.* reads like the same case and is the opposite one. For paimon, hive + // and iceberg the class names genuinely do not intersect -- FE carries hive-exec:core, they + // carry hive-metastore -- but that empty intersection is because they bundle no hive-exec class + // at all, so every org.apache.hadoop.hive.ql.* reference they make resolves to the FE's copy, + // here or on the fallback path. hudi does bundle one subset, hive-exec's ql/io/parquet/** + // unpacked into its own jar (see fe-connector-hudi/pom.xml), which this prefix shadows with + // the kernel's copy today; and that subset itself reaches a further 33 classes only the kernel + // has. Measured over the built plugin zips: paimon 15 such classes, hudi 46, hive and iceberg + // 3 each; the breakdown is on the hive-exec dependency in fe-core/pom.xml. fe/lib's + // hive-exec:core is therefore part of the plugin contract, not only of the CREATE FUNCTION + // one, and narrowing it to the UDF base classes needs those plugins made self-sufficient first. + // + // Everything else the plugins bundle under this namespace does change provider: + // hadoop-common/auth/annotations/ // hdfs-client/aws, hadoop-shaded-guava and -protobuf, and the huaweicloud fs.obs.* classes // (paimon), which the FE kernel ships too. All of them are the same artifact at the same // version on both sides, and both versions are pinned in fe/pom.xml -- hadoop.version is From 17dedb6a529cedc39e3f96aacbee7f9c460b54df Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 10 Sep 2026 16:36:26 +0800 Subject: [PATCH 03/19] [improvement](fe) Drop fe-kerberos from fe-core and guard the kernel against Hadoop fe-kerberos is a 15-class module that splits cleanly in two: an 8-class Hadoop half (UserGroupInformation logins, AuthenticationConfig(Configuration), HadoopAuthenticator and friends) and a Hadoop-free remainder. fe-core used exactly one type out of it - ExecutionAuthenticator, which the module's own javadoc calls a migration bridge over the fe-foundation interface of the same name - and always with the passthrough implementation: no fe-core code path runs inside a Kerberos context. The Hadoop half is used only by the hive, hudi, iceberg and paimon connector plugins, each of which declares fe-kerberos itself. Depending on it from fe-core put those 8 classes in fe/lib, where they can never link, since the FE kernel does not ship the Hadoop they reference. Lazy resolution keeps that harmless today, and it leaves a compiles-green, fails-at-runtime path open: a future caller picking an overload whose signature mentions no Hadoop type gets past javac on the strength of fe-kerberos.jar being on the classpath, and only then hits NoClassDefFoundError on org/apache/hadoop/security/UserGroupInformation. So fe-core now takes ExecutionAuthenticator straight from fe-foundation (org.apache.doris.foundation.security) and drops the dependency. The two anonymous empty implementations become ExecutionAuthenticator.DIRECT, the constant fe-foundation provides for exactly this - the foundation interface leaves execute(Callable) abstract rather than defaulting it to a passthrough. The change is source-compatible for callers (both interfaces have the same execute(Callable)/execute(Runnable) shape, and the two call sites just run a lambda) and carries no metadata risk: ExternalCatalog.executionAuthenticator has no @SerializedName, so HiddenAnnotationExclusionStrategy keeps it out of persistence entirely. Also removes what the dependency was masking: ThreadPoolManager's three *WithPreAuth members and ExternalCatalog.threadPoolWithPreAuth, its shutdown and its getter. The field never had an assignment anywhere in the repo and the methods had no callers - dead since the connectors were pluginised. FeCoreHasNoHadoopClassesTest is the new guard, and the reason this is worth a commit rather than a cleanup. It scans fe-core's own compiled output and fails on a constant-pool reference to org/apache/hadoop/ or org/apache/doris/kerberos/, which is the form a dependency actually takes - it catches the reference no import reveals, such as an inherited supertype or a type that appears only in a descriptor. It matches the JVM internal spelling only, since ConnectorPluginManager and FileSystemPluginManager legitimately hold "org.apache.hadoop." as a parent-first classloader-policy string. Two assertions keep it from passing vacuously: it must scan more than a thousand classes, and it must find a positive control. Verified in both directions - green today (0 offenders in 6176 classes), red with the exact class named when a probe holding a new org.apache.hadoop.conf.Configuration() is compiled into fe-core. fe/lib loses fe-kerberos.jar; the four connector plugin zips still carry it, as they declare it directly. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XYs8EpL1D6bjufhYSt3mC4 --- fe/fe-core/pom.xml | 16 +- .../doris/common/ThreadPoolManager.java | 45 ----- .../connector/DefaultConnectorContext.java | 8 +- .../doris/datasource/ExternalCatalog.java | 22 +-- .../common/FeCoreHasNoHadoopClassesTest.java | 182 ++++++++++++++++++ ...nnectorContextBackendStoragePropsTest.java | 4 +- ...DefaultConnectorContextFileSystemTest.java | 4 +- ...faultConnectorContextNormalizeUriTest.java | 4 +- ...faultConnectorContextStoragePropsTest.java | 4 +- 9 files changed, 211 insertions(+), 78 deletions(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/common/FeCoreHasNoHadoopClassesTest.java diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml index 273f13604b76cb..ff12c08f831b9b 100644 --- a/fe/fe-core/pom.xml +++ b/fe/fe-core/pom.xml @@ -173,11 +173,17 @@ under the License. ${project.groupId} fe-foundation - - ${project.groupId} - fe-kerberos - ${project.version} - + ${project.groupId} fe-common diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/ThreadPoolManager.java b/fe/fe-core/src/main/java/org/apache/doris/common/ThreadPoolManager.java index 6cbdab0a39993e..c7afa90602198d 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/common/ThreadPoolManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/common/ThreadPoolManager.java @@ -18,7 +18,6 @@ package org.apache.doris.common; -import org.apache.doris.kerberos.ExecutionAuthenticator; import org.apache.doris.metric.Metric; import org.apache.doris.metric.Metric.MetricUnit; import org.apache.doris.metric.MetricLabel; @@ -144,17 +143,6 @@ public static ThreadPoolExecutor newDaemonFixedThreadPool(int numThread, poolName, needRegisterMetric); } - public static ThreadPoolExecutor newDaemonFixedThreadPoolWithPreAuth( - int numThread, - int queueSize, - String poolName, - boolean needRegisterMetric, - ExecutionAuthenticator preAuth) { - return newDaemonThreadPoolWithPreAuth(numThread, numThread, KEEP_ALIVE_TIME, TimeUnit.SECONDS, - new LinkedBlockingQueue<>(queueSize), new BlockedPolicy(poolName, 60), - poolName, needRegisterMetric, preAuth); - } - public static ThreadPoolExecutor newDaemonFixedThreadPool(int numThread, int queueSize, String poolName, int timeoutSeconds, boolean needRegisterMetric) { @@ -248,39 +236,6 @@ private static ThreadFactory namedThreadFactory(String poolName) { .build(); } - public static ThreadPoolExecutor newDaemonThreadPoolWithPreAuth( - int corePoolSize, - int maximumPoolSize, - long keepAliveTime, - TimeUnit unit, - BlockingQueue workQueue, - RejectedExecutionHandler handler, - String poolName, - boolean needRegisterMetric, - ExecutionAuthenticator preAuth) { - ThreadFactory threadFactory = namedThreadFactoryWithPreAuth(poolName, preAuth); - ThreadPoolExecutor threadPool = new ThreadPoolExecutor(corePoolSize, maximumPoolSize, - keepAliveTime, unit, workQueue, threadFactory, handler); - if (needRegisterMetric) { - nameToThreadPoolMap.put(poolName, threadPool); - } - return threadPool; - } - - private static ThreadFactory namedThreadFactoryWithPreAuth(String poolName, ExecutionAuthenticator preAuth) { - return new ThreadFactoryBuilder() - .setDaemon(true) - .setNameFormat(poolName + "-%d") - .setThreadFactory(runnable -> new MoreFieldsThread(() -> { - try { - preAuth.execute(runnable); - } catch (Exception e) { - throw new RuntimeException(e); - } - })) - .build(); - } - private static class PriorityThreadPoolExecutor extends ThreadPoolExecutor { private final Comparator comparator; diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java index 810846c19a1734..16f218179b2019 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java @@ -40,9 +40,9 @@ import org.apache.doris.filesystem.FileIterator; import org.apache.doris.filesystem.FileSystem; import org.apache.doris.filesystem.Location; +import org.apache.doris.foundation.security.ExecutionAuthenticator; import org.apache.doris.fs.FileSystemFactory; import org.apache.doris.fs.SpiSwitchingFileSystem; -import org.apache.doris.kerberos.ExecutionAuthenticator; import org.apache.doris.system.Backend; import org.apache.doris.thrift.BackendService; import org.apache.doris.thrift.TNetworkAddress; @@ -86,8 +86,6 @@ public class DefaultConnectorContext implements ConnectorContext, ConnectorStora private static final Logger LOG = LogManager.getLogger(DefaultConnectorContext.class); - private static final ExecutionAuthenticator NOOP_AUTH = new ExecutionAuthenticator() {}; - private final String catalogName; private final long catalogId; private final Map environment; @@ -124,7 +122,7 @@ public void afterRequest() { }; public DefaultConnectorContext(String catalogName, long catalogId) { - this(catalogName, catalogId, () -> NOOP_AUTH); + this(catalogName, catalogId, () -> ExecutionAuthenticator.DIRECT); } /** @@ -137,7 +135,7 @@ public static DefaultConnectorContext forCatalogCreationValidation(String catalo Map rawStorageProperties) { Map rawSnapshot = Collections.unmodifiableMap( new HashMap<>(Objects.requireNonNull(rawStorageProperties, "rawStorageProperties"))); - return new DefaultConnectorContext(catalogName, catalogId, () -> NOOP_AUTH, + return new DefaultConnectorContext(catalogName, catalogId, () -> ExecutionAuthenticator.DIRECT, Collections::emptyMap, () -> new HashMap<>(rawSnapshot)); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java index ab51108b40aeb5..6d5c3c4627122e 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java @@ -34,7 +34,6 @@ import org.apache.doris.common.DdlException; import org.apache.doris.common.FeConstants; import org.apache.doris.common.Pair; -import org.apache.doris.common.ThreadPoolManager; import org.apache.doris.common.UserException; import org.apache.doris.common.Version; import org.apache.doris.common.util.Util; @@ -50,7 +49,7 @@ import org.apache.doris.datasource.plugin.PluginDrivenExternalDatabase; import org.apache.doris.datasource.test.TestExternalCatalog; import org.apache.doris.datasource.test.TestExternalDatabase; -import org.apache.doris.kerberos.ExecutionAuthenticator; +import org.apache.doris.foundation.security.ExecutionAuthenticator; import org.apache.doris.nereids.trees.plans.commands.info.CreateTableInfo; import org.apache.doris.persist.TruncateTableInfo; import org.apache.doris.persist.gson.GsonPostProcessable; @@ -80,7 +79,6 @@ import java.util.Map; import java.util.Optional; import java.util.Set; -import java.util.concurrent.ThreadPoolExecutor; import java.util.concurrent.atomic.AtomicBoolean; import java.util.function.Function; import java.util.stream.Collectors; @@ -173,7 +171,6 @@ public abstract class ExternalCatalog protected FeMetaCacheEntry> databases; protected transient IdNameIndex dbIdNameIndex = new IdNameIndex("external database"); protected ExecutionAuthenticator executionAuthenticator; - protected ThreadPoolExecutor threadPoolWithPreAuth; private volatile boolean isInitializing = false; @@ -188,13 +185,15 @@ public ExternalCatalog(long catalogId, String name, InitCatalogLog.Type logType, } /** - * Initializes the PreExecutionAuthenticator instance. - * This method ensures that the authenticator is created only once in a thread-safe manner. - * If additional authentication logic is required, it should be extended and implemented in subclasses. + * Initializes this catalog's {@link ExecutionAuthenticator}, once, in a thread-safe manner. + * The kernel's own answer is always the passthrough {@link ExecutionAuthenticator#DIRECT}: no + * fe-core path runs inside a Kerberos context, and the interface comes from fe-foundation so that + * fe-core needs no Hadoop-bearing module on its classpath to say so. A subclass that does have an + * authentication context overrides this and supplies its own. */ protected synchronized void initPreExecutionAuthenticator() { if (executionAuthenticator == null) { - executionAuthenticator = new ExecutionAuthenticator(){}; + executionAuthenticator = ExecutionAuthenticator.DIRECT; } } @@ -961,9 +960,6 @@ private void closeResourcesQuietly(String lifecycleStage) { } protected void closeResources() { - if (threadPoolWithPreAuth != null) { - ThreadPoolManager.shutdownExecutorService(threadPoolWithPreAuth); - } if (null != executionAuthenticator) { executionAuthenticator = null; } @@ -1693,10 +1689,6 @@ public TransactionManager getTransactionManager() { return transactionManager; } - public ThreadPoolExecutor getThreadPoolWithPreAuth() { - return threadPoolWithPreAuth; - } - /** * Check if an external view exists. * @param dbName diff --git a/fe/fe-core/src/test/java/org/apache/doris/common/FeCoreHasNoHadoopClassesTest.java b/fe/fe-core/src/test/java/org/apache/doris/common/FeCoreHasNoHadoopClassesTest.java new file mode 100644 index 00000000000000..5ba41b86e5875d --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/common/FeCoreHasNoHadoopClassesTest.java @@ -0,0 +1,182 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.common; + +import org.apache.doris.catalog.Env; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Set; +import java.util.TreeSet; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +/** + * The FE kernel must carry no Hadoop. + * + *

Hadoop belongs to the filesystem and connector plugins, which load it through their own + * classloaders; fe-core itself must not reference a single {@code org.apache.hadoop} type. Removing + * the jars from {@code fe/lib} is only half of that — the other half is that no fe-core class names + * one, because javac needs nothing but the class on the compile classpath and a reference that + * compiles today keeps compiling after the jar leaves, failing instead at runtime with + * {@code NoClassDefFoundError} on whichever code path first reaches it. + * + *

This scans fe-core's own compiled output rather than its source: a constant-pool entry is the + * form a dependency actually takes, so it catches the reference no import reveals — a supertype + * inherited from another module, a type that only appears in a method descriptor, a synthetic + * bridge. + * + *

Both spellings are needles. JVM internal form ({@code org/apache/hadoop/}) is how class + * references and field/method descriptors are written, and covers everything javac links. The dotted + * form is how a class name reaches {@code Class.forName} or {@code loadClass}, and a reflective load + * compiles against nothing at all: it would pass an internal-form-only scan and then fail at runtime, + * on whichever code path first reaches it, once the kernel jars are gone. fe-core has reflective + * loading paths, so the dotted spelling has to be a needle too. + * + *

{@link #POLICY_OWNERS} is the exception, and the only one: two classes hold + * {@code "org.apache.hadoop."} as a parent-first classloader-policy string, which is a statement + * about where plugins load Hadoop from rather than a use of it. They are allowlisted by name and the + * test asserts each of them still carries the string, so an allowlist entry cannot outlive the reason + * for it. + * + *

{@code org/apache/doris/kerberos/} is a needle too. That module splits into a Hadoop half + * (UGI logins, {@code AuthenticationConfig(Configuration)}) and a Hadoop-free remainder, and its + * Hadoop half would sit in {@code fe/lib} as classes that can never link there. fe-core takes the + * one interface it needs, {@code ExecutionAuthenticator}, from fe-foundation instead, and + * {@code fe-core/pom.xml} no longer depends on fe-kerberos at all. + */ +public class FeCoreHasNoHadoopClassesTest { + + /** JVM internal form: how class references and descriptors are spelled in the constant pool. */ + private static final List FORBIDDEN = Arrays.asList( + "org/apache/hadoop/", + "org/apache/doris/kerberos/"); + + /** Dotted form: how a class name is spelled when it is loaded reflectively. */ + private static final List FORBIDDEN_DOTTED = Arrays.asList( + "org.apache.hadoop.", + "org.apache.doris.kerberos."); + + /** + * The classes allowed to name a forbidden package in dotted form, because naming it is their job: + * both hold {@code "org.apache.hadoop."} in the parent-first prefix list they hand their plugin + * classloaders. Exact class files, so a policy string that migrates into a lambda or a helper has + * to be re-approved rather than inherited. + */ + private static final Set POLICY_OWNERS = new LinkedHashSet<>(Arrays.asList( + "org/apache/doris/connector/ConnectorPluginManager.class", + "org/apache/doris/fs/FileSystemPluginManager.class")); + + /** + * Present in every build, so finding it proves the scan reads real constant pools instead of + * quietly matching nothing. + */ + private static final String POSITIVE_CONTROL = "org/apache/doris/catalog/Env"; + + @Test + public void feCoreBytecodeReferencesNoHadoop() throws IOException { + Path classesRoot = feCoreClassesRoot(); + List offenders = new ArrayList<>(); + Set policyOwnersSeen = new TreeSet<>(); + boolean controlSeen = false; + int scanned = 0; + + try (Stream tree = Files.walk(classesRoot)) { + List classFiles = tree.filter(p -> p.toString().endsWith(".class")) + .collect(Collectors.toList()); + for (Path classFile : classFiles) { + scanned++; + String name = classesRoot.relativize(classFile).toString().replace(File.separatorChar, '/'); + // ISO-8859-1 is byte-preserving, so this is a byte search spelled as a String search. + String bytecode = new String(Files.readAllBytes(classFile), StandardCharsets.ISO_8859_1); + controlSeen |= bytecode.contains(POSITIVE_CONTROL); + boolean policyOwner = POLICY_OWNERS.contains(name); + String dotted = firstMatch(bytecode, FORBIDDEN_DOTTED); + if (policyOwner && dotted != null) { + policyOwnersSeen.add(name); + } + // A policy owner is excused for the dotted spelling only; linking Hadoop is not part + // of stating where plugins load it from. + String hit = firstMatch(bytecode, FORBIDDEN); + if (hit == null && !policyOwner) { + hit = dotted; + } + if (hit != null) { + offenders.add(name + " -> " + hit); + } + } + } + + Assertions.assertTrue(scanned > 1000, + "expected fe-core to compile into thousands of classes, scanned only " + scanned + + " under " + classesRoot + "; the scan is looking at the wrong directory"); + Assertions.assertTrue(controlSeen, + "the positive control " + POSITIVE_CONTROL + " was not found in any of the " + scanned + + " classes scanned, so this test cannot detect anything"); + // Doubles as the positive control for the dotted needles: if these two stopped matching, the + // dotted half of the scan would be silently matching nothing at all. + Assertions.assertEquals(POLICY_OWNERS, policyOwnersSeen, + "every allowlisted class must still hold a dotted forbidden prefix; one that does not " + + "is either renamed or no longer a classloader-policy owner, and its " + + "allowlist entry has to go with it"); + Assertions.assertTrue(offenders.isEmpty(), + "fe-core must not reference Hadoop (or fe-kerberos, whose Hadoop half cannot link in " + + "fe/lib), in either linked or reflective form. " + offenders.size() + " of " + + scanned + " classes do:\n" + + String.join("\n", offenders.subList(0, Math.min(offenders.size(), 20)))); + } + + /** The first needle present in {@code bytecode}, or null. */ + private static String firstMatch(String bytecode, List needles) { + for (String needle : needles) { + if (bytecode.contains(needle)) { + return needle; + } + } + return null; + } + + /** + * fe-core's compiled output, located through a class that lives in it. Reading it off the + * classloader rather than a hardcoded {@code target/classes} keeps the test correct under any + * build layout that still hands surefire a directory. + */ + private static Path feCoreClassesRoot() { + Path root; + try { + root = Paths.get(Env.class.getProtectionDomain().getCodeSource().getLocation().toURI()); + } catch (Exception e) { + throw new IllegalStateException("cannot locate fe-core's compiled classes", e); + } + Assertions.assertTrue(Files.isDirectory(root), + "expected fe-core's classes as a directory, got " + root + + "; run this from the maven build, not against a packaged jar"); + return root; + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextBackendStoragePropsTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextBackendStoragePropsTest.java index 7bf4d76a35b7aa..f6eb12ffaf3189 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextBackendStoragePropsTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextBackendStoragePropsTest.java @@ -19,7 +19,7 @@ import org.apache.doris.datasource.storage.StorageAdapter; import org.apache.doris.datasource.storage.StorageTypeId; -import org.apache.doris.kerberos.ExecutionAuthenticator; +import org.apache.doris.foundation.security.ExecutionAuthenticator; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -45,7 +45,7 @@ public class DefaultConnectorContextBackendStoragePropsTest { private static final Supplier NOOP_AUTH = - () -> new ExecutionAuthenticator() {}; + () -> ExecutionAuthenticator.DIRECT; /** A context whose storage-props supplier yields a real OSS storage-properties map, built with * the same {@code StorageAdapter.ofAll} machinery a real OSS catalog uses. */ diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextFileSystemTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextFileSystemTest.java index 30e26bcb4dedf0..9b4af5f007121a 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextFileSystemTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextFileSystemTest.java @@ -20,8 +20,8 @@ import org.apache.doris.datasource.storage.StorageAdapter; import org.apache.doris.datasource.storage.StorageTypeId; import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.foundation.security.ExecutionAuthenticator; import org.apache.doris.fs.SpiSwitchingFileSystem; -import org.apache.doris.kerberos.ExecutionAuthenticator; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -38,7 +38,7 @@ */ public class DefaultConnectorContextFileSystemTest { - private static final Supplier NOOP_AUTH = () -> new ExecutionAuthenticator() {}; + private static final Supplier NOOP_AUTH = () -> ExecutionAuthenticator.DIRECT; /** Records close() so the test can assert the engine forwards teardown to the cached filesystem. */ private static final class RecordingFileSystem extends SpiSwitchingFileSystem { diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextNormalizeUriTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextNormalizeUriTest.java index ce698b0ef0f7ad..62fc618fc2cf06 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextNormalizeUriTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextNormalizeUriTest.java @@ -19,7 +19,7 @@ import org.apache.doris.datasource.storage.StorageAdapter; import org.apache.doris.datasource.storage.StorageTypeId; -import org.apache.doris.kerberos.ExecutionAuthenticator; +import org.apache.doris.foundation.security.ExecutionAuthenticator; import org.apache.doris.thrift.TFileType; import org.junit.jupiter.api.Assertions; @@ -48,7 +48,7 @@ public class DefaultConnectorContextNormalizeUriTest { private static final Supplier NOOP_AUTH = - () -> new ExecutionAuthenticator() {}; + () -> ExecutionAuthenticator.DIRECT; /** A context whose storage-props supplier yields a real OSS storage-properties map, built with * the same {@code StorageAdapter.ofAll} machinery a real OSS catalog uses. */ diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextStoragePropsTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextStoragePropsTest.java index dd2eecc73b74b2..b4de8b6e008d16 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextStoragePropsTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextStoragePropsTest.java @@ -22,9 +22,9 @@ import org.apache.doris.filesystem.properties.FileSystemProperties; import org.apache.doris.filesystem.properties.StorageKind; import org.apache.doris.filesystem.spi.FileSystemProvider; +import org.apache.doris.foundation.security.ExecutionAuthenticator; import org.apache.doris.fs.FileSystemFactory; import org.apache.doris.fs.FileSystemPluginManager; -import org.apache.doris.kerberos.ExecutionAuthenticator; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.Assertions; @@ -46,7 +46,7 @@ public class DefaultConnectorContextStoragePropsTest { private static final Supplier NOOP_AUTH = - () -> new ExecutionAuthenticator() {}; + () -> ExecutionAuthenticator.DIRECT; @AfterEach public void resetFactory() { From 32c6cd131b42a91c8bd136736d8c5d18844da919 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 10 Sep 2026 16:53:38 +0800 Subject: [PATCH 04/19] [feature](fe) Add a shared library layer for plugins, and stop a missing dependency from killing FE startup Some libraries cannot be bundled per plugin. A library whose classes inherit from one another across jars has to be loaded once or the JVM refuses the link; one that holds process-wide static state - a client cache, a login context, a first-caller-wins registry - has to be loaded once or two plugins silently stop sharing it; one with a JNI native image can only bind to a single classloader per process. Hadoop is all three at once, which is why the FE kernel currently ships it and both plugin managers list org.apache.hadoop. as parent-first: giving each plugin its own copy is not the same thing duplicated, it is a behavior change. SharedLibraryLayer is where such a library goes instead of the kernel. It turns a root of bundle directories into one classloader that becomes the PARENT of every plugin classloader, so plugins reach it by the ordinary child-first fallback and a plugin carrying its own copy keeps using that one. The layout is the plugin directory convention - //*.jar then //lib/*.jar, bundles by name - which is also how a bundle ships a patched class that also exists in one of its dependency jars: patch in the bundle root, stock jar under lib/. The layer is memoized on the resolved root, which is the point of the class rather than an optimization: two layers over the same jars would mean two copies of every class in them, the exact situation it exists to prevent. A root that does not exist or holds no jar returns the parent unchanged, so a deployment that installs no bundle keeps the classloader graph it has today. A root that exists but cannot be read is raised instead, because degrading that to "nothing installed" turns a permissions mistake into a missing-class failure much later and somewhere else. Nothing calls resolve() yet - this commit adds the mechanism and its tests only, and no existing behavior changes. The second half is the gap that mechanism opens. Once a plugin can expect a dependency from a layer rather than from its own jars, "plugin installed, bundle not" becomes a reachable state, and the JVM reports it as NoClassDefFoundError - an Error, not a ReflectiveOperationException. DirectoryPluginRuntimeManager caught only the latter at the two points where it touches plugin bytecode, so the Error walked out of loadAll, out of initXxxPluginManager, and took FE startup with it. One uninstalled bundle must cost the plugins that need it, not the FE. Both catch sites now include LinkageError, matching what factory.name() and factory.description() already do a few lines below, and the failure message names the class that was missing and says where a dependency is expected to come from - a bundle that was never installed otherwise looks exactly like a broken plugin jar. The two failures land one step apart and both are covered: an absent supertype fails while the class is being resolved, an absent class reached from a static initializer fails at first initialization. The tests reproduce the real reachability rather than simulating it - the fabricated jar omits the dependency and the parent classloader refuses it - and both go red, with the NoClassDefFoundError escaping loadAll, if either catch is narrowed back. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XYs8EpL1D6bjufhYSt3mC4 --- .../loader/DirectoryPluginRuntimeManager.java | 68 +++++- .../extension/loader/SharedLibraryLayer.java | 202 +++++++++++++++++ ...ectoryPluginRuntimeManagerLinkageTest.java | 158 ++++++++++++++ .../loader/SharedLibraryLayerTest.java | 204 ++++++++++++++++++ .../testplugins/AbsentDependencyProbe.java | 30 +++ .../AbsentStaticInitTestPluginFactory.java | 47 ++++ .../AbsentSuperclassTestPluginFactory.java | 44 ++++ .../testplugins/SharedLibraryProbe.java | 30 +++ 8 files changed, 779 insertions(+), 4 deletions(-) create mode 100644 fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/SharedLibraryLayer.java create mode 100644 fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerLinkageTest.java create mode 100644 fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/SharedLibraryLayerTest.java create mode 100644 fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/AbsentDependencyProbe.java create mode 100644 fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/AbsentStaticInitTestPluginFactory.java create mode 100644 fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/AbsentSuperclassTestPluginFactory.java create mode 100644 fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/SharedLibraryProbe.java diff --git a/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManager.java b/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManager.java index 556242738c5c2f..eed9a44622730b 100644 --- a/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManager.java +++ b/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManager.java @@ -104,6 +104,13 @@ */ public class DirectoryPluginRuntimeManager { + /** + * How the JVM reports a class whose initializer has already failed, on every attempt after the + * first. The class was found - it is the initializer that is broken - so this is not a missing + * dependency, and the name it carries is not where the message starts. + */ + private static final String ALREADY_FAILED_TO_INITIALIZE = "Could not initialize class "; + private final ConcurrentMap> handlesByName = new ConcurrentHashMap<>(); private final Object lifecycleLock = new Object(); @@ -267,11 +274,18 @@ private PluginHandle loadFromPluginDir(Path pluginDir, ClassLoader parent, Cl Class discoveredClass; try { discoveredClass = classLoader.loadClass(factoryClassName); - } catch (ReflectiveOperationException e) { + } catch (ReflectiveOperationException | LinkageError e) { + // LinkageError included: defining the factory class resolves its supertypes, so a + // dependency the plugin neither bundles nor inherits from its parent surfaces here as + // NoClassDefFoundError - an Error, not a ReflectiveOperationException. Left uncaught it + // escapes loadAll entirely and takes FE startup down, because one plugin's missing + // dependency is not something the FE can be stopped by. Same reasoning as factory.name() + // and factory.description() below, which already catch it. throw new PluginLoadException( normalizedDir, LoadFailure.STAGE_INSTANTIATE, - "Failed to instantiate factory class '" + factoryClassName + "' in " + normalizedDir, + "Failed to instantiate factory class '" + factoryClassName + "' in " + normalizedDir + + missingClassAdvice(e), e); } @@ -293,11 +307,15 @@ private PluginHandle loadFromPluginDir(Path pluginDir, ClassLoader parent, Cl @SuppressWarnings("unchecked") Class factoryClass = (Class) discoveredClass.asSubclass(factoryType); factory = factoryClass.getDeclaredConstructor().newInstance(); - } catch (ReflectiveOperationException e) { + } catch (ReflectiveOperationException | LinkageError e) { + // newInstance() is where the factory class is first initialized, so a static initializer + // that reaches a missing dependency arrives as ExceptionInInitializerError - again an + // Error, and again one plugin's problem rather than the FE's. throw new PluginLoadException( normalizedDir, LoadFailure.STAGE_INSTANTIATE, - "Failed to instantiate factory class '" + factoryClassName + "' in " + normalizedDir, + "Failed to instantiate factory class '" + factoryClassName + "' in " + normalizedDir + + missingClassAdvice(e), e); } } catch (PluginLoadException e) { @@ -523,6 +541,48 @@ private String discoverSingleFactoryClassName(Class factoryType, ClassLoader return classNames.get(0); } + /** + * Turns a load failure caused by an absent class into an actionable sentence, or "" for any other + * failure. A plugin misses a class either because it does not bundle it or because it expected to + * inherit it from the layer its classloader delegates to, and the message must not leave the reader + * guessing which - a shared library bundle that was never installed looks exactly like a broken + * plugin jar otherwise. + */ + // Package-private so that the JVM messages it has to tell apart can be asserted directly: no + // plugin makes the loader initialize one class twice, so the second-attempt wording below is not + // reachable from loadAll. + static String missingClassAdvice(Throwable failure) { + String missing = missingClassName(failure); + if (missing == null) { + return ""; + } + return ". The class " + missing + " is in neither this plugin's own jars nor its parent" + + " classloader; if it is meant to come from a shared library bundle, check that the" + + " bundle is installed under the FE shared library root"; + } + + /** The absent class named by a NoClassDefFoundError / ClassNotFoundException anywhere in the chain. */ + private static String missingClassName(Throwable failure) { + for (Throwable t = failure; t != null; t = t.getCause() == t ? null : t.getCause()) { + if (!(t instanceof NoClassDefFoundError) && !(t instanceof ClassNotFoundException)) { + continue; + } + String message = t.getMessage(); + if (message == null) { + continue; + } + message = message.trim(); + if (message.isEmpty() || message.startsWith(ALREADY_FAILED_TO_INITIALIZE)) { + // Keep walking: what really went missing, if anything did, is further down the chain. + continue; + } + // NoClassDefFoundError's message is sometimes "a/B (wrong name: c/D)" or carries a + // trailing explanation; the class name is the first token either way. + return message.split("\\s+", 2)[0]; + } + return null; + } + private static void closeClassLoader(ClassLoader classLoader) { if (classLoader == null) { return; diff --git a/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/SharedLibraryLayer.java b/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/SharedLibraryLayer.java new file mode 100644 index 00000000000000..115cb1f22cdac6 --- /dev/null +++ b/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/SharedLibraryLayer.java @@ -0,0 +1,202 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.extension.loader; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.net.MalformedURLException; +import java.net.URL; +import java.nio.file.Files; +import java.nio.file.NoSuchFileException; +import java.nio.file.Path; +import java.nio.file.attribute.BasicFileAttributes; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.Objects; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; +import java.util.function.Predicate; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +/** + * A library layer shared by every plugin classloader of every family. + * + *

Some libraries cannot be bundled per plugin. A library whose classes inherit from one another + * across jars has to be loaded once or the JVM refuses the link; one that holds process-wide static + * state (a client cache, a login context, a first-caller-wins registry) has to be loaded once or two + * plugins silently stop sharing it; one with a JNI native image can only be bound to a single + * classloader per process. Hadoop is all three at once. Giving each plugin its own copy is therefore + * not "the same thing, duplicated" - it changes behavior. + * + *

So such libraries are installed once, under a shared root, and this class turns that root into + * a single classloader that is then the PARENT of every plugin classloader: + * + *

+ *   app classloader (fe/lib)
+ *     └── shared library layer      <- plugins/shared/<bundle>/*.jar + lib/*.jar
+ *           ├── filesystem plugin   (child-first)
+ *           ├── connector plugin    (child-first)
+ *           └── ...
+ * 
+ * + *

The layer is itself child-first with only the mandatory parent-first prefixes, so a bundle uses + * its own dependencies where it has them and falls back to fe/lib otherwise; logging stays the + * kernel's, which keeps the bundle's output in fe.log. Plugins reach the bundle by the ordinary + * child-first fallback: a plugin that carries its own copy of the library keeps using it and is + * simply not sharing, which is the pre-existing behavior for a third-party plugin. + * + *

Layout

+ * + *

{@code //*.jar} then {@code //lib/*.jar}, bundles in name order - + * the same convention {@link DirectoryPluginRuntimeManager} uses for a plugin directory. Root jars + * before {@code lib/} is what lets a bundle ship a patched copy of a class that also exists in one + * of its dependency jars: put the patch in the bundle root and the stock jar under {@code lib/}. + * + *

Absence is normal

+ * + *

A root that does not exist, or holds no jar, yields {@code parent} unchanged, so a deployment + * that installs no bundle keeps exactly the classloader graph it had before this class existed. An + * unreadable root is not that case and is raised, because degrading it to "nothing installed" turns + * a permissions mistake into a missing-class failure much later and somewhere else. + * + *

One instance per root

+ * + *

Memoized on the physical root - symlinks resolved - for the whole process. This is the + * point of the class rather than an optimization: two callers that got two layers over the same jars + * would get two copies of every class in them, which is the situation the layer exists to prevent, + * and a release directory reached both through a {@code current} symlink and through its real path is + * the same jars under two names. The first caller's parent is the one that ends up in the graph - + * there is one FE app classloader, so every caller passes the same one. The layer is never closed; it + * lives as long as the process, like the app classloader. + */ +public final class SharedLibraryLayer { + + private static final ConcurrentMap LAYERS = new ConcurrentHashMap<>(); + + private SharedLibraryLayer() { + } + + /** + * Returns the classloader for the shared bundles under {@code root}, or {@code parent} itself + * when there is nothing to install there. + * + * @param root the shared library root; null is treated as "not configured" + * @param parent the classloader the layer delegates to, normally the FE app classloader + * @throws UncheckedIOException if the root exists but cannot be read + */ + public static ClassLoader resolve(Path root, ClassLoader parent) { + Objects.requireNonNull(parent, "parent"); + if (root == null) { + return parent; + } + Path physical = physicalRoot(root); + if (physical == null) { + return parent; + } + // A mapping function that throws leaves the map untouched, so an unreadable root is raised on + // every call rather than memoized as "nothing installed" until the process is restarted. + return LAYERS.computeIfAbsent(physical, dir -> build(dir, parent)); + } + + /** + * The physical root, or {@code null} when nothing is installed there. {@code toRealPath} rather + * than {@code toAbsolutePath().normalize()}: the latter removes lexical {@code ..} but leaves + * symlinks in place, and two spellings of one directory memoize twice. + */ + private static Path physicalRoot(Path root) { + try { + return root.toRealPath(); + } catch (NoSuchFileException e) { + return null; + } catch (IOException e) { + throw new UncheckedIOException("Failed to resolve shared library root: " + root, e); + } + } + + private static ClassLoader build(Path root, ClassLoader parent) { + if (!isDirectory(root)) { + return parent; + } + List jars = collectBundleJars(root); + if (jars.isEmpty()) { + return parent; + } + URL[] urls = new URL[jars.size()]; + for (int i = 0; i < jars.size(); i++) { + try { + urls[i] = jars.get(i).toUri().toURL(); + } catch (MalformedURLException e) { + throw new IllegalStateException("Invalid shared library jar path: " + jars.get(i), e); + } + } + // Mandatory prefixes only: the layer adds no business parent-first rules of its own. A bundle + // is a library, not a plugin, so nothing here participates in an SPI contract with the kernel. + return new PluginLoader(ClassLoadingPolicy.defaultPolicy().toParentFirstPackages()) + .createClassLoader(urls, parent); + } + + /** Bundle directories in name order; within each, root jars before {@code lib/} jars. */ + private static List collectBundleJars(Path root) { + List jars = new ArrayList<>(); + for (Path bundle : listSorted(root, SharedLibraryLayer::isDirectory)) { + jars.addAll(listSorted(bundle, SharedLibraryLayer::isJar)); + Path lib = bundle.resolve("lib"); + if (isDirectory(lib)) { + jars.addAll(listSorted(lib, SharedLibraryLayer::isJar)); + } + } + return jars; + } + + private static List listSorted(Path dir, Predicate filter) { + try (Stream stream = Files.list(dir)) { + return stream.filter(filter) + .sorted(Comparator.comparing(path -> path.getFileName().toString())) + .collect(Collectors.toList()); + } catch (IOException e) { + throw new UncheckedIOException("Failed to list shared library directory: " + dir, e); + } + } + + private static boolean isDirectory(Path path) { + return is(path, BasicFileAttributes::isDirectory); + } + + private static boolean isJar(Path path) { + return path.getFileName().toString().endsWith(".jar") && is(path, BasicFileAttributes::isRegularFile); + } + + /** + * Whether {@code path} is there and of the given kind. Not {@code Files.isDirectory} / + * {@code Files.isRegularFile}: those answer {@code false} when the attributes cannot be read at + * all, so a bundle directory that can be listed but not searched - readable, not executable - + * would have every jar in it quietly filtered out and the layer would degrade to a no-op, which + * is the fail-quiet this class exists to avoid. Absence stays absence; an I/O failure is raised. + */ + private static boolean is(Path path, Predicate kind) { + try { + return kind.test(Files.readAttributes(path, BasicFileAttributes.class)); + } catch (NoSuchFileException e) { + return false; + } catch (IOException e) { + throw new UncheckedIOException("Failed to read shared library entry: " + path, e); + } + } +} diff --git a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerLinkageTest.java b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerLinkageTest.java new file mode 100644 index 00000000000000..e6657f28d4bc72 --- /dev/null +++ b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerLinkageTest.java @@ -0,0 +1,158 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.extension.loader; + +import org.apache.doris.extension.loader.testplugins.AbsentDependencyProbe; +import org.apache.doris.extension.loader.testplugins.AbsentStaticInitTestPluginFactory; +import org.apache.doris.extension.loader.testplugins.AbsentSuperclassTestPluginFactory; +import org.apache.doris.extension.spi.PluginFactory; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.IOException; +import java.nio.file.Path; +import java.util.Collections; + +/** + * A plugin whose dependency is absent must fail alone. + * + *

This is the shape of "the plugin is installed but the library bundle it needs is not": the + * missing class is reported by the JVM as {@code NoClassDefFoundError}, an {@link Error}, which is + * not a {@link ReflectiveOperationException} and so used to walk straight out of {@code loadAll} and + * out of the FE's startup path with it. One uninstalled bundle must cost the plugins that need it, + * not the FE. + * + *

The two cases are the two places the loader touches plugin bytecode, and they fail one step + * apart: resolving the class (a supertype the jar does not carry) and first initializing it (a + * static initializer that reaches outside the jar). + * + *

The advice sentence itself is asserted separately, against fabricated throwables. The message it + * has to reject - the JVM's wording for a class whose initializer had already failed - only appears on + * the second attempt at initializing one class, and no plugin makes the loader do that: {@code loadAll} + * gives up and closes the classloader on the first. + */ +class DirectoryPluginRuntimeManagerLinkageTest { + + @TempDir + Path tempDir; + + @Test + void testAbsentSuperclassIsAFailureNotAThrow() throws IOException { + Path root = tempDir.resolve("plugins"); + DirectoryPluginRuntimeManagerMetadataTest.createPluginJar( + root.resolve("absent-superclass").resolve("absent-superclass.jar"), + AbsentSuperclassTestPluginFactory.class, "1.0"); + + assertReportsMissingDependency(load(root)); + } + + @Test + void testAbsentStaticInitDependencyIsAFailureNotAThrow() throws IOException { + Path root = tempDir.resolve("plugins"); + DirectoryPluginRuntimeManagerMetadataTest.createPluginJar( + root.resolve("absent-static-init").resolve("absent-static-init.jar"), + AbsentStaticInitTestPluginFactory.class, "1.0"); + + assertReportsMissingDependency(load(root)); + } + + @Test + void testAlreadyFailedInitializationIsNotReportedAsAMissingClass() { + // "Could not initialize class X" means X was found and its initializer threw - a broken plugin, + // not an uninstalled bundle. Read as a missing class, its first token would advertise a class + // named "Could" and send the reader off to look at the shared library root. + String advice = DirectoryPluginRuntimeManager.missingClassAdvice( + new NoClassDefFoundError("Could not initialize class org.example.BrokenFactory")); + + Assertions.assertEquals("", advice, + () -> "an initializer that already failed is not a missing dependency: " + advice); + } + + @Test + void testAMissingClassIsNamedInEveryFormTheJvmReportsItIn() { + Assertions.assertTrue(adviceFor(new NoClassDefFoundError("org/example/Probe")) + .contains("The class org/example/Probe is"), + "the plain form names the class directly"); + Assertions.assertTrue( + adviceFor(new NoClassDefFoundError("org/example/Probe (wrong name: org/other/Probe)")) + .contains("The class org/example/Probe is"), + "the wrong-name form carries a trailing explanation the class name comes before"); + Assertions.assertTrue( + adviceFor(new ExceptionInInitializerError(new ClassNotFoundException("org.example.Probe"))) + .contains("The class org.example.Probe is"), + "a static initializer's first failure wraps the real miss"); + + // An initializer failure on top of a genuine miss: skipping the wording must not stop the walk. + NoClassDefFoundError alreadyFailed = + new NoClassDefFoundError("Could not initialize class org.example.BrokenFactory"); + alreadyFailed.initCause(new NoClassDefFoundError("org/example/Probe")); + Assertions.assertTrue(adviceFor(alreadyFailed).contains("The class org/example/Probe is"), + "the miss below the initializer failure is what the reader needs"); + } + + private static String adviceFor(Throwable failure) { + String advice = DirectoryPluginRuntimeManager.missingClassAdvice(failure); + Assertions.assertTrue(advice.contains("shared library bundle"), + () -> "the advice must say where a missing dependency is expected to come from: " + advice); + return advice; + } + + /** + * Loads {@code root} against a parent that refuses {@link AbsentDependencyProbe}. The fabricated + * jars leave that class out, so the plugin classloader cannot find it in its own jars either - + * exactly the reachability a plugin has when the shared bundle carrying its dependency is not + * installed. + */ + private LoadReport load(Path root) { + ClassLoader hidingParent = new ClassLoader(getClass().getClassLoader()) { + @Override + protected Class loadClass(String name, boolean resolve) throws ClassNotFoundException { + if (AbsentDependencyProbe.class.getName().equals(name)) { + throw new ClassNotFoundException(name + " is withheld by this test"); + } + return super.loadClass(name, resolve); + } + }; + // Assertions.assertDoesNotThrow is the point of the test: before the loader caught LinkageError + // this call threw, and nothing between here and FE startup would have caught it. + return Assertions.assertDoesNotThrow(() -> new DirectoryPluginRuntimeManager().loadAll( + Collections.singletonList(root), + hidingParent, + PluginFactory.class, + null, + DirectoryPluginRuntimeManagerMetadataTest.TEST_GATE)); + } + + private static void assertReportsMissingDependency(LoadReport report) { + Assertions.assertTrue(report.getSuccesses().isEmpty(), "the plugin cannot have loaded"); + Assertions.assertEquals(1, report.getFailures().size()); + + LoadFailure failure = report.getFailures().get(0); + Assertions.assertEquals(LoadFailure.STAGE_INSTANTIATE, failure.getStage()); + Assertions.assertInstanceOf(LinkageError.class, failure.getCause(), + () -> "expected the linkage failure to be preserved as the cause, got " + failure.getCause()); + Assertions.assertTrue( + failure.getMessage().contains(AbsentDependencyProbe.class.getName().replace('.', '/')), + () -> "the message must name the class that is missing: " + failure.getMessage()); + Assertions.assertTrue(failure.getMessage().contains("shared library bundle"), + () -> "the message must say where a missing dependency is expected to come from: " + + failure.getMessage()); + } +} diff --git a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/SharedLibraryLayerTest.java b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/SharedLibraryLayerTest.java new file mode 100644 index 00000000000000..81cb6808c8364b --- /dev/null +++ b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/SharedLibraryLayerTest.java @@ -0,0 +1,204 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.extension.loader; + +import org.apache.doris.extension.loader.testplugins.SharedLibraryProbe; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Assumptions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.IOException; +import java.io.InputStream; +import java.io.UncheckedIOException; +import java.net.URISyntaxException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.jar.JarEntry; +import java.util.jar.JarOutputStream; + +/** + * The shared library layer's contract: absence is a no-op, an unreadable root is not, one layer per + * physical root, and jar order is the layout convention (bundle root before {@code lib/}, bundles by + * name). + * + *

Each test gets its own {@code @TempDir}, which keeps them independent of the process-wide + * memoization the layer deliberately does. + */ +class SharedLibraryLayerTest { + + private static final String PROBE = SharedLibraryProbe.class.getName(); + + @TempDir + Path tempDir; + + @Test + void testAbsentRootYieldsParentUnchanged() { + ClassLoader parent = getClass().getClassLoader(); + Assertions.assertSame(parent, SharedLibraryLayer.resolve(tempDir.resolve("not-installed"), parent)); + } + + @Test + void testNullRootYieldsParentUnchanged() { + ClassLoader parent = getClass().getClassLoader(); + Assertions.assertSame(parent, SharedLibraryLayer.resolve(null, parent)); + } + + @Test + void testRootWithoutJarsYieldsParentUnchanged() throws IOException { + Path root = tempDir.resolve("shared"); + Files.createDirectories(root.resolve("hadoop").resolve("lib")); + Files.createFile(root.resolve("hadoop").resolve("README.txt")); + + ClassLoader parent = getClass().getClassLoader(); + Assertions.assertSame(parent, SharedLibraryLayer.resolve(root, parent), + "a bundle directory carrying no jar must not change the loader graph"); + } + + @Test + void testSameRootYieldsTheSameLayer() throws IOException { + Path root = tempDir.resolve("shared"); + createProbeJar(root.resolve("hadoop").resolve("hadoop-deps.jar")); + + ClassLoader parent = getClass().getClassLoader(); + ClassLoader first = SharedLibraryLayer.resolve(root, parent); + ClassLoader second = SharedLibraryLayer.resolve(root, parent); + + Assertions.assertNotSame(parent, first, "a root holding a jar must produce a layer"); + Assertions.assertSame(first, second, + "two families resolving the same root must share one layer, or they get two copies " + + "of every class in it"); + // The same root spelled differently is still the same root. + Assertions.assertSame(first, SharedLibraryLayer.resolve(root.resolve("hadoop").resolve(".."), parent)); + } + + @Test + void testSymlinkedRootYieldsTheSameLayer() throws IOException { + Path root = tempDir.resolve("shared"); + createProbeJar(root.resolve("hadoop").resolve("hadoop-deps.jar")); + Path alias = tempDir.resolve("current-shared"); + try { + Files.createSymbolicLink(alias, root); + } catch (IOException | UnsupportedOperationException e) { + Assumptions.abort("this filesystem does not support symbolic links: " + e); + return; + } + + ClassLoader parent = getClass().getClassLoader(); + + // A release directory reached through a "current" symlink and through its real path is one + // set of jars. Two layers over it would be two copies of every shared class, and a shared + // library is shared precisely because a second copy of it changes behavior. + Assertions.assertSame(SharedLibraryLayer.resolve(root, parent), + SharedLibraryLayer.resolve(alias, parent), + "the memoization key is the physical root, not the spelling the caller used"); + } + + @Test + void testUnreadableRootIsRaisedRatherThanTakenForAbsent() throws IOException { + Path loop = tempDir.resolve("shared"); + try { + Files.createSymbolicLink(loop, loop); + } catch (IOException | UnsupportedOperationException e) { + Assumptions.abort("this filesystem does not support symbolic links: " + e); + return; + } + + ClassLoader parent = getClass().getClassLoader(); + + // A root that is there but cannot be read is a deployment mistake, and answering it with + // "nothing installed" turns it into a NoClassDefFoundError much later and somewhere else. + Assertions.assertThrows(UncheckedIOException.class, + () -> SharedLibraryLayer.resolve(loop, parent)); + // Nothing was memoized, so repairing the root does not need a restart to take effect. + Assertions.assertThrows(UncheckedIOException.class, + () -> SharedLibraryLayer.resolve(loop, parent)); + } + + @Test + void testLayerLoadsItsOwnCopyChildFirst() throws Exception { + Path root = tempDir.resolve("shared"); + Path jar = root.resolve("hadoop").resolve("hadoop-deps.jar"); + createProbeJar(jar); + + ClassLoader layer = SharedLibraryLayer.resolve(root, getClass().getClassLoader()); + Class loaded = layer.loadClass(PROBE); + + Assertions.assertNotSame(SharedLibraryProbe.class, loaded, + "the layer is child-first, so it must define its own copy rather than delegate"); + Assertions.assertEquals(jar.toRealPath(), codeSourceOf(loaded)); + } + + @Test + void testBundleRootJarPrecedesLibJar() throws Exception { + Path root = tempDir.resolve("shared"); + Path rootJar = root.resolve("hadoop").resolve("hadoop-deps.jar"); + Path libJar = root.resolve("hadoop").resolve("lib").resolve("hadoop-common.jar"); + createProbeJar(rootJar); + createProbeJar(libJar); + + ClassLoader layer = SharedLibraryLayer.resolve(root, getClass().getClassLoader()); + + Assertions.assertEquals(rootJar.toRealPath(), codeSourceOf(layer.loadClass(PROBE)), + "the bundle root is how a bundle ships a patched class that also exists under lib/"); + } + + @Test + void testBundlesResolveInNameOrder() throws Exception { + Path root = tempDir.resolve("shared"); + Path alpha = root.resolve("alpha").resolve("alpha.jar"); + Path beta = root.resolve("beta").resolve("beta.jar"); + // Created out of order, so a passing assertion cannot be filesystem listing order. + createProbeJar(beta); + createProbeJar(alpha); + + ClassLoader layer = SharedLibraryLayer.resolve(root, getClass().getClassLoader()); + + Assertions.assertEquals(alpha.toRealPath(), codeSourceOf(layer.loadClass(PROBE))); + } + + /** + * The jar a class was actually defined from. Physical path on both sides of the comparison: the + * layer builds its URLs from the resolved root, and a temporary directory is itself reached + * through a symlink on macOS. + */ + private static Path codeSourceOf(Class clazz) throws URISyntaxException, IOException { + return Paths.get(clazz.getProtectionDomain().getCodeSource().getLocation().toURI()).toRealPath(); + } + + /** A bundle jar holding one copy of the probe class, so several jars can compete for it. */ + private static void createProbeJar(Path jarPath) throws IOException { + Files.createDirectories(jarPath.getParent()); + String classEntry = PROBE.replace('.', '/') + ".class"; + try (JarOutputStream jar = new JarOutputStream(Files.newOutputStream(jarPath))) { + jar.putNextEntry(new JarEntry(classEntry)); + try (InputStream classBytes = SharedLibraryProbe.class.getClassLoader() + .getResourceAsStream(classEntry)) { + Assertions.assertNotNull(classBytes, "class bytes not found: " + classEntry); + byte[] buffer = new byte[8192]; + int read; + while ((read = classBytes.read(buffer)) != -1) { + jar.write(buffer, 0, read); + } + } + jar.closeEntry(); + } + } +} diff --git a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/AbsentDependencyProbe.java b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/AbsentDependencyProbe.java new file mode 100644 index 00000000000000..a4223688e1de0b --- /dev/null +++ b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/AbsentDependencyProbe.java @@ -0,0 +1,30 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.extension.loader.testplugins; + +/** + * Stands in for a dependency a plugin does not bundle. Loader tests fabricate plugin jars that + * leave this class out and hand the loader a parent that refuses it, which is how a plugin missing + * a shared library reaches the loader in production. + */ +public class AbsentDependencyProbe { + + public static Object marker() { + return "absent-dependency-probe"; + } +} diff --git a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/AbsentStaticInitTestPluginFactory.java b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/AbsentStaticInitTestPluginFactory.java new file mode 100644 index 00000000000000..ac70c1c50f0792 --- /dev/null +++ b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/AbsentStaticInitTestPluginFactory.java @@ -0,0 +1,47 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.extension.loader.testplugins; + +import org.apache.doris.extension.spi.Plugin; +import org.apache.doris.extension.spi.PluginFactory; + +/** + * This class defines cleanly - the omitted dependency appears only inside its static initializer, + * whose first run is the loader's {@code newInstance()}. The failure therefore lands one step later + * than {@link AbsentSuperclassTestPluginFactory}'s, at the other catch site. + */ +public class AbsentStaticInitTestPluginFactory implements PluginFactory { + + private static final Object MARKER = AbsentDependencyProbe.marker(); + + @Override + public String name() { + return "absent-static-init-test"; + } + + @Override + public String description() { + return "Plugin whose static initializer reaches a class that is not on any classpath: " + MARKER; + } + + @Override + public Plugin create() { + return new Plugin() { + }; + } +} diff --git a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/AbsentSuperclassTestPluginFactory.java b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/AbsentSuperclassTestPluginFactory.java new file mode 100644 index 00000000000000..0a038173b9ce35 --- /dev/null +++ b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/AbsentSuperclassTestPluginFactory.java @@ -0,0 +1,44 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.extension.loader.testplugins; + +import org.apache.doris.extension.spi.Plugin; +import org.apache.doris.extension.spi.PluginFactory; + +/** + * Its superclass is the one the fabricated jar omits, so defining this class fails with + * {@code NoClassDefFoundError} - the failure lands while the loader is still resolving the class. + */ +public class AbsentSuperclassTestPluginFactory extends AbsentDependencyProbe implements PluginFactory { + + @Override + public String name() { + return "absent-superclass-test"; + } + + @Override + public String description() { + return "Plugin whose superclass is not on any classpath"; + } + + @Override + public Plugin create() { + return new Plugin() { + }; + } +} diff --git a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/SharedLibraryProbe.java b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/SharedLibraryProbe.java new file mode 100644 index 00000000000000..515501f32e439e --- /dev/null +++ b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/SharedLibraryProbe.java @@ -0,0 +1,30 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.extension.loader.testplugins; + +/** + * Copied verbatim into more than one fabricated shared-bundle jar so a test can ask which jar the + * layer actually loaded it from, via the class's code source. It has no dependencies of its own, + * so nothing but jar order decides the answer. + */ +public class SharedLibraryProbe { + + public static String id() { + return "shared-library-probe"; + } +} From b9fc9a88873e8557ce61bbb9d42e50ed3b52c819 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 10 Sep 2026 17:14:46 +0800 Subject: [PATCH 05/19] [feature](fe) Build Hadoop as one shared library bundle for the FE's plugins Nine FE plugins need Hadoop and today each carries its own copy: 175MB across the plugin zips, nine independent FileSystem.CACHEs, nine UserGroupInformation logins, and a jindo native image that can only bind to one classloader per process. SharedLibraryLayer already knows how to load one such bundle for all of them; this is the bundle. fe-hadoop-runtime has no sources. It exists so the bundle has one dependency resolution and one artifact list, which is what turns "every plugin gets the same Hadoop" from a coincidence of nine poms agreeing into something checkable. It resolves what fe-filesystem-hdfs-base resolves - hadoop-client and hadoop-auth with the same exclusions, plus hadoop-common directly, since hadoop-client is the slim client POM and prunes com.jcraft:jsch, which the connector plugins ship today - and its versions come from the parent's dependencyManagement, so bundle and plugins cannot drift. Membership is an allowlist, not the resolved closure, and the two boundaries are in the assembly descriptor at length. What must be shared: classes that inherit across jars, plus the process-wide state that is what "two plugins talk to one cluster" actually means; hadoop-hdfs-client is in because hdfs:// resolves through META-INF/services rather than a fs.hdfs.impl entry, and the patched FileSystem scans those with its own loader. What must not be: anything reachable through fs..impl stays with the plugin that configures it, hadoop-aws above all, because the filesystem plugins take software.amazon.awssdk parent-first from fe/lib on purpose. General libraries - jackson, guava, netty, commons-* - stay out because fe/lib already has them at the version fe-core arbitrated and the layer falls back there; a second copy is how a shared layer starts answering with different versions than the kernel. The bundle produced here is exactly the set the nine plugins carry today: 28 jars in lib/, same filenames, therefore same versions, plus hadoop-deps.jar at the root, where the layer's root-before-lib ordering makes the Doris-patched FileSystem win over hadoop-common's. An allowlist fails silently when it is too narrow, so the module asserts the shape it produced: the patched jar is at the root, Hadoop made it into lib/, and nothing Doris-owned did. Nothing reads the bundle yet. build.sh unpacks it to fe/plugins/shared/hadoop/ and stops there - plugins/shared/ is not on the FE CLASSPATH and no plugin manager scans it, so this release ships the bundle alongside the kernel's Hadoop rather than instead of it. Verified by starting an FE on the built output: 14 filesystem plugins, 9 connector plugins and 2 authorization plugins load with zero failures, and not one log line mentions the new directory. fe-hadoop-runtime has to be named in build.sh's module list explicitly: nothing depends on it, so -am does not reach it, and a module that is never built leaves the deploy step silently unpacking whatever zip an older build left behind. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XYs8EpL1D6bjufhYSt3mC4 --- build.sh | 28 +++ .../connector/ConnectorPluginManager.java | 18 +- fe/fe-hadoop-runtime/pom.xml | 176 ++++++++++++++++++ .../src/main/assembly/hadoop-runtime-zip.xml | 122 ++++++++++++ fe/pom.xml | 1 + 5 files changed, 339 insertions(+), 6 deletions(-) create mode 100644 fe/fe-hadoop-runtime/pom.xml create mode 100644 fe/fe-hadoop-runtime/src/main/assembly/hadoop-runtime-zip.xml diff --git a/build.sh b/build.sh index 55c5bacb919001..1d1c69ed5fd5eb 100755 --- a/build.sh +++ b/build.sh @@ -795,6 +795,10 @@ if [[ "${BUILD_FE}" -eq 1 ]]; then modules+=("fe-extension-spi") modules+=("fe-extension-loader") modules+=("fe-core") + # Hadoop as one shared library bundle for the plugins, unpacked into plugins/shared/hadoop/ + # below. Nothing depends on it, so -am does not reach it; it has to be named here or the zip is + # simply never built and the deploy step below silently ships whatever an older build left. + modules+=("fe-hadoop-runtime") # Filesystem API and SPI plugin modules (loaded at runtime as plugins) modules+=("fe-filesystem/fe-filesystem-api") modules+=("fe-filesystem/fe-filesystem-spi") @@ -1269,6 +1273,30 @@ if [[ "${BUILD_FE}" -eq 1 ]]; then mkdir -p "${DORIS_OUTPUT}/fe/plugins/hadoop_conf/" mkdir -p "${DORIS_OUTPUT}/fe/plugins/java_extensions/" + # Shared library bundles: libraries that every plugin needing them must load from ONE place, + # because their classes inherit across jars and their static state (FileSystem.CACHE, the UGI + # login, the FileSystem service registry) is what makes two plugins agree about one cluster. + # SharedLibraryLayer turns this directory into the parent classloader of every filesystem and + # connector plugin. Deliberately at plugins/ level and not under plugins/filesystem or + # plugins/connector: DirectoryPluginRuntimeManager treats every direct subdirectory of those as + # a plugin, and would report this one as a plugin that failed to load. + # NOTE: plugins/shared/ is NOT added to the FE CLASSPATH - it is loaded by FE, not by the JVM + # launcher. See bin/start_fe.sh. + # Created only when there is a bundle to put in it - the mkdir below already builds the whole + # chain. An empty plugins/shared would occupy that name in Config.plugin_dir for nothing. + SHARED_LIB_DIR="${DORIS_OUTPUT}/fe/plugins/shared" + HADOOP_RUNTIME_ZIP="${DORIS_HOME}/fe/fe-hadoop-runtime/target/doris-fe-hadoop-runtime.zip" + if [[ -f "${HADOOP_RUNTIME_ZIP}" ]]; then + # Same rule as the plugin directories: unzip -o overwrites but never removes, so a version + # bump would leave both copies of every versioned jar here and the layer would bind whichever + # the sorted URL order reached first. Clear what the zip owns and unpack fresh. + rm -rf "${SHARED_LIB_DIR}/hadoop/lib" + rm -f "${SHARED_LIB_DIR}/hadoop"/*.jar + mkdir -p "${SHARED_LIB_DIR}/hadoop" + unzip -q -o "${HADOOP_RUNTIME_ZIP}" -d "${SHARED_LIB_DIR}/hadoop/" + fi + unset SHARED_LIB_DIR HADOOP_RUNTIME_ZIP + # Deploy filesystem provider plugins as independent plugin directories # Each sub-directory is one storage backend loaded at runtime by FileSystemPluginManager. FS_PLUGIN_DIR="${DORIS_OUTPUT}/fe/plugins/filesystem" diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java index bd77adbe70a099..cc857b87d0ff9b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java @@ -127,12 +127,18 @@ public class ConnectorPluginManager { // -wins registry -- cannot be frozen by whichever plugin's context loader happens to touch it // first. // - // NOTE: the intended end state is an FE kernel with no hadoop classes at all, every plugin - // bringing its own. At that point the fallback above takes over on its own, and the plugin - // becomes responsible for shipping a patched FileSystem the same way the kernel does today - - // which is what the BE plugins already do, since their loader has no hadoop to delegate to: - // each declares hadoop-deps, and that jar's Doris-Shadows-Classes manifest entry puts it ahead - // of hadoop-common in the plugin directory (see be-java-extensions/jni-bootstrap PluginRuntime). + // NOTE: the intended end state is an FE kernel with no hadoop classes at all. The BE plugins + // got there by each bundling its own hadoop -- their loader has no hadoop to delegate to -- + // with the patched FileSystem placed ahead of hadoop-common by the Doris-Shadows-Classes + // manifest entry of hadoop-deps (see be-java-extensions/jni-bootstrap PluginRuntime). The FE + // cannot copy that: the fs.cache.key patch above only means anything while every catalog + // shares ONE FileSystem.CACHE, and UserGroupInformation's login is process state, so hadoop + // has to be one copy for all plugins. It becomes a shared library bundle instead, + // plugins/shared/hadoop (fe-hadoop-runtime, unpacked by build.sh), which SharedLibraryLayer + // turns into the parent of every plugin classloader, with hadoop-deps.jar at the bundle root + // so the patched FileSystem precedes hadoop-common's there too. Once the kernel carries no + // hadoop this prefix goes: a plugin then resolves org.apache.hadoop.* child-first and, for + // whatever it does not bundle, falls back to that layer rather than to fe/lib. // // Package-private so ConnectorPluginHadoopPatchTest asserts against this list, not a copy of it. static final List CONNECTOR_PARENT_FIRST_PREFIXES = diff --git a/fe/fe-hadoop-runtime/pom.xml b/fe/fe-hadoop-runtime/pom.xml new file mode 100644 index 00000000000000..11247cbd659485 --- /dev/null +++ b/fe/fe-hadoop-runtime/pom.xml @@ -0,0 +1,176 @@ + + + + 4.0.0 + + org.apache.doris + fe + ${revision} + + + fe-hadoop-runtime + pom + Doris FE Hadoop Runtime + + Ships Hadoop as one shared library bundle for the FE's plugins, rather than a copy inside + each plugin that needs it. Produces doris-fe-hadoop-runtime.zip, unpacked by build.sh into + fe/plugins/shared/hadoop/ and turned into a classloader by SharedLibraryLayer. + + This module has no sources. It exists to give the bundle one dependency resolution and one + artifact list, which is what makes "the same Hadoop for every plugin" checkable rather than + a coincidence of nine plugin poms agreeing. + + + + + + ${project.groupId} + hadoop-deps + ${project.version} + + + * + * + + + + + + + org.apache.hadoop + hadoop-client + + + org.slf4j + slf4j-log4j12 + + + log4j + log4j + + + + + org.apache.hadoop + hadoop-auth + + + + org.apache.hadoop + hadoop-common + + + + + doris-fe-hadoop-runtime + + + maven-assembly-plugin + + false + + src/main/assembly/hadoop-runtime-zip.xml + + + + + make-assembly + package + + single + + + + + + maven-antrun-plugin + 3.1.0 + + + verify-bundle-shape + package + + run + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/fe/fe-hadoop-runtime/src/main/assembly/hadoop-runtime-zip.xml b/fe/fe-hadoop-runtime/src/main/assembly/hadoop-runtime-zip.xml new file mode 100644 index 00000000000000..e7a4050648f3fb --- /dev/null +++ b/fe/fe-hadoop-runtime/src/main/assembly/hadoop-runtime-zip.xml @@ -0,0 +1,122 @@ + + + + + hadoop-runtime + + zip + + false + + + + + / + false + runtime + ${artifact.artifactId}.${artifact.extension} + + org.apache.doris:hadoop-deps + + + + + /lib + false + runtime + + + org.apache.hadoop:hadoop-annotations + org.apache.hadoop:hadoop-auth + org.apache.hadoop:hadoop-client + org.apache.hadoop:hadoop-common + org.apache.hadoop:hadoop-hdfs-client + org.apache.hadoop:hadoop-mapreduce-client-common + org.apache.hadoop:hadoop-mapreduce-client-core + org.apache.hadoop:hadoop-mapreduce-client-jobclient + org.apache.hadoop:hadoop-yarn-client + + org.apache.hadoop.thirdparty:hadoop-shaded-guava + org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25 + + org.apache.kerby:kerb-core + org.apache.kerby:kerb-crypto + org.apache.kerby:kerb-util + org.apache.kerby:kerby-asn1 + org.apache.kerby:kerby-config + org.apache.kerby:kerby-pkix + org.apache.kerby:kerby-util + + com.fasterxml.woodstox:woodstox-core + org.codehaus.woodstox:stax2-api + org.apache.commons:commons-configuration2 + commons-net:commons-net + dnsjava:dnsjava + com.jcraft:jsch + com.google.re2j:re2j + + org.apache.curator:curator-client + org.apache.curator:curator-framework + org.apache.curator:curator-recipes + + + + diff --git a/fe/pom.xml b/fe/pom.xml index a04b3ad9838436..5ca799a3266233 100644 --- a/fe/pom.xml +++ b/fe/pom.xml @@ -235,6 +235,7 @@ under the License. fe-extension-spi fe-connector fe-extension-loader + fe-hadoop-runtime fe-sql-parser fe-core hive-udf From b07c2f5f8507b1c0c46244185f1c676845801151 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 15 Sep 2026 21:49:49 +0800 Subject: [PATCH 06/19] [fix](fe) Bound the plugin loader's cause walk, name only real class names, and carry the cause into the failure message Review of the loader change in this branch found four defects in what a plugin that fails to link leaves behind. The cause-chain walk in missingClassName had no bound: the `t.getCause() == t` guard is dead against java.lang.Throwable (getCause() already returns null for a self-cause), and a two-node cycle is legal Java - a.initCause(b) after b was built with cause a. A plugin constructor throwing such a chain spun FE startup forever under the loader's lifecycle lock, before any port was open. The walk is now identity-bounded, in both the advice and the new cause summary. The walk read the first whitespace token of any NoClassDefFoundError or ClassNotFoundException message as the class name. Plugin code wraps lookups in sentences ("Cannot load driver class com.x.Y" advertised a missing class named "Cannot"), and the JVM's own "a/B (wrong name: c/D)" form means the class WAS found under the wrong jar path - a broken jar, for which the shared bundle is exactly the wrong place to send the reader. Only a message that is one class name in the binary or internal spelling is read as a miss; a sentence-shaped node is skipped in favour of the JDK's own node below it; the wrong-name form gets its own sentence. For every LinkageError that is not a missing class - a static initializer throwing IllegalStateException, a VerifyError - the failure message carried nothing: ExceptionInInitializerError's message is null, so a consumer recording the message alone (the authorization manager's rejection list, any log line that consumed the throwable through a placeholder) recorded a bare class name. The innermost cause is now appended to the LoadFailure message. Both catch sites caught ReflectiveOperationException | LinkageError while their comment appealed to factory.name()'s RuntimeException | LinkageError precedent. asSubclass() throws ClassCastException for a service file naming a class that is not a factory, and defineClass throws SecurityException for a signed package the plugin also ships unsigned classes into; neither was caught, so the classloader leaked and the exception reached FE startup. Both sites now catch RuntimeException too, and the ClassCastException case names the factory type that was expected. The comments that described failures as always arriving as ExceptionInInitializerError, and the "Could not initialize class" wording as unreachable from loadAll, said what JDK 17 does not do: an Error thrown by an initializer propagates unwrapped (JLS 12.4.2), and the second-attempt wording is reachable through a parent-first class whose initializer failed under an earlier plugin. Both are reworded; the catch shape they justify is unchanged. Tests: a cyclic chain is walked once (under a timeout, so a regression hangs the test rather than the suite); an initializer that throws a plain exception yields a failure whose message carries the reason; a service file naming a non-factory class is a load failure, not a throw; the sentence-shaped, trailing-dot and wrong-name messages are negative controls, with the JDK node below a sentence still found. Co-Authored-By: Claude Opus 5 --- .../loader/DirectoryPluginRuntimeManager.java | 105 ++++++++++++----- ...ectoryPluginRuntimeManagerLinkageTest.java | 110 ++++++++++++++++-- ...ctoryPluginRuntimeManagerMetadataTest.java | 7 +- .../loader/testplugins/NotAFactory.java | 28 +++++ .../ThrowingStaticInitTestPluginFactory.java | 54 +++++++++ 5 files changed, 265 insertions(+), 39 deletions(-) create mode 100644 fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/NotAFactory.java create mode 100644 fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/ThrowingStaticInitTestPluginFactory.java diff --git a/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManager.java b/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManager.java index eed9a44622730b..2fe41a3d0f005c 100644 --- a/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManager.java +++ b/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManager.java @@ -31,12 +31,16 @@ import java.util.Collections; import java.util.Comparator; import java.util.Enumeration; +import java.util.IdentityHashMap; import java.util.List; import java.util.Objects; import java.util.Optional; +import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentMap; import java.util.jar.JarFile; +import java.util.regex.Matcher; +import java.util.regex.Pattern; import java.util.stream.Collectors; import java.util.stream.Stream; @@ -110,6 +114,15 @@ public class DirectoryPluginRuntimeManager { * dependency, and the name it carries is not where the message starts. */ private static final String ALREADY_FAILED_TO_INITIALIZE = "Could not initialize class "; + /** + * A JVM {@code NoClassDefFoundError} message whose whole text is one class name, in either the binary + * ({@code org.example.Probe$Inner}) or the internal ({@code org/example/Probe$Inner}) spelling. Only + * such a message is read as "this class is missing"; anything sentence-shaped is plugin wording. + */ + private static final Pattern CLASS_NAME_MESSAGE = Pattern.compile("[\\w$]+([./][\\w$]+)*"); + /** The JVM's "found under the wrong path" wording: {@code a/B (wrong name: c/D)}. */ + private static final Pattern WRONG_NAME_MESSAGE = + Pattern.compile("([\\w$]+(?:[./][\\w$]+)*)\\s*\\(wrong name:\\s*([\\w$]+(?:[./][\\w$]+)*)\\)"); private final ConcurrentMap> handlesByName = new ConcurrentHashMap<>(); private final Object lifecycleLock = new Object(); @@ -274,18 +287,20 @@ private PluginHandle loadFromPluginDir(Path pluginDir, ClassLoader parent, Cl Class discoveredClass; try { discoveredClass = classLoader.loadClass(factoryClassName); - } catch (ReflectiveOperationException | LinkageError e) { + } catch (ReflectiveOperationException | RuntimeException | LinkageError e) { // LinkageError included: defining the factory class resolves its supertypes, so a // dependency the plugin neither bundles nor inherits from its parent surfaces here as // NoClassDefFoundError - an Error, not a ReflectiveOperationException. Left uncaught it // escapes loadAll entirely and takes FE startup down, because one plugin's missing - // dependency is not something the FE can be stopped by. Same reasoning as factory.name() - // and factory.description() below, which already catch it. + // dependency is not something the FE can be stopped by. RuntimeException for the same + // reason: defineClass raises SecurityException for a signed package the plugin also + // ships unsigned classes into. Same catch shape as factory.name() and + // factory.description() below. throw new PluginLoadException( normalizedDir, LoadFailure.STAGE_INSTANTIATE, "Failed to instantiate factory class '" + factoryClassName + "' in " + normalizedDir - + missingClassAdvice(e), + + missingClassAdvice(e) + rootCauseSummary(e), e); } @@ -307,15 +322,20 @@ private PluginHandle loadFromPluginDir(Path pluginDir, ClassLoader parent, Cl @SuppressWarnings("unchecked") Class factoryClass = (Class) discoveredClass.asSubclass(factoryType); factory = factoryClass.getDeclaredConstructor().newInstance(); - } catch (ReflectiveOperationException | LinkageError e) { - // newInstance() is where the factory class is first initialized, so a static initializer - // that reaches a missing dependency arrives as ExceptionInInitializerError - again an - // Error, and again one plugin's problem rather than the FE's. + } catch (ReflectiveOperationException | RuntimeException | LinkageError e) { + // newInstance() is where the factory class is first initialized. A static initializer + // that fails arrives either as the Error itself - NoClassDefFoundError for a missing + // dependency - or, for a non-Error, wrapped in ExceptionInInitializerError; both are + // LinkageErrors, and again one plugin's problem rather than the FE's. asSubclass() + // throws ClassCastException (a RuntimeException) for a service file naming a class that + // is not a factory at all, which is the same plugin-local problem one line earlier. throw new PluginLoadException( normalizedDir, LoadFailure.STAGE_INSTANTIATE, "Failed to instantiate factory class '" + factoryClassName + "' in " + normalizedDir - + missingClassAdvice(e), + + (e instanceof ClassCastException + ? ": it does not implement " + factoryType.getName() : "") + + missingClassAdvice(e) + rootCauseSummary(e), e); } } catch (PluginLoadException e) { @@ -546,24 +566,21 @@ private String discoverSingleFactoryClassName(Class factoryType, ClassLoader * failure. A plugin misses a class either because it does not bundle it or because it expected to * inherit it from the layer its classloader delegates to, and the message must not leave the reader * guessing which - a shared library bundle that was never installed looks exactly like a broken - * plugin jar otherwise. + * plugin jar otherwise. The one JVM wording that does mean a broken jar - a class file found under + * a path that disagrees with the name in its bytecode - gets its own sentence, because the shared + * bundle is exactly the wrong place to look for that. */ - // Package-private so that the JVM messages it has to tell apart can be asserted directly: no - // plugin makes the loader initialize one class twice, so the second-attempt wording below is not - // reachable from loadAll. + // Package-private so that the JVM messages it has to tell apart can be asserted directly. The + // "Could not initialize class" wording is the JVM's second attempt at a class whose initializer + // already failed: not something loadAll provokes on a plugin's own classes (it gives up on the + // first attempt), but reachable through a parent-first class whose initializer failed under an + // earlier plugin, so it is skipped rather than read as a class name. static String missingClassAdvice(Throwable failure) { - String missing = missingClassName(failure); - if (missing == null) { - return ""; - } - return ". The class " + missing + " is in neither this plugin's own jars nor its parent" - + " classloader; if it is meant to come from a shared library bundle, check that the" - + " bundle is installed under the FE shared library root"; - } - - /** The absent class named by a NoClassDefFoundError / ClassNotFoundException anywhere in the chain. */ - private static String missingClassName(Throwable failure) { - for (Throwable t = failure; t != null; t = t.getCause() == t ? null : t.getCause()) { + // The chain is plugin-built below the first node, and a cycle is legal Java (a.initCause(b) + // after b was constructed with cause a): bound the walk by identity, or a plugin can hang FE + // startup here, under lifecycleLock, before any port is open. + Set seen = Collections.newSetFromMap(new IdentityHashMap<>()); + for (Throwable t = failure; t != null && seen.add(t); t = t.getCause()) { if (!(t instanceof NoClassDefFoundError) && !(t instanceof ClassNotFoundException)) { continue; } @@ -576,11 +593,41 @@ private static String missingClassName(Throwable failure) { // Keep walking: what really went missing, if anything did, is further down the chain. continue; } - // NoClassDefFoundError's message is sometimes "a/B (wrong name: c/D)" or carries a - // trailing explanation; the class name is the first token either way. - return message.split("\\s+", 2)[0]; + Matcher wrongName = WRONG_NAME_MESSAGE.matcher(message); + if (wrongName.matches()) { + return ". The class " + wrongName.group(1) + " was found in this plugin's jars, but its" + + " bytecode names it " + wrongName.group(2) + ": the jar entry is under the wrong" + + " path, which is a broken plugin jar rather than a missing dependency"; + } + if (!CLASS_NAME_MESSAGE.matcher(message).matches()) { + // Plugin code may wrap a lookup failure in a sentence ("Cannot load driver class + // com.x.Y"); nothing here can tell which word is the class, so keep walking - the JDK's + // own node, if there is one below, carries the bare name. + continue; + } + return ". The class " + message + " is in neither this plugin's own jars nor its parent" + + " classloader; if it is meant to come from a shared library bundle, check that the" + + " bundle is installed under the FE shared library root"; } - return null; + return ""; + } + + /** + * The innermost cause of a load failure as {@code Class: message}, or "" when the failure has no + * cause, so that a consumer logging the {@link LoadFailure} message alone still records why - an + * {@link ExceptionInInitializerError} prints as its bare class name and says nothing on its own. + * Identity-bounded for the same reason as {@link #missingClassAdvice}. + */ + static String rootCauseSummary(Throwable failure) { + Set seen = Collections.newSetFromMap(new IdentityHashMap<>()); + Throwable root = null; + for (Throwable t = failure; t != null && seen.add(t); t = t.getCause()) { + root = t; + } + if (root == null || root == failure) { + return ""; + } + return "; caused by " + root; } private static void closeClassLoader(ClassLoader classLoader) { diff --git a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerLinkageTest.java b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerLinkageTest.java index e6657f28d4bc72..749cf3dab79200 100644 --- a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerLinkageTest.java +++ b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerLinkageTest.java @@ -20,6 +20,8 @@ import org.apache.doris.extension.loader.testplugins.AbsentDependencyProbe; import org.apache.doris.extension.loader.testplugins.AbsentStaticInitTestPluginFactory; import org.apache.doris.extension.loader.testplugins.AbsentSuperclassTestPluginFactory; +import org.apache.doris.extension.loader.testplugins.NotAFactory; +import org.apache.doris.extension.loader.testplugins.ThrowingStaticInitTestPluginFactory; import org.apache.doris.extension.spi.PluginFactory; import org.junit.jupiter.api.Assertions; @@ -27,7 +29,9 @@ import org.junit.jupiter.api.io.TempDir; import java.io.IOException; +import java.lang.reflect.InvocationTargetException; import java.nio.file.Path; +import java.time.Duration; import java.util.Collections; /** @@ -43,10 +47,10 @@ * apart: resolving the class (a supertype the jar does not carry) and first initializing it (a * static initializer that reaches outside the jar). * - *

The advice sentence itself is asserted separately, against fabricated throwables. The message it - * has to reject - the JVM's wording for a class whose initializer had already failed - only appears on - * the second attempt at initializing one class, and no plugin makes the loader do that: {@code loadAll} - * gives up and closes the classloader on the first. + *

The advice sentence itself is asserted separately, against fabricated throwables, including the + * shapes it must not misread: the JVM's wording for a class whose initializer had already failed (which + * a parent-first class can carry in from an earlier plugin), a plugin's own sentence-shaped wording, and + * the "wrong name" form that means a broken jar rather than a missing dependency. */ class DirectoryPluginRuntimeManagerLinkageTest { @@ -90,10 +94,6 @@ void testAMissingClassIsNamedInEveryFormTheJvmReportsItIn() { Assertions.assertTrue(adviceFor(new NoClassDefFoundError("org/example/Probe")) .contains("The class org/example/Probe is"), "the plain form names the class directly"); - Assertions.assertTrue( - adviceFor(new NoClassDefFoundError("org/example/Probe (wrong name: org/other/Probe)")) - .contains("The class org/example/Probe is"), - "the wrong-name form carries a trailing explanation the class name comes before"); Assertions.assertTrue( adviceFor(new ExceptionInInitializerError(new ClassNotFoundException("org.example.Probe"))) .contains("The class org.example.Probe is"), @@ -107,6 +107,100 @@ void testAMissingClassIsNamedInEveryFormTheJvmReportsItIn() { "the miss below the initializer failure is what the reader needs"); } + @Test + void testAnInitializerThatThrowsIsAFailureThatNamesTheReason() throws IOException { + // A non-Error thrown by arrives wrapped in ExceptionInInitializerError, whose own + // message is null: logged as toString() it says nothing. The reason has to be carried into + // the failure message itself, because that message is all some consumers record. + Path root = tempDir.resolve("plugins"); + DirectoryPluginRuntimeManagerMetadataTest.createPluginJar( + root.resolve("throwing-static-init").resolve("throwing-static-init.jar"), + ThrowingStaticInitTestPluginFactory.class, "1.0"); + + LoadReport report = load(root); + Assertions.assertTrue(report.getSuccesses().isEmpty(), "the plugin cannot have loaded"); + Assertions.assertEquals(1, report.getFailures().size()); + LoadFailure failure = report.getFailures().get(0); + Assertions.assertEquals(LoadFailure.STAGE_INSTANTIATE, failure.getStage()); + Assertions.assertInstanceOf(ExceptionInInitializerError.class, failure.getCause(), + () -> "a non-Error from is wrapped by the JVM, got " + failure.getCause()); + Assertions.assertTrue(failure.getMessage().contains(ThrowingStaticInitTestPluginFactory.REASON), + () -> "the initializer's own reason must reach the message: " + failure.getMessage()); + Assertions.assertFalse(failure.getMessage().contains("shared library bundle"), + () -> "nothing is missing here, so no bundle advice: " + failure.getMessage()); + } + + @Test + void testAServiceFileNamingANonFactoryIsAFailureNotAThrow() throws IOException { + // asSubclass() answers a service entry that names a class of the wrong type with a + // ClassCastException: a RuntimeException, so neither of the two other catch families sees it. + Path root = tempDir.resolve("plugins"); + DirectoryPluginRuntimeManagerMetadataTest.createPluginJar( + root.resolve("not-a-factory").resolve("not-a-factory.jar"), + NotAFactory.class, "1.0", DirectoryPluginRuntimeManagerMetadataTest.TEST_GATE.getExpectedVersion()); + + LoadReport report = load(root); + Assertions.assertTrue(report.getSuccesses().isEmpty(), "the plugin cannot have loaded"); + Assertions.assertEquals(1, report.getFailures().size()); + LoadFailure failure = report.getFailures().get(0); + Assertions.assertEquals(LoadFailure.STAGE_INSTANTIATE, failure.getStage()); + Assertions.assertInstanceOf(ClassCastException.class, failure.getCause()); + Assertions.assertTrue(failure.getMessage().contains("does not implement " + PluginFactory.class.getName()), + () -> "the message must name the factory type the class was expected to implement: " + + failure.getMessage()); + } + + @Test + void testACyclicCauseChainIsWalkedOnce() { + // Legal Java: a's cause is still the sentinel when initCause runs, so the chain is a -> b -> a. + // A plugin's constructor can throw this, and the walk runs on the FE's startup thread under + // the loader's lifecycle lock, before any port is open. + RuntimeException a = new RuntimeException("a"); + RuntimeException b = new RuntimeException("b", a); + a.initCause(b); + + String advice = Assertions.assertTimeoutPreemptively(Duration.ofSeconds(10), + () -> DirectoryPluginRuntimeManager.missingClassAdvice(new InvocationTargetException(a))); + Assertions.assertEquals("", advice, "no node of the cycle names a missing class"); + + String summary = Assertions.assertTimeoutPreemptively(Duration.ofSeconds(10), + () -> DirectoryPluginRuntimeManager.rootCauseSummary(new InvocationTargetException(a))); + Assertions.assertTrue(summary.contains("java.lang.RuntimeException: b"), + () -> "the last node reached before the chain repeats is the summary: " + summary); + } + + @Test + void testOnlyAClassNameShapedMessageIsReadAsAMissingClass() { + // Plugin code may wrap a lookup failure in a sentence; its first word is not a class name. + Assertions.assertEquals("", DirectoryPluginRuntimeManager.missingClassAdvice( + new ClassNotFoundException("Cannot load driver class com.example.Driver")), + "a sentence-shaped message names no class the reader could act on"); + // ...but the JDK's own node below it does, and the walk has to reach it. + Assertions.assertTrue(adviceFor(new ClassNotFoundException( + "SPI class 'org.example.Probe' was not found in BE's classloader", + new ClassNotFoundException("org.example.Probe"))) + .contains("The class org.example.Probe is"), + "the bare name one node down is the one to report"); + // A trailing dot or a path is plugin wording too, never the class. + Assertions.assertEquals("", DirectoryPluginRuntimeManager.missingClassAdvice( + new ClassNotFoundException("com.example.Driver. Bundle this class into that jar."))); + // Binary and internal spellings, with nested-class and digit characters, are class names. + Assertions.assertTrue(adviceFor(new NoClassDefFoundError("org/example/Outer$Inner2")) + .contains("The class org/example/Outer$Inner2 is")); + } + + @Test + void testTheWrongNameFormIsABrokenJarNotAMissingDependency() { + // "a/B (wrong name: c/D)" means the class file WAS found, under a path that disagrees with the + // name in its bytecode; sending the reader to the shared bundle would be exactly wrong. + String advice = DirectoryPluginRuntimeManager.missingClassAdvice( + new NoClassDefFoundError("org/example/Probe (wrong name: org/other/Probe)")); + Assertions.assertTrue(advice.contains("org/example/Probe"), advice); + Assertions.assertTrue(advice.contains("org/other/Probe"), advice); + Assertions.assertTrue(advice.contains("wrong path"), advice); + Assertions.assertFalse(advice.contains("shared library bundle"), advice); + } + private static String adviceFor(Throwable failure) { String advice = DirectoryPluginRuntimeManager.missingClassAdvice(failure); Assertions.assertTrue(advice.contains("shared library bundle"), diff --git a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerMetadataTest.java b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerMetadataTest.java index 405c82252965de..0747b31c0b082e 100644 --- a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerMetadataTest.java +++ b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerMetadataTest.java @@ -119,8 +119,11 @@ static void createPluginJar(Path jarPath, Class factory createPluginJar(jarPath, factoryClass, implementationVersion, TEST_GATE.getExpectedVersion()); } - /** Same, with an explicit declared plugin API version (null writes no such attribute at all). */ - static void createPluginJar(Path jarPath, Class factoryClass, + /** + * Same, with an explicit declared plugin API version (null writes no such attribute at all). The + * class is any class on purpose: a service file may name one that is not a factory at all. + */ + static void createPluginJar(Path jarPath, Class factoryClass, String implementationVersion, String declaredApiVersion) throws IOException { Files.createDirectories(jarPath.getParent()); Manifest manifest = new Manifest(); diff --git a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/NotAFactory.java b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/NotAFactory.java new file mode 100644 index 00000000000000..572f95d07c5338 --- /dev/null +++ b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/NotAFactory.java @@ -0,0 +1,28 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.extension.loader.testplugins; + +/** + * A class a plugin's {@code META-INF/services} file can name although it implements no factory type at + * all: the loader's {@code asSubclass} refuses it with a {@link ClassCastException}, which is a + * {@link RuntimeException} rather than a {@link ReflectiveOperationException} or a {@link LinkageError}. + */ +public final class NotAFactory { + public NotAFactory() { + } +} diff --git a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/ThrowingStaticInitTestPluginFactory.java b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/ThrowingStaticInitTestPluginFactory.java new file mode 100644 index 00000000000000..cd573b53598f6b --- /dev/null +++ b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/ThrowingStaticInitTestPluginFactory.java @@ -0,0 +1,54 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.extension.loader.testplugins; + +import org.apache.doris.extension.spi.Plugin; +import org.apache.doris.extension.spi.PluginFactory; + +/** + * This class defines cleanly and its static initializer throws a plain {@link IllegalStateException}: + * the "keytab not found" shape of a plugin that is installed but misconfigured. Being a non-Error, the + * JVM wraps it in {@link ExceptionInInitializerError}, whose own message is null - so unless the loader + * carries the cause into the failure message, the operator is left with the bare class name. + */ +public class ThrowingStaticInitTestPluginFactory implements PluginFactory { + + public static final String REASON = "keytab not found: /etc/doris/plugin.keytab"; + + private static final Object MARKER = fail(); + + private static Object fail() { + throw new IllegalStateException(REASON); + } + + @Override + public String name() { + return "throwing-static-init-test"; + } + + @Override + public String description() { + return "Plugin whose static initializer throws: " + MARKER; + } + + @Override + public Plugin create() { + return new Plugin() { + }; + } +} From da29cc0624c7654e3da726c90e369ac3efb26bca Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 15 Sep 2026 21:51:04 +0800 Subject: [PATCH 07/19] [fix](fe) Log a skipped plugin with its stack trace, and guard the managers' first calls into plugin code Once a plugin that fails to link is skipped instead of taking FE startup down, the record of why it was skipped is all the operator has. The three managers that consume the loader's report logged each failure as "message={}, cause={}" - four placeholders for four arguments, which log4j renders with the throwable's toString() and no stack trace; for an ExceptionInInitializerError that is the bare class name. The summary line stayed at INFO. Each failure is now logged with three placeholders and the throwable trailing, so the trace is printed (the shape the loader's README prescribes and FileSystemPluginManager already used for its post-load call), and the two shipped families - connector and filesystem - log the summary at ERROR when any plugin failed, because an FE serving without a shipped plugin is an FE serving degraded. The loader's widened catch stops at the loader. The connector manager's first calls into a loaded plugin - getType() and acceptedCreateTableEngineNames() - ran with no guard, and the lineage manager ran factory.create() and plugin.initialize() inside catch (Exception): a NoClassDefFoundError there, one step after loadAll returned, still escaped to Env.initialize and exited the FE. Both now catch RuntimeException | LinkageError, refuse that plugin and release its classloader, mirroring the guard FileSystemPluginManager has on sensitivePropertyKeys(). Built-in connector providers keep failing loudly: a classpath provider that cannot link is a broken build, not a broken plugin directory. Tests: a directory connector provider whose getType() throws NoClassDefFoundError is refused rather than thrown, and rethrown for a built-in; a lineage plugin jar whose create() throws NoClassDefFoundError is skipped by start() with no active plugin and no throw. Co-Authored-By: Claude Opus 5 --- .../connector/ConnectorPluginManager.java | 40 ++++++++-- .../doris/fs/FileSystemPluginManager.java | 22 ++++-- .../lineage/LineageEventProcessor.java | 10 ++- .../connector/ConnectorPluginManagerTest.java | 24 ++++++ .../lineage/LineageEventProcessorTest.java | 76 +++++++++++++++++++ 5 files changed, 158 insertions(+), 14 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java index cc857b87d0ff9b..15f065411c295b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java @@ -239,8 +239,23 @@ void loadBuiltins(ClassLoader classLoader) { * @return true if the provider was admitted */ boolean registerDiscovered(ConnectorProvider provider, boolean failFast) { - String type = provider.getType(); - Set engineNames = provider.acceptedCreateTableEngineNames(); + String type; + Set engineNames; + try { + type = provider.getType(); + engineNames = provider.acceptedCreateTableEngineNames(); + } catch (RuntimeException | LinkageError e) { + // The first calls into plugin code after loading: a getType() that touches a class the + // plugin neither bundles nor inherits arrives here as a LinkageError the loader never saw. + // For a directory plugin that is one plugin's problem, not the FE's - same guard as + // FileSystemPluginManager's sensitivePropertyKeys() call. Built-ins keep failing loudly. + if (failFast) { + throw e; + } + LOG.error("Rejected connector provider {}: getType()/acceptedCreateTableEngineNames() failed." + + " The connector will not be available.", provider.getClass().getName(), e); + return false; + } String problem = typeNameProblem(type); if (problem == null) { problem = createTableEngineNameProblem(engineNames); @@ -312,13 +327,24 @@ public void loadPlugins(List pluginRoots) { classLoadingPolicy, API_VERSION_GATE); - LOG.info("Connector plugin load summary: rootsScanned={}, dirsScanned={}, " - + "successCount={}, failureCount={}", - report.getRootsScanned(), report.getDirsScanned(), - report.getSuccesses().size(), report.getFailures().size()); + if (report.getFailures().isEmpty()) { + LOG.info("Connector plugin load summary: rootsScanned={}, dirsScanned={}, " + + "successCount={}, failureCount=0", + report.getRootsScanned(), report.getDirsScanned(), report.getSuccesses().size()); + } else { + // A shipped plugin that failed to load is an FE serving degraded: every catalog of that + // type is unusable until the plugin directory is repaired, so the summary is an ERROR. + LOG.error("Connector plugin load summary: rootsScanned={}, dirsScanned={}, " + + "successCount={}, failureCount={}; the FE continues without the plugins" + + " that failed, each is reported below with its cause", + report.getRootsScanned(), report.getDirsScanned(), + report.getSuccesses().size(), report.getFailures().size()); + } for (LoadFailure failure : report.getFailures()) { - LOG.warn("Connector plugin load failure: dir={}, stage={}, message={}, cause={}", + // Three placeholders, four arguments: the trailing throwable is logged with its stack + // trace, which a "cause={}" placeholder would reduce to toString(). + LOG.warn("Connector plugin load failure: dir={}, stage={}, message={}", failure.getPluginDir(), failure.getStage(), failure.getMessage(), failure.getCause()); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java b/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java index 4c3d59548307cf..db404f7a8061fb 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java @@ -143,13 +143,25 @@ public void loadPlugins(List pluginRoots) { classLoadingPolicy, API_VERSION_GATE); - LOG.info("Filesystem plugin load summary: rootsScanned={}, dirsScanned={}, " - + "successCount={}, failureCount={}", - report.getRootsScanned(), report.getDirsScanned(), - report.getSuccesses().size(), report.getFailures().size()); + if (report.getFailures().isEmpty()) { + LOG.info("Filesystem plugin load summary: rootsScanned={}, dirsScanned={}, " + + "successCount={}, failureCount=0", + report.getRootsScanned(), report.getDirsScanned(), report.getSuccesses().size()); + } else { + // A shipped plugin that failed to load is an FE serving degraded: every repository, vault + // and catalog on that storage is unusable until the plugin directory is repaired, so the + // summary is an ERROR. + LOG.error("Filesystem plugin load summary: rootsScanned={}, dirsScanned={}, " + + "successCount={}, failureCount={}; the FE continues without the plugins" + + " that failed, each is reported below with its cause", + report.getRootsScanned(), report.getDirsScanned(), + report.getSuccesses().size(), report.getFailures().size()); + } for (LoadFailure failure : report.getFailures()) { - LOG.warn("Filesystem plugin load failure: dir={}, stage={}, message={}, cause={}", + // Three placeholders, four arguments: the trailing throwable is logged with its stack + // trace, which a "cause={}" placeholder would reduce to toString(). + LOG.warn("Filesystem plugin load failure: dir={}, stage={}, message={}", failure.getPluginDir(), failure.getStage(), failure.getMessage(), failure.getCause()); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/lineage/LineageEventProcessor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/lineage/LineageEventProcessor.java index 8c20f27366c358..5138cda08e5db2 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/lineage/LineageEventProcessor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/lineage/LineageEventProcessor.java @@ -173,8 +173,10 @@ pluginRoots, getClass().getClassLoader(), LineagePluginFactory.class, policy, API_VERSION_GATE); for (LoadFailure failure : report.getFailures()) { + // Three placeholders, four arguments: the trailing throwable is logged with its + // stack trace, which a "cause={}" placeholder would reduce to toString(). LOG.warn("Skip lineage plugin directory due to load failure:" - + " pluginDir={}, stage={}, message={}, cause={}", + + " pluginDir={}, stage={}, message={}", failure.getPluginDir(), failure.getStage(), failure.getMessage(), failure.getCause()); } @@ -218,8 +220,12 @@ pluginRoots, getClass().getClassLoader(), plugins.add(plugin); LOG.info("Loaded lineage plugin: {}, pluginPath={}", pluginName, props.get("plugin.path")); } - } catch (Exception e) { + } catch (Exception | LinkageError e) { + // create() and initialize() are the first calls into the plugin implementation (the + // factory only named it): a dependency it lacks arrives here as NoClassDefFoundError, + // which must cost this plugin alone, not FE startup. Its classloader goes with it. LOG.warn("Failed to create/initialize lineage plugin: {}", pluginName, e); + runtimeManager.discard(pluginName); } } initPlugins(plugins); diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorPluginManagerTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorPluginManagerTest.java index 643b7bbeeeee09..282cc54c45c08e 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorPluginManagerTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorPluginManagerTest.java @@ -336,6 +336,30 @@ void refusedProviderDoesNotLeaveItsTypeOrEngineNameClaimed() { Assertions.assertEquals(Collections.singletonList("good_type"), manager.getRegisteredTypes()); } + @Test + void testADirectoryProviderWhoseFirstCallFailsToLinkIsRefusedNotThrown() { + // getType() is the first call into plugin code after loading. A dependency the plugin neither + // bundles nor inherits surfaces there as NoClassDefFoundError, which must cost that plugin + // alone: the FE is not stopped by one directory plugin. + ConnectorProvider unlinkable = new ConnectorProvider() { + @Override + public String getType() { + throw new NoClassDefFoundError("org/example/AbsentDependency"); + } + + @Override + public Connector create(Map properties, ConnectorContext context) { + return null; + } + }; + Assertions.assertFalse(Assertions.assertDoesNotThrow(() -> manager.registerDiscovered(unlinkable, false)), + "a provider that cannot answer getType() must be refused"); + Assertions.assertTrue(manager.getRegisteredTypes().isEmpty(), "nothing may have been registered"); + + // Built-ins keep failing loudly: a classpath provider that cannot link is a broken FE build. + Assertions.assertThrows(NoClassDefFoundError.class, () -> manager.registerDiscovered(unlinkable, true)); + } + private static ConnectorProvider createProviderWithEngines(String type, String... engineNames) { Set engines = new HashSet<>(Arrays.asList(engineNames)); return new ConnectorProvider() { diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/lineage/LineageEventProcessorTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/lineage/LineageEventProcessorTest.java index 54f72032630164..2a0191aff257ae 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/lineage/LineageEventProcessorTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/lineage/LineageEventProcessorTest.java @@ -17,12 +17,20 @@ package org.apache.doris.nereids.lineage; +import org.apache.doris.common.Config; +import org.apache.doris.extension.loader.ApiVersionGate; import org.apache.doris.extension.spi.PluginContext; import org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; @@ -30,6 +38,10 @@ import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; +import java.util.jar.Attributes; +import java.util.jar.JarEntry; +import java.util.jar.JarOutputStream; +import java.util.jar.Manifest; /** * Unit tests for {@link LineageEventProcessor} SPI-based plugin management @@ -42,6 +54,70 @@ public class LineageEventProcessorTest { private static final long TEST_CONTEXT_TIMESTAMP_MS = 1000L; private static final long TEST_CONTEXT_DURATION_MS = 50L; + // ==================== a plugin that fails after loading ==================== + + /** Its factory loads fine; the plugin implementation it names reaches a class the jar lacks. */ + public static class UnlinkableLineagePluginFactory implements LineagePluginFactory { + @Override + public String name() { + return "unlinkable-lineage-test"; + } + + @Override + public String description() { + return "create() links a class this plugin does not carry"; + } + + @Override + public LineagePlugin create() { + throw new NoClassDefFoundError("org/example/AbsentHttpClient"); + } + } + + /** + * {@code create()} / {@code initialize()} are the first calls into the plugin implementation, one + * step after the loader handed back the factory. A {@code NoClassDefFoundError} there is an Error + * the old {@code catch (Exception)} let through to FE startup; it must cost this plugin alone. + */ + @Test + public void testAPluginWhoseCreateCannotLinkIsSkippedNotFatal(@TempDir Path tempDir) throws IOException { + writeLineagePluginJar(tempDir.resolve("lineage").resolve("unlinkable").resolve("unlinkable.jar"), + UnlinkableLineagePluginFactory.class); + String savedPluginDir = Config.plugin_dir; + String[] savedActive = Config.activate_lineage_plugin; + Config.plugin_dir = tempDir.toString(); + Config.activate_lineage_plugin = new String[0]; + try { + LineageEventProcessor processor = new LineageEventProcessor(); + Assertions.assertDoesNotThrow(processor::start); + Assertions.assertFalse(processor.hasActivePlugins(), "the plugin that failed to link is not active"); + } finally { + Config.plugin_dir = savedPluginDir; + Config.activate_lineage_plugin = savedActive; + } + } + + private static void writeLineagePluginJar(Path jarPath, Class factoryClass) + throws IOException { + Files.createDirectories(jarPath.getParent()); + ApiVersionGate gate = ApiVersionGate.forFamily("lineage", LineagePluginFactory.class); + Manifest manifest = new Manifest(); + manifest.getMainAttributes().put(Attributes.Name.MANIFEST_VERSION, "1.0"); + manifest.getMainAttributes().putValue(gate.getManifestAttribute(), gate.getExpectedVersion()); + String classEntry = factoryClass.getName().replace('.', '/') + ".class"; + try (JarOutputStream jar = new JarOutputStream(Files.newOutputStream(jarPath), manifest)) { + jar.putNextEntry(new JarEntry(classEntry)); + try (InputStream bytes = factoryClass.getClassLoader().getResourceAsStream(classEntry)) { + Assertions.assertNotNull(bytes, "class bytes not found: " + classEntry); + jar.write(bytes.readAllBytes()); + } + jar.closeEntry(); + jar.putNextEntry(new JarEntry("META-INF/services/" + LineagePluginFactory.class.getName())); + jar.write((factoryClass.getName() + "\n").getBytes(StandardCharsets.UTF_8)); + jar.closeEntry(); + } + } + // ==================== hasActivePlugins / refreshPlugins ==================== @Test From 8cacb3313221b83953ab82c7f64447e3eeaa28dc Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 15 Sep 2026 21:51:13 +0800 Subject: [PATCH 08/19] [fix](fe) Mask the shipped filesystem plugins' credential aliases without the plugin having loaded The aliases a filesystem plugin declares sensitive were masked only because each loaded plugin's sensitivePropertyKeys() was registered into DatasourcePrintableMap.SENSITIVE_KEY at startup. With a shipped plugin skipped at load - the state this branch makes survivable - the registration never ran, while every catalog, repository and vault created with that plugin's aliases stayed in metadata. SHOW CREATE CATALOG, SHOW CATALOG, the catalogs() TVF and the audit-log encryption of CREATE/LOAD/TVF/OUTFILE statements all mask through the same exact-key set, so cos.session_token, AZURE_CLIENT_SECRET, ozone.secret_key and sixteen more aliases were printed in clear. The static set now carries the 19 aliases the shipped plugins declare beyond the legacy union it already inlined, and exposes the static contents as an immutable BUILTIN_SENSITIVE_KEYS so a test can pin the providers' aliases to it without the registration that runs in the same JVM masking a gap: the test loads every FileSystemProvider on the test classpath and fails on any sensitive alias the static set does not contain. Co-Authored-By: Claude Opus 5 --- .../common/util/DatasourcePrintableMap.java | 32 +++++++++++-- .../util/DatasourcePrintableMapTest.java | 48 +++++++++++++++++++ 2 files changed, 76 insertions(+), 4 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/util/DatasourcePrintableMap.java b/fe/fe-core/src/main/java/org/apache/doris/common/util/DatasourcePrintableMap.java index 7db1013d806a3b..41d9df9f5bbe34 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/common/util/DatasourcePrintableMap.java +++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/DatasourcePrintableMap.java @@ -24,6 +24,7 @@ import java.util.Arrays; import java.util.Collection; +import java.util.Collections; import java.util.Map; import java.util.Set; import java.util.TreeMap; @@ -32,6 +33,13 @@ public class DatasourcePrintableMap extends BasicPrintableMap { private boolean hidePassword; private Set additionalHiddenKeys = Sets.newHashSet(); + /** + * The aliases masked with no filesystem plugin loaded: everything the static block below enumerates, + * case-insensitive and immutable. {@link #SENSITIVE_KEY} starts as a copy of it and grows through + * {@link #registerSensitiveKeys}. Exposed so that a test can pin the shipped plugins' aliases to the + * static set without the registration that runs in the same JVM masking a gap. + */ + public static final Set BUILTIN_SENSITIVE_KEYS; public static final Set SENSITIVE_KEY; public static final Set HIDDEN_KEY; public static final String PASSWORD_MASK = "*XXX"; @@ -79,10 +87,8 @@ public class DatasourcePrintableMap extends BasicPrintableMap { SENSITIVE_KEY.add("iceberg.rest.session-token"); // Inlined union of the legacy typed storage classes' @ConnectorProperty(sensitive = true) // key aliases (S3/GCS/Azure/OSS/OSS-HDFS/COS/OBS/Minio Properties). The set is - // case-insensitive, so alias spellings differing only in case are listed once. Locked - // against the legacy classes by DatasourcePrintableMapSensitiveKeysParityTest until the - // legacy package is deleted; SPI providers additionally feed registerSensitiveKeys at - // FE startup. + // case-insensitive, so alias spellings differing only in case are listed once. SPI providers + // additionally feed registerSensitiveKeys at FE startup. SENSITIVE_KEY.addAll(Arrays.asList( "s3.access_key", "AWS_ACCESS_KEY", "access_key", "glue.access_key", "aws.glue.access-key", "client.credentials-provider.glue.access_key", @@ -101,6 +107,24 @@ public class DatasourcePrintableMap extends BasicPrintableMap { "cos.access_key", "cos.secret_key", "obs.access_key", "obs.secret_key", "minio.access_key", "minio.secret_key", "minio.session_token")); + // The aliases the shipped filesystem plugins declare sensitive beyond the legacy union above. + // Masking must not depend on a plugin having loaded: a shipped plugin that fails to load is + // skipped and the FE serves on, while every catalog, repository and vault created with its + // aliases stays in metadata and is printed by SHOW CREATE CATALOG, SHOW CATALOG, the + // catalogs() TVF and the audit log through this set. Pinned to the providers on the test + // classpath by DatasourcePrintableMapTest.testShippedFilesystemPluginAliasesAreMaskedWithoutThePlugin. + SENSITIVE_KEY.addAll(Arrays.asList( + "aws.glue.session-token", + "AZURE_ACCOUNT_KEY", "AZURE_CLIENT_SECRET", + "cos.session_token", "COS_SECRET_KEY", "COS_SESSION_TOKEN", "COS_TOKEN", + "gs.session_token", + "obs.session_token", "OBS_SECRET_KEY", "OBS_SESSION_TOKEN", "OBS_TOKEN", + "oss.security_token", "oss.hdfs.security_token", + "OSS_SECRET_KEY", "OSS_SESSION_TOKEN", "OSS_TOKEN", + "ozone.secret_key", "ozone.session_token")); + Set builtin = Sets.newTreeSet(String.CASE_INSENSITIVE_ORDER); + builtin.addAll(SENSITIVE_KEY); + BUILTIN_SENSITIVE_KEYS = Collections.unmodifiableSet(builtin); HIDDEN_KEY = Sets.newHashSet(); // Exact literals of the legacy S3Properties.Env.FS_KEYS list. HIDDEN_KEY.addAll(Arrays.asList("AWS_ENDPOINT", "AWS_REGION", "AWS_ACCESS_KEY", "AWS_SECRET_KEY", diff --git a/fe/fe-core/src/test/java/org/apache/doris/common/util/DatasourcePrintableMapTest.java b/fe/fe-core/src/test/java/org/apache/doris/common/util/DatasourcePrintableMapTest.java index c1c085fdaef1cc..5d083b26a96c1f 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/common/util/DatasourcePrintableMapTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/common/util/DatasourcePrintableMapTest.java @@ -17,12 +17,18 @@ package org.apache.doris.common.util; +import org.apache.doris.filesystem.spi.FileSystemProvider; + import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import java.util.ArrayList; import java.util.HashMap; +import java.util.List; import java.util.Map; +import java.util.ServiceLoader; import java.util.Set; +import java.util.TreeSet; public class DatasourcePrintableMapTest { @@ -59,6 +65,48 @@ public void testSensitiveKeysContainAliyunDLFProperties() { Assertions.assertTrue(DatasourcePrintableMap.SENSITIVE_KEY.contains("oss.secret_key")); } + /** + * Masking must not depend on a filesystem plugin having loaded. A shipped plugin that fails to load + * is skipped and the FE serves on, but every catalog, repository and vault created with its aliases + * is still in metadata and is printed through this set (SHOW CREATE CATALOG, SHOW CATALOG, the + * catalogs() TVF, the audit log). So every alias a shipped provider declares sensitive has to be in + * the static set, not only in what {@link DatasourcePrintableMap#registerSensitiveKeys} adds at + * startup. Checked against {@link DatasourcePrintableMap#BUILTIN_SENSITIVE_KEYS} rather than + * {@code SENSITIVE_KEY}, which another test in this JVM may already have grown by registration. + * + *

The providers come from the test classpath, which carries every shipped filesystem plugin + * (fe-filesystem-obs and -cos only unless the build passes {@code -Ddisable.obs} / {@code -Ddisable.cos}). + */ + @Test + public void testShippedFilesystemPluginAliasesAreMaskedWithoutThePlugin() { + List providerNames = new ArrayList<>(); + Set unmasked = new TreeSet<>(String.CASE_INSENSITIVE_ORDER); + for (FileSystemProvider provider : ServiceLoader.load(FileSystemProvider.class)) { + providerNames.add(provider.name()); + for (String alias : provider.sensitivePropertyKeys()) { + if (!DatasourcePrintableMap.BUILTIN_SENSITIVE_KEYS.contains(alias)) { + unmasked.add(provider.name() + ":" + alias); + } + } + } + Assertions.assertTrue(providerNames.size() >= 10, + () -> "the shipped filesystem providers must be on the test classpath, found only " + + providerNames); + Assertions.assertTrue(unmasked.isEmpty(), + () -> "sensitive aliases printed in clear when the plugin declaring them is not loaded: " + + unmasked); + + // And the set is consulted case-insensitively by the printer itself, plugin or no plugin. + Map map = new HashMap<>(); + map.put("cos.session_token", "sts-token"); + map.put("AZURE_CLIENT_SECRET", "client-secret"); + map.put("ozone.secret_key", "ozone-secret"); + String printed = new DatasourcePrintableMap<>(map, " = ", true, false, true).toString(); + Assertions.assertFalse(printed.contains("sts-token"), printed); + Assertions.assertFalse(printed.contains("client-secret"), printed); + Assertions.assertFalse(printed.contains("ozone-secret"), printed); + } + @Test public void testBasicConstructor() { Map testMap = new HashMap<>(); From e6e5bc941174042b1776bf9138a00daa5759da28 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 15 Sep 2026 21:55:01 +0800 Subject: [PATCH 09/19] [fix](fe) Never persist a storage type guessed while its filesystem provider is absent With a filesystem plugin skipped at load, two paths turned "the provider is absent" into a persisted type. Repository.gsonPostProcess migrates a pre-SPI record ("fs" = name + properties) into fs_descriptor at image load by binding its provider; when that failed it fell back to the BROKER provider, whose bind accepts any map, with the legacy storage name as the broker name. The descriptor carries an explicit type and is written by the next checkpoint, so an S3 repository loaded on an FE whose S3 plugin failed to link became "WITH BROKER S3" permanently, and stayed so after the plugin was repaired (4.1 had the same fallback but re-serialised only name and properties, so the binding was retried at every start). The legacy name is the storage type for a typed repository and the broker name for a broker one: the broker fallback is now taken only when the name is a registered broker. Any other record no provider claims is kept verbatim, so the migration is retried at the next start; until then the repository has no descriptor, SHOW REPOSITORIES shows the reason in ErrMsg, every I/O path and ALTER/SHOW CREATE REPOSITORY report it, and the periodic ping does not rewrap it. Building the repository's filesystem at load also tolerates a LinkageError now, the catch CatalogFactory uses for the same half-installed-plugin case. CREATE RESOURCE decided AZURE through StorageAdapter.matchesProviderGuess, which answered false both for "not an Azure map" and for "the Azure provider is not loaded", so the documented Azure syntax (type=s3 with a *.blob.core.windows.net endpoint) created an S3Resource while the plugin was absent - and ALTER RESOURCE cannot change a type. matchesProviderGuess now throws for an absent provider, like ofProvider, and CREATE RESOURCE evaluates the endpoint predicate the Azure provider shares with the routing layer to refuse an Azure-shaped map in that state; a real S3 map is unaffected. Tests: a legacy HDFS record loaded with the HDFS provider absent keeps its "fs" field and gains no fs_descriptor, reports through listSnapshots, getBrokerAddress, getInfo and getCreateStatement, and migrates to HDFS once the provider is back; a legacy broker record migrates only when its broker is registered; an Azure-shaped CREATE RESOURCE is refused with the Azure provider absent and resolves to AZURE with it present; matchesProviderGuess reports an absent provider instead of false. Co-Authored-By: Claude Opus 5 --- .../apache/doris/backup/BackupHandler.java | 3 + .../org/apache/doris/backup/Repository.java | 75 +++++++++++++++-- .../datasource/storage/StorageAdapter.java | 8 +- .../commands/ShowCreateRepositoryCommand.java | 3 + .../commands/info/CreateResourceInfo.java | 22 ++++- .../apache/doris/backup/RepositoryTest.java | 84 +++++++++++++++++++ .../doris/fs/AzureGuessRoutingParityTest.java | 16 ++++ .../fs/TestFileSystemPluginManagers.java | 53 ++++++++++++ .../commands/info/CreateResourceInfoTest.java | 66 +++++++++++++++ 9 files changed, 317 insertions(+), 13 deletions(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/fs/TestFileSystemPluginManagers.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateResourceInfoTest.java diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java b/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java index a3965c8e42e29e..cd561e30646e6a 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java +++ b/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java @@ -237,6 +237,9 @@ public void alterRepository(String repoName, Map newProps) if (oldRepo == null) { throw new DdlException("Repository does not exist"); } + if (!oldRepo.hasFileSystemDescriptor()) { + throw new DdlException("Repository " + repoName + " is not available: " + oldRepo.getErrorMsg()); + } // Merge new properties with the existing repository's properties Map mergedProps = mergeProperties(oldRepo, newProps); // Create new Repository instance with merged properties diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java b/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java index d9be5a0c921c1d..5c85a400dfd5b7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java +++ b/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java @@ -255,32 +255,69 @@ public void gsonPostProcess() { StorageAdapter storageAdapter = StorageAdapter.of(props); fileSystemDescriptor = FileSystemDescriptor.fromStorageAdapter(storageAdapter, ""); } catch (RuntimeException e) { - LOG.warn("Repository '{}': primary storage migration failed ({}), trying broker fallback", - name, e.getMessage()); + // The legacy record's name is the storage type ("S3", "HDFS", ...) for a typed + // repository and the broker name for a broker one. A registered broker name is the one + // positive signal that the record was a broker repository, so only then is BROKER the + // right descriptor. Anything else that no provider claims is left as it was: the + // descriptor carries an explicit type and is persisted by the next checkpoint, so a + // BROKER fallback here would permanently rewrite an S3 repository whose plugin merely + // failed to load. Keeping the legacy record instead retries the migration at the next + // start, and every use of the repository reports the reason until then. + if (!isRegisteredBroker(fsName)) { + errMsg = "legacy filesystem metadata was not migrated: no loaded filesystem provider" + + " claims its properties and '" + fsName + "' is not a registered broker (" + + e.getMessage() + "). Install or repair the filesystem plugin for this" + + " repository and restart the FE, or DROP and re-CREATE it."; + LOG.warn("Repository '{}': {}", name, errMsg); + return; + } + LOG.warn("Repository '{}': primary storage migration failed ({}), migrating as a broker" + + " repository on broker '{}'", name, e.getMessage(), fsName); try { StorageAdapter brokerAdapter = StorageAdapter.ofBroker(fsName, props); fileSystemDescriptor = FileSystemDescriptor.fromStorageAdapter(brokerAdapter, fsName); } catch (RuntimeException e2) { - LOG.error("Repository '{}': failed to migrate legacy filesystem metadata: {}", - name, e2.getMessage()); + errMsg = "legacy filesystem metadata was not migrated: " + e2.getMessage(); + LOG.error("Repository '{}': {}", name, errMsg); return; } } fsProps = fileSystemDescriptor.getProperties(); } else { - LOG.error("Repository '{}': metadata corrupt — both 'fs' and 'fs_descriptor' fields are missing", name); + errMsg = "metadata corrupt: both 'fs' and 'fs_descriptor' fields are missing"; + LOG.error("Repository '{}': {}", name, errMsg); return; } // Initialize SPI filesystem for I/O; broker resolves a live endpoint per I/O call if (fileSystemDescriptor.getStorageType() != FsStorageType.BROKER) { try { this.spiFs = FileSystemFactory.getFileSystem(StorageAdapter.of(fsProps)); - } catch (IOException | RuntimeException e) { + } catch (IOException | RuntimeException | LinkageError e) { + // LinkageError included: this runs at image load and edit-log replay, where a plugin + // whose factory loaded but whose bind()/create() first links a jar it lacks must cost + // this repository, not the FE - the same catch CatalogFactory uses for catalogs. LOG.warn("Failed to initialize SPI filesystem for repository {}: {}", name, e.getMessage()); } } } + /** + * Whether the repository has a storage descriptor. False only for a legacy record whose migration + * {@link #gsonPostProcess()} declined because no loaded filesystem provider claimed it; the record + * is then written back unchanged, {@link #getErrorMsg()} says why, and every I/O path fails with it. + */ + public boolean hasFileSystemDescriptor() { + return fileSystemDescriptor != null; + } + + private static boolean isRegisteredBroker(String brokerName) { + if (Strings.isNullOrEmpty(brokerName)) { + return false; + } + Env env = Env.getCurrentEnv(); + return env != null && env.getBrokerMgr() != null && env.getBrokerMgr().containsBroker(brokerName); + } + public long getId() { return id; } @@ -318,6 +355,9 @@ private org.apache.doris.filesystem.FileSystem acquireSpiFs() throws IOException if (spiFs != null) { return spiFs; } + if (fileSystemDescriptor == null) { + throw new IOException("Repository '" + name + "' is not available: " + errMsg); + } if (fileSystemDescriptor.getStorageType() != FsStorageType.BROKER) { // spiFs should have been initialized in the constructor or gsonPostProcess. // If it is null here, initialization failed silently during deserialization. @@ -502,6 +542,11 @@ public boolean ping() { if (FeConstants.runningUnitTest) { return true; } + if (fileSystemDescriptor == null) { + // Not migrated (see gsonPostProcess): errMsg already says why, and the periodic ping must + // not keep rewrapping it. + return false; + } // for s3 sdk, the headObject() method does not support list "dir", // so we check FILE_REPO_INFO instead. // Use getLocation() like every other repo path assembly: concrete filesystems only @@ -817,6 +862,9 @@ public Status download(String remoteFilePath, String localFilePath) { } public Status getBrokerAddress(Long beId, Env env, List brokerAddrs) { + if (fileSystemDescriptor == null) { + return new Status(ErrCode.COMMON_ERROR, "Repository '" + name + "' is not available: " + errMsg); + } // get backend Backend be = Env.getCurrentSystemInfo().getBackend(beId); if (be == null) { @@ -853,9 +901,15 @@ public List getInfo() { info.add(TimeUtils.longToTimeString(createTime)); info.add(String.valueOf(isReadOnly)); info.add(location); - info.add(fileSystemDescriptor.getStorageType() != FsStorageType.BROKER - ? "-" : fileSystemDescriptor.getName()); - info.add(fileSystemDescriptor.getStorageType().name()); + if (fileSystemDescriptor == null) { + // A legacy record whose migration was declined (see gsonPostProcess): the reason is in errMsg. + info.add("-"); + info.add(legacyFs != null && legacyFs.name != null ? legacyFs.name : FeConstants.null_string); + } else { + info.add(fileSystemDescriptor.getStorageType() != FsStorageType.BROKER + ? "-" : fileSystemDescriptor.getName()); + info.add(fileSystemDescriptor.getStorageType().name()); + } info.add(errMsg == null ? FeConstants.null_string : errMsg); return info; } @@ -886,6 +940,9 @@ public List> getSnapshotInfos(String snapshotName, String timestamp } public String getCreateStatement() { + if (fileSystemDescriptor == null) { + throw new IllegalStateException("Repository '" + name + "' is not available: " + errMsg); + } StringBuilder stmtBuilder = new StringBuilder(); stmtBuilder.append("CREATE "); if (this.isReadOnly) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java index e5df6e21196828..72b08397614429 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java @@ -183,6 +183,11 @@ public static StorageAdapter ofProvider(String providerName, Map * Returns whether the named provider's guess heuristics claim the raw properties — the * facade twin of the legacy per-dialect {@code guessIsMe} statics (guess only, explicit * {@code fs..support} flags deliberately NOT consulted, matching the legacy call sites). + * + * @throws StoragePropertiesException if no provider of that name is loaded, like + * {@link #ofProvider}: a caller deciding a persisted type on this answer must be able to + * tell "not that provider's map" from "that provider is absent", or an absent plugin + * silently turns into the wrong type on disk. */ public static boolean matchesProviderGuess(String providerName, Map origProps) { Map props = origProps == null ? new HashMap<>() : origProps; @@ -194,7 +199,8 @@ public static boolean matchesProviderGuess(String providerName, Map info = deserialized.getInfo(); + Assertions.assertEquals("hdfsRepoWithoutPlugin", info.get(1)); + Assertions.assertEquals("-", info.get(5)); + Assertions.assertEquals("HDFS", info.get(6)); + Assertions.assertEquals(deserialized.getErrorMsg(), info.get(7)); + Assertions.assertThrows(IllegalStateException.class, deserialized::getCreateStatement); + + // The provider is back: the untouched record migrates to its real type. + StorageAdapter.initPluginManager(TestFileSystemPluginManagers.withoutProviders()); + Repository repaired = GsonUtils.GSON.fromJson(rewritten, Repository.class); + Assertions.assertTrue(repaired.hasFileSystemDescriptor()); + Assertions.assertEquals(FsStorageType.HDFS, repaired.getFileSystemDescriptor().getStorageType()); + } finally { + StorageAdapter.initPluginManager(null); + } + } + + /** The broker fallback is taken on the one positive signal only: a registered broker name. */ + @Test + public void testLegacyRecordNamingAnUnregisteredBrokerIsNotMigratedToBroker() { + String legacyJson = "{" + + "\"id\":40000," + + "\"n\":\"legacyRepoGoneBroker\"," + + "\"iro\":false," + + "\"lo\":\"bos://backup/legacy\"," + + "\"ct\":-1," + + "\"fs\":{\"n\":\"gone_broker\",\"prop\":{}}" + + "}"; + Mockito.when(mockedBrokerMgr.containsBroker("gone_broker")).thenReturn(false); + + Repository deserialized = GsonUtils.GSON.fromJson(legacyJson, Repository.class); + + Assertions.assertFalse(deserialized.hasFileSystemDescriptor()); + Assertions.assertTrue(deserialized.getErrorMsg().contains("'gone_broker' is not a registered broker"), + deserialized.getErrorMsg()); + Assertions.assertFalse(GsonUtils.GSON.toJson(deserialized).contains("fs_descriptor")); + } + + /** * H1: Verify migration when legacy props contain HDFS-specific keys that are recognized by * {@link StorageAdapter#of(java.util.Map)} (primary path, not broker fallback). diff --git a/fe/fe-core/src/test/java/org/apache/doris/fs/AzureGuessRoutingParityTest.java b/fe/fe-core/src/test/java/org/apache/doris/fs/AzureGuessRoutingParityTest.java index 193859ed975f32..3bf0a8ff4ef694 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/fs/AzureGuessRoutingParityTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/fs/AzureGuessRoutingParityTest.java @@ -20,6 +20,7 @@ import org.apache.doris.common.Config; import org.apache.doris.datasource.property.common.AwsCredentialsProviderMode; import org.apache.doris.datasource.storage.StorageAdapter; +import org.apache.doris.foundation.property.StoragePropertiesException; import com.google.common.collect.ImmutableMap; import org.junit.jupiter.api.Assertions; @@ -90,6 +91,21 @@ public void testGuessMatchesFrozenExpectations() { } } + @Test + public void testGuessAgainstAnAbsentProviderIsAnErrorNotFalse() { + // "false" would be indistinguishable from "not an Azure map" to a caller that persists a type + // on the answer, so an absent provider is reported the way ofProvider() reports it. + StorageAdapter.initPluginManager(TestFileSystemPluginManagers.withoutProviders("AZURE")); + try { + Map props = ImmutableMap.of("s3.endpoint", "https://acct.blob.core.windows.net"); + StoragePropertiesException absent = Assertions.assertThrows(StoragePropertiesException.class, + () -> StorageAdapter.matchesProviderGuess("AZURE", props)); + Assertions.assertTrue(absent.getMessage().contains("'AZURE' is not available"), absent.getMessage()); + } finally { + StorageAdapter.initPluginManager(null); + } + } + @Test public void testConfigExtendedSuffixReachesThePlugin() { String[] saved = Config.azure_blob_host_suffixes; diff --git a/fe/fe-core/src/test/java/org/apache/doris/fs/TestFileSystemPluginManagers.java b/fe/fe-core/src/test/java/org/apache/doris/fs/TestFileSystemPluginManagers.java new file mode 100644 index 00000000000000..43fbf9f98d9f30 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/fs/TestFileSystemPluginManagers.java @@ -0,0 +1,53 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.fs; + +import org.apache.doris.datasource.storage.StorageAdapter; +import org.apache.doris.filesystem.spi.FileSystemProvider; + +import java.util.Arrays; +import java.util.List; +import java.util.Locale; +import java.util.ServiceLoader; +import java.util.stream.Collectors; + +/** + * Builds the registry of an FE that started with some of its shipped filesystem plugins absent - the + * state a plugin directory that failed to load leaves behind - from the providers on the test classpath. + * Install with {@link StorageAdapter#initPluginManager} and restore with {@code initPluginManager(null)} + * in a {@code finally}: the facade's registry is process-wide. + */ +public final class TestFileSystemPluginManagers { + + private TestFileSystemPluginManagers() { + } + + /** Every provider on the test classpath except the named ones (case-insensitive). */ + public static FileSystemPluginManager withoutProviders(String... absentNames) { + List absent = Arrays.stream(absentNames) + .map(name -> name.toUpperCase(Locale.ROOT)) + .collect(Collectors.toList()); + FileSystemPluginManager manager = new FileSystemPluginManager(); + for (FileSystemProvider provider : ServiceLoader.load(FileSystemProvider.class)) { + if (!absent.contains(provider.name().toUpperCase(Locale.ROOT))) { + manager.registerProvider(provider); + } + } + return manager; + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateResourceInfoTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateResourceInfoTest.java new file mode 100644 index 00000000000000..04817d7e63b754 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateResourceInfoTest.java @@ -0,0 +1,66 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.trees.plans.commands.info; + +import org.apache.doris.catalog.Resource.ResourceType; +import org.apache.doris.common.AnalysisException; +import org.apache.doris.datasource.storage.StorageAdapter; +import org.apache.doris.fs.TestFileSystemPluginManagers; + +import com.google.common.collect.ImmutableMap; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +public class CreateResourceInfoTest { + + private static final ImmutableMap AZURE_SHAPED = ImmutableMap.of( + "type", "s3", + "s3.endpoint", "https://acct.blob.core.windows.net", + "s3.access_key", "ak", + "s3.secret_key", "sk"); + + @Test + public void testAzureShapedPropertiesResolveToAzure() throws Exception { + CreateResourceInfo info = new CreateResourceInfo(false, false, "azure_res", AZURE_SHAPED); + info.analyzeResourceType(); + Assertions.assertEquals(ResourceType.AZURE, info.getResourceType()); + } + + /** + * The resource type is persisted and ALTER RESOURCE cannot change it, so with the Azure plugin + * absent an Azure-shaped {@code type=s3} map must be refused - not created as an S3Resource that + * stays one after the plugin is repaired. A real S3 map is unaffected by the absent plugin. + */ + @Test + public void testAzureShapedPropertiesAreRefusedWhileTheAzurePluginIsAbsent() throws Exception { + StorageAdapter.initPluginManager(TestFileSystemPluginManagers.withoutProviders("AZURE")); + try { + CreateResourceInfo info = new CreateResourceInfo(false, false, "azure_res", AZURE_SHAPED); + AnalysisException refused = Assertions.assertThrows(AnalysisException.class, info::analyzeResourceType); + Assertions.assertTrue(refused.getMessage().contains("Azure Blob endpoint"), refused.getMessage()); + Assertions.assertTrue(refused.getMessage().contains("'AZURE' is not available"), refused.getMessage()); + + CreateResourceInfo s3 = new CreateResourceInfo(false, false, "s3_res", ImmutableMap.of( + "type", "s3", "s3.endpoint", "s3.us-east-1.amazonaws.com")); + s3.analyzeResourceType(); + Assertions.assertEquals(ResourceType.S3, s3.getResourceType()); + } finally { + StorageAdapter.initPluginManager(null); + } + } +} From 59f02704936f50d62497cbd38f00d57b105e6b54 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 15 Sep 2026 23:53:48 +0800 Subject: [PATCH 10/19] [fix](fe) Diagnose a plugin load failure without regexes, and never let the diagnosis itself throw The class-name patterns added for the missing-class advice were token(sep token)* group loops. java.util.regex recurses once per iteration of such a loop, so a NoClassDefFoundError or ClassNotFoundException whose message is a few thousand dot- or slash-separated tokens - a service file naming a legal-but-absurd class name, or plugin-authored wording - overflowed the FE's default 1 MB thread stack. That happened inside the catch block, so the StackOverflowError was not a PluginLoadException: the classloader leaked, loadAll recorded nothing, and the Error walked out to DorisFE's exit with 1024 frames of regex internals and no mention of the plugin or the class - the outcome the widened catches exist to remove, for inputs the base tolerated. Both patterns are replaced by a linear scan: a class name is tokens of name characters joined by single separators, where a name character is anything that is neither whitespace nor one of the JVM's own delimiters, so a Unicode or compiler-mangled name still counts and a sentence never does. The wrong-name form is split on its literal marker. Independently, every message built for a caught plugin failure now goes through one builder that computes the advice and the cause summary under catch (RuntimeException | Error) and degrades to "": a diagnostic can never again turn a caught failure into an escaped one. The name() and description() sites use the same builder, so a miss there is reported like a miss one line earlier. Two smaller defects in the same helper: HotSpot's wrong-name message is " (wrong name: )" and the sentence bound the two the other way round, sending the operator to the wrong entry; and the cause summary was empty for a cause-less LinkageError - VerifyError, ClassFormatError, UnsupportedClassVersionError - which is exactly the failure whose message-only consumers (the authorization manager's rejection list) had nothing else to record. Tests: a 66 KB class name through loadAll is a LoadFailure, not a throw; the wrong-name case is driven through loadAll with a jar whose entry path and bytecode name disagree, asserting which name follows "The class" and which "names it"; a class file with a patched major version yields a real UnsupportedClassVersionError whose text reaches the message; a parent that answers supertype resolution with a SecurityException is a load failure at the first site; the JDK 17 second-attempt chain is summarised; and the missing-class assertion now also gates the cause summary. Co-Authored-By: Claude Opus 5 --- .../loader/DirectoryPluginRuntimeManager.java | 118 ++++++++--- ...ectoryPluginRuntimeManagerLinkageTest.java | 190 +++++++++++++++++- 2 files changed, 268 insertions(+), 40 deletions(-) diff --git a/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManager.java b/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManager.java index 2fe41a3d0f005c..4ed4d023970144 100644 --- a/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManager.java +++ b/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManager.java @@ -39,8 +39,6 @@ import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentMap; import java.util.jar.JarFile; -import java.util.regex.Matcher; -import java.util.regex.Pattern; import java.util.stream.Collectors; import java.util.stream.Stream; @@ -115,14 +113,10 @@ public class DirectoryPluginRuntimeManager { */ private static final String ALREADY_FAILED_TO_INITIALIZE = "Could not initialize class "; /** - * A JVM {@code NoClassDefFoundError} message whose whole text is one class name, in either the binary - * ({@code org.example.Probe$Inner}) or the internal ({@code org/example/Probe$Inner}) spelling. Only - * such a message is read as "this class is missing"; anything sentence-shaped is plugin wording. + * The JVM's "found under the wrong path" wording, {@code (wrong name: )}: the class file exists under the requested name's path, its bytecode says otherwise. */ - private static final Pattern CLASS_NAME_MESSAGE = Pattern.compile("[\\w$]+([./][\\w$]+)*"); - /** The JVM's "found under the wrong path" wording: {@code a/B (wrong name: c/D)}. */ - private static final Pattern WRONG_NAME_MESSAGE = - Pattern.compile("([\\w$]+(?:[./][\\w$]+)*)\\s*\\(wrong name:\\s*([\\w$]+(?:[./][\\w$]+)*)\\)"); + private static final String WRONG_NAME_MARKER = " (wrong name: "; private final ConcurrentMap> handlesByName = new ConcurrentHashMap<>(); private final Object lifecycleLock = new Object(); @@ -299,8 +293,8 @@ private PluginHandle loadFromPluginDir(Path pluginDir, ClassLoader parent, Cl throw new PluginLoadException( normalizedDir, LoadFailure.STAGE_INSTANTIATE, - "Failed to instantiate factory class '" + factoryClassName + "' in " + normalizedDir - + missingClassAdvice(e) + rootCauseSummary(e), + failureMessage("Failed to instantiate factory class '" + factoryClassName + "'", + normalizedDir, e), e); } @@ -332,10 +326,10 @@ private PluginHandle loadFromPluginDir(Path pluginDir, ClassLoader parent, Cl throw new PluginLoadException( normalizedDir, LoadFailure.STAGE_INSTANTIATE, - "Failed to instantiate factory class '" + factoryClassName + "' in " + normalizedDir + failureMessage("Failed to instantiate factory class '" + factoryClassName + "'" + (e instanceof ClassCastException - ? ": it does not implement " + factoryType.getName() : "") - + missingClassAdvice(e) + rootCauseSummary(e), + ? ": it does not implement " + factoryType.getName() : ""), + normalizedDir, e), e); } } catch (PluginLoadException e) { @@ -353,7 +347,7 @@ private PluginHandle loadFromPluginDir(Path pluginDir, ClassLoader parent, Cl throw new PluginLoadException( normalizedDir, LoadFailure.STAGE_INSTANTIATE, - "Failed to get plugin name from discovered factory in " + normalizedDir, + failureMessage("Failed to get plugin name from discovered factory", normalizedDir, e), e); } String nameValidationError = PluginNames.validate(pluginName); @@ -377,7 +371,7 @@ private PluginHandle loadFromPluginDir(Path pluginDir, ClassLoader parent, Cl throw new PluginLoadException( normalizedDir, LoadFailure.STAGE_INSTANTIATE, - "Failed to get plugin description from discovered factory in " + normalizedDir, + failureMessage("Failed to get plugin description from discovered factory", normalizedDir, e), e); } @@ -561,6 +555,25 @@ private String discoverSingleFactoryClassName(Class factoryType, ClassLoader return classNames.get(0); } + /** + * The message of a failure that happened while the loader was calling into plugin code: what was + * being done, where, and - as far as the failure says so - why. Every site that catches a plugin + * failure builds its message here, so no site can fall behind the others in what it records. + * + *

The diagnostics read plugin-controlled text (the failure's messages), so they run under a guard + * that degrades to nothing: a diagnostic that throws would turn a failure the loader had already + * caught into one that escapes it, leaks the classloader and takes FE startup down. + */ + private static String failureMessage(String what, Path pluginDir, Throwable failure) { + String diagnostics; + try { + diagnostics = missingClassAdvice(failure) + rootCauseSummary(failure); + } catch (RuntimeException | Error e) { + diagnostics = ""; + } + return what + " in " + pluginDir + diagnostics; + } + /** * Turns a load failure caused by an absent class into an actionable sentence, or "" for any other * failure. A plugin misses a class either because it does not bundle it or because it expected to @@ -593,13 +606,13 @@ static String missingClassAdvice(Throwable failure) { // Keep walking: what really went missing, if anything did, is further down the chain. continue; } - Matcher wrongName = WRONG_NAME_MESSAGE.matcher(message); - if (wrongName.matches()) { - return ". The class " + wrongName.group(1) + " was found in this plugin's jars, but its" - + " bytecode names it " + wrongName.group(2) + ": the jar entry is under the wrong" - + " path, which is a broken plugin jar rather than a missing dependency"; + String[] wrongName = wrongNameForm(message); + if (wrongName != null) { + return ". The class " + wrongName[1] + " was found in this plugin's jars, but its bytecode" + + " names it " + wrongName[0] + ": the jar entry is under the wrong path, which is" + + " a broken plugin jar rather than a missing dependency"; } - if (!CLASS_NAME_MESSAGE.matcher(message).matches()) { + if (!isClassName(message)) { // Plugin code may wrap a lookup failure in a sentence ("Cannot load driver class // com.x.Y"); nothing here can tell which word is the class, so keep walking - the JDK's // own node, if there is one below, carries the bare name. @@ -613,23 +626,68 @@ static String missingClassAdvice(Throwable failure) { } /** - * The innermost cause of a load failure as {@code Class: message}, or "" when the failure has no - * cause, so that a consumer logging the {@link LoadFailure} message alone still records why - an - * {@link ExceptionInInitializerError} prints as its bare class name and says nothing on its own. - * Identity-bounded for the same reason as {@link #missingClassAdvice}. + * The innermost cause of a load failure as {@code Class: message} - or the failure itself when it + * has no cause - so that a consumer recording the {@link LoadFailure} message alone still records + * why: an {@link ExceptionInInitializerError} prints as its bare class name and says nothing on its + * own, and a cause-less {@link VerifyError} or {@link UnsupportedClassVersionError} is the whole + * story. Identity-bounded for the same reason as {@link #missingClassAdvice}. */ static String rootCauseSummary(Throwable failure) { + if (failure == null) { + return ""; + } Set seen = Collections.newSetFromMap(new IdentityHashMap<>()); - Throwable root = null; + Throwable root = failure; for (Throwable t = failure; t != null && seen.add(t); t = t.getCause()) { root = t; } - if (root == null || root == failure) { - return ""; - } return "; caused by " + root; } + /** + * Whether the text is one class name in the binary ({@code org.example.Probe$Inner}) or internal + * ({@code org/example/Probe$Inner}) spelling: tokens of name characters joined by one {@code .} or + * {@code /} each, no leading or trailing separator. A name character is anything that is neither + * whitespace nor one of the JVM's own delimiters, so a Unicode or compiler-mangled name still counts + * and a sentence never does. A linear scan on purpose: a {@code (token(sep token)*)} regex over + * java.util.regex recurses once per token, and the message is plugin-controlled text - a name of a + * few thousand tokens would overflow the FE's default thread stack inside a catch block. + */ + static boolean isClassName(String text) { + boolean expectToken = true; + for (int i = 0; i < text.length(); i++) { + char c = text.charAt(i); + if (c == '.' || c == '/') { + if (expectToken) { + return false; + } + expectToken = true; + } else if (Character.isWhitespace(c) || c == ';' || c == '[' || c == '(' || c == ')') { + return false; + } else { + expectToken = false; + } + } + return !expectToken; + } + + /** + * Splits the JVM's {@code (wrong name: )} message into its two class + * names, or returns null when the text has any other shape. + */ + private static String[] wrongNameForm(String message) { + int marker = message.indexOf(WRONG_NAME_MARKER); + if (marker <= 0 || !message.endsWith(")")) { + return null; + } + String bytecodeName = message.substring(0, marker); + String requestedName = message.substring(marker + WRONG_NAME_MARKER.length(), message.length() - 1); + if (!isClassName(bytecodeName) || !isClassName(requestedName)) { + return null; + } + return new String[] {bytecodeName, requestedName}; + } + private static void closeClassLoader(ClassLoader classLoader) { if (classLoader == null) { return; diff --git a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerLinkageTest.java b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerLinkageTest.java index 749cf3dab79200..28a1e9688817a1 100644 --- a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerLinkageTest.java +++ b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerLinkageTest.java @@ -20,6 +20,7 @@ import org.apache.doris.extension.loader.testplugins.AbsentDependencyProbe; import org.apache.doris.extension.loader.testplugins.AbsentStaticInitTestPluginFactory; import org.apache.doris.extension.loader.testplugins.AbsentSuperclassTestPluginFactory; +import org.apache.doris.extension.loader.testplugins.MetadataTestPluginFactory; import org.apache.doris.extension.loader.testplugins.NotAFactory; import org.apache.doris.extension.loader.testplugins.ThrowingStaticInitTestPluginFactory; import org.apache.doris.extension.spi.PluginFactory; @@ -29,10 +30,19 @@ import org.junit.jupiter.api.io.TempDir; import java.io.IOException; +import java.io.InputStream; import java.lang.reflect.InvocationTargetException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; import java.nio.file.Path; import java.time.Duration; import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.jar.Attributes; +import java.util.jar.JarEntry; +import java.util.jar.JarOutputStream; +import java.util.jar.Manifest; /** * A plugin whose dependency is absent must fail alone. @@ -190,15 +200,169 @@ void testOnlyAClassNameShapedMessageIsReadAsAMissingClass() { } @Test - void testTheWrongNameFormIsABrokenJarNotAMissingDependency() { - // "a/B (wrong name: c/D)" means the class file WAS found, under a path that disagrees with the - // name in its bytecode; sending the reader to the shared bundle would be exactly wrong. - String advice = DirectoryPluginRuntimeManager.missingClassAdvice( - new NoClassDefFoundError("org/example/Probe (wrong name: org/other/Probe)")); - Assertions.assertTrue(advice.contains("org/example/Probe"), advice); - Assertions.assertTrue(advice.contains("org/other/Probe"), advice); - Assertions.assertTrue(advice.contains("wrong path"), advice); - Assertions.assertFalse(advice.contains("shared library bundle"), advice); + void testTheWrongNameFormIsABrokenJarNotAMissingDependency() throws IOException { + // HotSpot's wording is " (wrong name: )": the class file + // WAS found, under the requested name's path, and its bytecode says otherwise. Sending the + // reader to the shared bundle would be exactly wrong. Driven through loadAll with a real jar + // whose entry path and bytecode name disagree, so the orientation of the sentence is checked + // against the JVM rather than against a fabricated string. + String requested = NotAFactory.class.getName().replace("NotAFactory", "Renamed"); + String entry = requested.replace('.', '/') + ".class"; + Map entries = new LinkedHashMap<>(); + entries.put(entry, classBytes(NotAFactory.class)); + Path root = tempDir.resolve("plugins"); + writeJar(root.resolve("wrong-name").resolve("wrong-name.jar"), requested, entries); + + LoadReport report = load(root); + Assertions.assertEquals(1, report.getFailures().size(), () -> "" + report.getSuccesses()); + String message = report.getFailures().get(0).getMessage(); + String bytecodeName = NotAFactory.class.getName().replace('.', '/'); + String requestedName = requested.replace('.', '/'); + Assertions.assertTrue(message.contains("The class " + requestedName + " was found in this plugin's jars"), + message); + Assertions.assertTrue(message.contains("names it " + bytecodeName + ":"), message); + Assertions.assertTrue(message.contains("wrong path"), message); + Assertions.assertFalse(message.contains("shared library bundle"), message); + } + + @Test + void testALongClassNameIsDiagnosedWithoutOverflowingTheStack() throws IOException { + // The message is plugin-controlled text: a service file naming a class of the longest legal + // length arrives as a ClassNotFoundException whose message is that name. A regex with one + // recursion per token overflowed the FE's default 1 MB stack here - inside the catch block, + // which turned a tolerated failure into an FE exit. + String longName = String.join(".", Collections.nCopies(6000, "abcdefghij")); + Path root = tempDir.resolve("plugins"); + writeJar(root.resolve("long-name").resolve("long-name.jar"), longName, Collections.emptyMap()); + + LoadReport report = load(root); + Assertions.assertEquals(1, report.getFailures().size()); + LoadFailure failure = report.getFailures().get(0); + Assertions.assertEquals(LoadFailure.STAGE_INSTANTIATE, failure.getStage()); + Assertions.assertTrue(failure.getMessage().contains("The class " + longName + " is"), + "the name is read as a class name, however long"); + + // The direct forms too, on both diagnostics. + Assertions.assertDoesNotThrow(() -> DirectoryPluginRuntimeManager.missingClassAdvice( + new NoClassDefFoundError(longName.replace('.', '/')))); + Assertions.assertDoesNotThrow(() -> DirectoryPluginRuntimeManager.missingClassAdvice( + new NoClassDefFoundError(longName.replace('.', '/') + " (wrong name: " + longName + ")"))); + } + + @Test + void testACauselessLinkageErrorIsSummarisedByItself() throws IOException { + // A class file compiled for a newer JDK fails with UnsupportedClassVersionError, which has no + // cause: the failure itself is the whole story, and the message-only consumers must get it. + byte[] bytes = classBytes(MetadataTestPluginFactory.class); + bytes[6] = (byte) 0xFF; + bytes[7] = (byte) 0xFF; + Map entries = new LinkedHashMap<>(); + entries.put(MetadataTestPluginFactory.class.getName().replace('.', '/') + ".class", bytes); + Path root = tempDir.resolve("plugins"); + writeJar(root.resolve("newer-jdk").resolve("newer-jdk.jar"), MetadataTestPluginFactory.class.getName(), + entries); + + LoadReport report = load(root); + Assertions.assertEquals(1, report.getFailures().size()); + LoadFailure failure = report.getFailures().get(0); + Assertions.assertInstanceOf(UnsupportedClassVersionError.class, failure.getCause()); + Assertions.assertTrue(failure.getMessage().contains("; caused by java.lang.UnsupportedClassVersionError"), + failure.getMessage()); + } + + @Test + void testARuntimeExceptionWhileResolvingTheFactoryClassIsAFailureNotAThrow() throws IOException { + // Site A: defining the factory class resolves its supertypes through the parent, and a parent + // may answer with a SecurityException (a signed package the plugin also ships unsigned classes + // into). A RuntimeException, so neither ReflectiveOperationException nor LinkageError. + Path root = tempDir.resolve("plugins"); + DirectoryPluginRuntimeManagerMetadataTest.createPluginJar( + root.resolve("refused-supertype").resolve("refused-supertype.jar"), + AbsentSuperclassTestPluginFactory.class, "1.0"); + ClassLoader refusingParent = new ClassLoader(getClass().getClassLoader()) { + @Override + protected Class loadClass(String name, boolean resolve) throws ClassNotFoundException { + if (AbsentDependencyProbe.class.getName().equals(name)) { + throw new SecurityException("signer information does not match: " + name); + } + return super.loadClass(name, resolve); + } + }; + + LoadReport report = load(root, refusingParent); + Assertions.assertEquals(1, report.getFailures().size()); + LoadFailure failure = report.getFailures().get(0); + Assertions.assertEquals(LoadFailure.STAGE_INSTANTIATE, failure.getStage()); + Assertions.assertInstanceOf(SecurityException.class, failure.getCause()); + Assertions.assertTrue(failure.getMessage().contains("; caused by java.lang.SecurityException: signer"), + failure.getMessage()); + } + + @Test + void testTheJdk17SecondAttemptShapeIsSummarised() { + // JDK 17's second attempt at a class whose initializer already failed: the miss is named only in + // the ExceptionInInitializerError's message, one node down. No advice (nothing new is missing), + // but the summary must carry that text. + NoClassDefFoundError secondAttempt = + new NoClassDefFoundError("Could not initialize class org.example.BrokenFactory"); + secondAttempt.initCause(new ExceptionInInitializerError( + "Exception java.lang.NoClassDefFoundError: org/example/Probe [in thread \"main\"]")); + Assertions.assertEquals("", DirectoryPluginRuntimeManager.missingClassAdvice(secondAttempt)); + Assertions.assertTrue(DirectoryPluginRuntimeManager.rootCauseSummary(secondAttempt) + .contains("java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError:" + + " org/example/Probe")); + } + + @Test + void testOnlyAClassNameShapedTextIsAClassName() { + Assertions.assertTrue(DirectoryPluginRuntimeManager.isClassName("org.example.Probe$Inner2")); + Assertions.assertTrue(DirectoryPluginRuntimeManager.isClassName("org/example/Probe$Inner2")); + Assertions.assertTrue(DirectoryPluginRuntimeManager.isClassName("Probe")); + Assertions.assertTrue(DirectoryPluginRuntimeManager.isClassName("org.example." + new String( + new int[] {0xdc, 'n', 0xef, 'c', 'o', 'd', 'e'}, 0, 7)), + "a Unicode identifier is a class name"); + Assertions.assertTrue(DirectoryPluginRuntimeManager.isClassName("org.example.Mangled-Name"), + "a compiler-mangled name is a class name"); + Assertions.assertFalse(DirectoryPluginRuntimeManager.isClassName("")); + Assertions.assertFalse(DirectoryPluginRuntimeManager.isClassName("com.example.Driver.")); + Assertions.assertFalse(DirectoryPluginRuntimeManager.isClassName(".com.example.Driver")); + Assertions.assertFalse(DirectoryPluginRuntimeManager.isClassName("com..example.Driver")); + Assertions.assertFalse(DirectoryPluginRuntimeManager.isClassName("Cannot load driver class com.x.Y")); + Assertions.assertFalse(DirectoryPluginRuntimeManager.isClassName("[Lorg/example/Probe;")); + Assertions.assertFalse(DirectoryPluginRuntimeManager.isClassName("org/example/Probe (wrong name: x/Y)")); + } + + private static byte[] classBytes(Class clazz) throws IOException { + String entry = clazz.getName().replace('.', '/') + ".class"; + try (InputStream in = clazz.getClassLoader().getResourceAsStream(entry)) { + Assertions.assertNotNull(in, "class bytes not found: " + entry); + return in.readAllBytes(); + } + } + + /** + * Writes a plugin jar whose service file names {@code serviceClassName} and whose class entries are + * exactly {@code entries} - which lets a test disagree with itself on purpose: an entry under one + * path carrying another class's bytes, a class file with a patched version, or no class at all. + */ + private static void writeJar(Path jarPath, String serviceClassName, Map entries) + throws IOException { + Files.createDirectories(jarPath.getParent()); + Manifest manifest = new Manifest(); + manifest.getMainAttributes().put(Attributes.Name.MANIFEST_VERSION, "1.0"); + manifest.getMainAttributes().putValue( + DirectoryPluginRuntimeManagerMetadataTest.TEST_GATE.getManifestAttribute(), + DirectoryPluginRuntimeManagerMetadataTest.TEST_GATE.getExpectedVersion()); + try (JarOutputStream jar = new JarOutputStream(Files.newOutputStream(jarPath), manifest)) { + for (Map.Entry entry : entries.entrySet()) { + jar.putNextEntry(new JarEntry(entry.getKey())); + jar.write(entry.getValue()); + jar.closeEntry(); + } + jar.putNextEntry(new JarEntry("META-INF/services/" + PluginFactory.class.getName())); + jar.write((serviceClassName + "\n").getBytes(StandardCharsets.UTF_8)); + jar.closeEntry(); + } } private static String adviceFor(Throwable failure) { @@ -224,11 +388,15 @@ protected Class loadClass(String name, boolean resolve) throws ClassNotFoundE return super.loadClass(name, resolve); } }; + return load(root, hidingParent); + } + + private static LoadReport load(Path root, ClassLoader parent) { // Assertions.assertDoesNotThrow is the point of the test: before the loader caught LinkageError // this call threw, and nothing between here and FE startup would have caught it. return Assertions.assertDoesNotThrow(() -> new DirectoryPluginRuntimeManager().loadAll( Collections.singletonList(root), - hidingParent, + parent, PluginFactory.class, null, DirectoryPluginRuntimeManagerMetadataTest.TEST_GATE)); @@ -248,5 +416,7 @@ private static void assertReportsMissingDependency(LoadReport rep Assertions.assertTrue(failure.getMessage().contains("shared library bundle"), () -> "the message must say where a missing dependency is expected to come from: " + failure.getMessage()); + Assertions.assertTrue(failure.getMessage().contains("; caused by java.lang.ClassNotFoundException: "), + () -> "the message must carry the innermost cause: " + failure.getMessage()); } } From ba673af70c2aecfe3e7293c0db7fb424c46dedfa Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 15 Sep 2026 23:57:30 +0800 Subject: [PATCH 11/19] [fix](fe) Do not bind a persisted job's storage provider at deserialisation StorageDesc.gsonPostProcess bound the filesystem provider - bindPrimary probes every loaded provider, then the claiming one's bind() - with no catch, for every persisted BrokerLoadJob and ExportJob: every LOAD ... WITH S3 / HDFS / BROKER of the last three days and every EXPORT of the last seven are in the image. With that provider absent - the state this branch makes an FE serve in - the bind threw an unchecked StoragePropertiesException: at image load it was wrapped into an IOException and the FE exited with "fail to load image"; on a follower whose image held no such job, the next OP_CREATE_LOAD_JOB threw inside BDBJournalCursor.next, whose catch exits the process, and the FE crash-looped on the same journal on every restart, unable ever to become master, with a log line naming only a journal key. The bind at deserialisation is now tolerant: a RuntimeException or LinkageError leaves the adapter unbound with one WARN naming the descriptor. Every getter already re-runs initStorageAdapter() lazily, so a job on an absent provider binds at its first use and fails there with a Status, persisting as CANCELLED, and a job whose provider is back binds as before. The two-argument BrokerDesc constructor, the same absent-provider case seen from the statement side: a map no provider claims fell back to a BROKER descriptor with whatever name the caller had - null for OUTFILE and the REST file API - so the statement failed at dispatch with "Unknown broker name(null)" and sent the operator to debug brokers. The fallback is kept for a statement that named a broker (its properties are the broker's own configuration) and the binding error, which lists the loaded providers, is rethrown otherwise. Tests: a BrokerLoadJob and a BrokerDesc round trip with their provider absent deserialise without binding, bind at use once the provider is back, and report the provider list at use meanwhile; a nameless BrokerDesc on an unclaimed map reports the providers instead of a broker while a named one keeps its fallback. Co-Authored-By: Claude Opus 5 --- .../org/apache/doris/analysis/BrokerDesc.java | 10 ++- .../apache/doris/analysis/StorageDesc.java | 17 +++- .../analysis/StorageDescPersistTest.java | 86 +++++++++++++++++++ 3 files changed, 110 insertions(+), 3 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/BrokerDesc.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/BrokerDesc.java index dad70399e5fe7f..8b71ea73510b1d 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/analysis/BrokerDesc.java +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/BrokerDesc.java @@ -88,8 +88,14 @@ public BrokerDesc(String name, Map properties) { // Override the storage type based on property configuration this.storageType = StorageBackend.StorageType.valueOfIgnoreCase(storageAdapter.getStorageName()); } catch (StoragePropertiesException e) { - // Currently ignored: these properties might be broker-specific. - // Just keep the storage type as BROKER, and bind a broker adapter + // A map no provider claims is a broker's own configuration when the statement named + // a broker: keep the storage type as BROKER and bind a broker adapter. Without a + // broker name (OUTFILE, the REST file API) nothing could serve it as BROKER either, + // and "Unknown broker name(null)" at dispatch would hide the real reason - typically + // that the filesystem provider for these properties is not loaded - so report that. + if (StringUtils.isBlank(name)) { + throw e; + } this.storageAdapter = StorageAdapter.ofBroker(name, this.properties); this.storageType = StorageBackend.StorageType.BROKER; } diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/StorageDesc.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/StorageDesc.java index 18978c085464c1..683fd3a4a675f3 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/analysis/StorageDesc.java +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/StorageDesc.java @@ -21,6 +21,8 @@ import org.apache.doris.persist.gson.GsonPostProcessable; import com.google.gson.annotations.SerializedName; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; import java.io.IOException; import java.util.HashMap; @@ -37,6 +39,8 @@ * The broker's StorageBackend.StorageType desc */ public class StorageDesc extends ResourceDesc implements GsonPostProcessable { + private static final Logger LOG = LogManager.getLogger(StorageDesc.class); + @Deprecated @SerializedName("st") @@ -110,6 +114,17 @@ public StorageAdapter getStorageAdapter() { @Override public void gsonPostProcess() throws IOException { - initStorageAdapter(); + // Binding runs plugin code: bindPrimary probes every loaded provider and bind() belongs to + // the one that claims the map. This method runs at image load and journal replay for every + // persisted load and export job, so with that provider absent - a filesystem plugin that + // failed to load - a throw here would take the image load down, or kill a serving follower + // at the next OP_CREATE_LOAD_JOB. Leave the adapter unbound instead: every getter re-runs + // initStorageAdapter() lazily, so the job binds at use and fails there with a Status. + try { + initStorageAdapter(); + } catch (RuntimeException | LinkageError e) { + LOG.warn("Storage descriptor (name={}, type={}) could not bind its filesystem provider at" + + " load; the binding is retried at use: {}", name, storageType, e.getMessage()); + } } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java index b029e9695099a5..85815b0c1ae9be 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java @@ -18,6 +18,8 @@ package org.apache.doris.analysis; import org.apache.doris.datasource.storage.StorageAdapter; +import org.apache.doris.foundation.property.StoragePropertiesException; +import org.apache.doris.fs.TestFileSystemPluginManagers; import org.apache.doris.load.EtlJobType; import org.apache.doris.load.loadv2.BrokerLoadJob; import org.apache.doris.persist.gson.GsonUtils; @@ -75,6 +77,90 @@ public void testBrokerLoadJobRestoreS3StoragePropertiesAfterGsonRoundTrip() thro restoredBrokerDesc.getStorageAdapter().getOrigProps().get("s3.bucket")); } + /** + * Deserialisation runs for every persisted load and export job at image load and journal replay. + * With the job's filesystem provider absent - the plugin failed to load - it must not throw: + * the FE would fail to load its image, or a serving follower would be killed by the next such + * journal. The binding is left for the first use, where it fails with a Status, and succeeds + * once the provider is back. + */ + @Test + public void testBrokerLoadJobRoundTripSurvivesAnAbsentProvider() throws Exception { + Map properties = Maps.newHashMap(); + properties.put("s3.endpoint", "s3.us-east-1.amazonaws.com"); + properties.put("s3.region", "us-east-1"); + properties.put("s3.access_key", "ak"); + properties.put("s3.secret_key", "sk"); + properties.put("s3.bucket", "test-bucket"); + BrokerDesc brokerDesc = new BrokerDesc("S3", StorageBackend.StorageType.S3, properties); + BrokerLoadJob job = new BrokerLoadJob(); + setField(BrokerLoadJob.class.getSuperclass(), job, "brokerDesc", brokerDesc); + String json = GsonUtils.GSON.toJson(job); + + StorageAdapter.initPluginManager(TestFileSystemPluginManagers.withoutProviders("S3")); + try { + BrokerLoadJob restored = Assertions.assertDoesNotThrow( + () -> GsonUtils.GSON.fromJson(json, BrokerLoadJob.class)); + BrokerDesc restoredDesc = (BrokerDesc) getField(BrokerLoadJob.class.getSuperclass(), restored, "brokerDesc"); + Assertions.assertNull(getField(StorageDesc.class, restoredDesc, "storageAdapter"), + "nothing binds at load while the provider is absent"); + Assertions.assertEquals("test-bucket", restoredDesc.getProperties().get("s3.bucket")); + // The first use binds, and reports the absent provider instead of a broker. + StoragePropertiesException atUse = + Assertions.assertThrows(StoragePropertiesException.class, restoredDesc::getStorageAdapter); + Assertions.assertTrue(atUse.getMessage().contains("Loaded filesystem providers"), atUse.getMessage()); + + StorageAdapter.initPluginManager(TestFileSystemPluginManagers.withoutProviders()); + Assertions.assertEquals("S3", restoredDesc.getStorageAdapter().getStorageName(), + "with the provider back, the same descriptor binds"); + } finally { + StorageAdapter.initPluginManager(null); + } + } + + @Test + public void testBrokerDescRoundTripSurvivesAnAbsentBrokerProvider() throws Exception { + Map properties = Maps.newHashMap(); + properties.put("broker.username", "user"); + String json = GsonUtils.GSON.toJson(new BrokerDesc("test_broker", properties)); + + StorageAdapter.initPluginManager(TestFileSystemPluginManagers.withoutProviders("Broker")); + try { + BrokerDesc restored = Assertions.assertDoesNotThrow(() -> GsonUtils.GSON.fromJson(json, BrokerDesc.class)); + Assertions.assertNull(getField(StorageDesc.class, restored, "storageAdapter")); + Assertions.assertThrows(StoragePropertiesException.class, restored::getStorageAdapter); + } finally { + StorageAdapter.initPluginManager(null); + } + } + + /** + * The two-argument constructor's broker fallback is for a statement that named a broker. Without + * one (OUTFILE, the REST file API) a map no provider claims is not a broker's, and the real reason + * - the provider is not loaded - must reach the user instead of "Unknown broker name(null)". + */ + @Test + public void testNamelessBrokerDescReportsTheAbsentProviderInsteadOfABroker() { + Map properties = Maps.newHashMap(); + properties.put("s3.endpoint", "s3.us-east-1.amazonaws.com"); + properties.put("s3.region", "us-east-1"); + properties.put("s3.access_key", "ak"); + properties.put("s3.secret_key", "sk"); + StorageAdapter.initPluginManager(TestFileSystemPluginManagers.withoutProviders("S3")); + try { + StoragePropertiesException refused = Assertions.assertThrows(StoragePropertiesException.class, + () -> new BrokerDesc(null, properties)); + Assertions.assertTrue(refused.getMessage().contains("Loaded filesystem providers"), refused.getMessage()); + + // A named broker keeps its fallback: its properties are the broker's own configuration. + BrokerDesc named = new BrokerDesc("my_broker", properties); + Assertions.assertEquals(StorageBackend.StorageType.BROKER, named.getStorageType()); + Assertions.assertEquals("my_broker", named.getStorageAdapter().getBrokerName()); + } finally { + StorageAdapter.initPluginManager(null); + } + } + private static void setField(Class clazz, Object target, String fieldName, Object value) throws ReflectiveOperationException { Field field = clazz.getDeclaredField(fieldName); From 73a9008a4fcacb0d6bbf75c10a9bb6daf87b8ea8 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 15 Sep 2026 23:57:53 +0800 Subject: [PATCH 12/19] [fix](fe) Make a repository whose provider is absent report on every path instead of throwing, and apply a replayed ALTER REPOSITORY as logged Four consequences of "the FE serves with a filesystem plugin absent" in the repository code, found by the review of the previous fix. A repository that already carries fs_descriptor - every one created or migrated on SPI code and checkpointed since - re-bound its provider on every getLocation() call and threw an unchecked StoragePropertiesException from every path that assembles a remote path, before acquireSpiFs's IOException or any Status was reached. A backup or restore job resumed across the restart threw out of run() on every 3 s tick, was never cancelled until its one-day timeout, held the BE snapshots it had taken, and BackupHandler ran no job after it in that cycle; ping() called getLocation() outside its try, so errMsg was never set and SHOW REPOSITORIES showed the repository healthy while the RepositoryMgr daemon logged a stack every ten minutes. The provider is now bound once, where the descriptor is (the constructor or gsonPostProcess): the normalised location and the SPI filesystem come from that one binding, getLocation() returns the raw location when it did not bind, and a binding failure sets the reason every consumer reports - getUnavailableReason(), errMsg for SHOW REPOSITORIES, the IOException of every I/O path, the Status of getBrokerAddress(), which is the first repository call every backup and restore job makes, so such a job is cancelled instead of looped. ping() never throws, and both loops - BackupHandler's over jobs and RepositoryMgr's over repositories - guard each item so one cannot stop the rest of the cycle. The legacy-migration rule read the name as a broker's whenever a broker of that name was registered. A typed legacy record names its storage type, and a broker may legally be called HDFS (ADD BROKER validates non-empty only), so with the HDFS plugin absent such a record still became BROKER "HDFS" and was persisted - the original defect through a narrower door. When the name is a shipped provider's and that provider is not loaded, its absence is what explains "no provider claims the properties", whatever brokers exist: the record is kept unmigrated. The migration binds also tolerate a LinkageError, which the earlier fix applied to the filesystem construction only. RepositoryMgr.alterRepo applied the command path's gate - the existing copy must carry an S3/AZURE descriptor - to replay too, and EditLog discards the returned Status. On an FE whose copy is the unmigrated record, a healthy master's ALTER REPOSITORY (credential rotation) was silently not applied; had that FE become master before a healthy checkpoint, its first checkpoint would have dropped the rotation for every FE. A replay is now applied as logged: the master validated the command and the replayed record carries its own descriptor. A replay whose repository does not exist locally is logged at ERROR, since nothing else records it. A BROKER descriptor's getBackendConfigProperties() re-bound the raw WITH BROKER properties by routing, which no provider claims, so the first BACKUP on a broker repository threw out of the job loop the same way; it is bound as a broker now, the way acquireSpiFs binds it. Tests: the descriptor-present/provider-absent shape reports through ping() (executed, not short-circuited), listSnapshots, getBrokerAddress, getInfo and SHOW CREATE, keeps its raw location, re-serialises unchanged and binds once the provider is back; the broker-name collision keeps the typed record unmigrated; a provider that throws NoClassDefFoundError during migration costs the record, not the load; an unmigrated record's ping keeps its reason across calls; a replayed ALTER replaces an unmigrated copy while the command path still refuses it; a job that throws does not stop the next job of the cycle; a BROKER descriptor answers its backend properties without routing. Co-Authored-By: Claude Opus 5 --- .../apache/doris/backup/BackupHandler.java | 15 +- .../org/apache/doris/backup/Repository.java | 148 ++++++++---- .../apache/doris/backup/RepositoryMgr.java | 25 +- .../datasource/storage/StorageAdapter.java | 10 + .../apache/doris/fs/FileSystemDescriptor.java | 8 +- .../commands/ShowCreateRepositoryCommand.java | 3 +- .../doris/backup/BackupHandlerTest.java | 23 ++ .../apache/doris/backup/RepositoryTest.java | 221 +++++++++++++++++- .../doris/fs/FileSystemDescriptorTest.java | 43 ++++ 9 files changed, 437 insertions(+), 59 deletions(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemDescriptorTest.java diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java b/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java index cd561e30646e6a..cce3939e729da7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java +++ b/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java @@ -193,7 +193,15 @@ protected void runAfterCatalogReady() { for (AbstractJob job : getAllCurrentJobs()) { job.setEnv(env); - job.run(); + try { + job.run(); + } catch (Exception | LinkageError e) { + // One job's throw must not skip every job after it in this cycle, nor go unrecorded: + // a job that keeps throwing here is retried each cycle until its timeout, so the log + // is the only place its cause shows up. + LOG.warn("backup/restore job {} threw out of run() and will be retried next cycle", + job.getJobId(), e); + } } } @@ -237,8 +245,9 @@ public void alterRepository(String repoName, Map newProps) if (oldRepo == null) { throw new DdlException("Repository does not exist"); } - if (!oldRepo.hasFileSystemDescriptor()) { - throw new DdlException("Repository " + repoName + " is not available: " + oldRepo.getErrorMsg()); + if (oldRepo.getUnavailableReason() != null) { + throw new DdlException("Repository " + repoName + " is not available: " + + oldRepo.getUnavailableReason()); } // Merge new properties with the existing repository's properties Map mergedProps = mergeProperties(oldRepo, newProps); diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java b/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java index 5c85a400dfd5b7..bfc7d38cb3ecd8 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java +++ b/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java @@ -30,6 +30,7 @@ import org.apache.doris.common.util.NetUtils; import org.apache.doris.common.util.TimeUtils; import org.apache.doris.datasource.storage.StorageAdapter; +import org.apache.doris.datasource.storage.StorageRegistry; import org.apache.doris.datasource.storage.StorageTypeId; import org.apache.doris.filesystem.DorisInputFile; import org.apache.doris.filesystem.DorisOutputFile; @@ -159,6 +160,22 @@ private static class LegacyFsRecord { * Null for BROKER repositories, which resolve a live broker endpoint per I/O call. */ private transient org.apache.doris.filesystem.FileSystem spiFs; + /** + * {@link #location} normalised by the bound provider (compat schemes such as {@code cos://} become + * the provider's own), resolved once where the provider is bound - the constructor or + * {@link #gsonPostProcess()}. Null for a BROKER repository and for a repository whose provider did + * not bind at load: {@link #getLocation()} then returns the raw location, so no path assembly + * re-binds the provider per call and throws past the I/O paths' error reporting. + */ + private transient String normalizedLocation; + + /** + * Why this repository cannot serve I/O, or null when it can: a legacy record whose migration + * {@link #gsonPostProcess()} declined, a corrupt record, or a descriptor whose filesystem provider + * did not bind at load (plugin absent, or the plugin rejected the properties). Not persisted. + */ + private transient String unavailableReason; + public FileSystemDescriptor getFileSystemDescriptor() { return fileSystemDescriptor; } @@ -178,13 +195,16 @@ public Repository(long id, String name, boolean isReadOnly, String location, this.fileSystemDescriptor = FileSystemDescriptor.fromStorageAdapter(storageAdapter, fsName); this.createTime = System.currentTimeMillis(); // Initialize SPI filesystem for I/O; broker resolves a live endpoint per I/O call - if (fileSystemDescriptor.getStorageType() != FsStorageType.BROKER - && !FeConstants.runningUnitTest) { - try { - this.spiFs = FileSystemFactory.getFileSystem(storageAdapter); - } catch (IOException e) { - throw new IllegalStateException( - "Failed to initialize SPI filesystem for repository '" + name + "': " + e.getMessage(), e); + if (fileSystemDescriptor.getStorageType() != FsStorageType.BROKER) { + this.normalizedLocation = storageAdapter.validateAndNormalizeUri(location); + if (!FeConstants.runningUnitTest) { + try { + this.spiFs = FileSystemFactory.getFileSystem(storageAdapter); + } catch (IOException e) { + throw new IllegalStateException( + "Failed to initialize SPI filesystem for repository '" + name + "': " + e.getMessage(), + e); + } } } } @@ -251,23 +271,32 @@ public void gsonPostProcess() { LOG.info("Repository '{}': migrating legacy 'fs' field to 'fs_descriptor'", name); Map props = legacyFs.properties != null ? legacyFs.properties : new HashMap<>(); String fsName = legacyFs.name != null ? legacyFs.name : ""; + // Both binds run plugin code (bindPrimary probes every loaded provider) at image load and + // edit-log replay: LinkageError included, so a half-installed plugin costs this + // repository, not the FE - the same catch CatalogFactory uses for catalogs. try { StorageAdapter storageAdapter = StorageAdapter.of(props); fileSystemDescriptor = FileSystemDescriptor.fromStorageAdapter(storageAdapter, ""); - } catch (RuntimeException e) { - // The legacy record's name is the storage type ("S3", "HDFS", ...) for a typed - // repository and the broker name for a broker one. A registered broker name is the one - // positive signal that the record was a broker repository, so only then is BROKER the - // right descriptor. Anything else that no provider claims is left as it was: the - // descriptor carries an explicit type and is persisted by the next checkpoint, so a - // BROKER fallback here would permanently rewrite an S3 repository whose plugin merely - // failed to load. Keeping the legacy record instead retries the migration at the next - // start, and every use of the repository reports the reason until then. - if (!isRegisteredBroker(fsName)) { - errMsg = "legacy filesystem metadata was not migrated: no loaded filesystem provider" - + " claims its properties and '" + fsName + "' is not a registered broker (" - + e.getMessage() + "). Install or repair the filesystem plugin for this" - + " repository and restart the FE, or DROP and re-CREATE it."; + } catch (RuntimeException | LinkageError e) { + // A typed legacy record names its storage type ("S3", "HDFS", "AZURE", ...); a broker + // record names its broker. The descriptor carries an explicit type and is persisted by + // the next checkpoint, so a wrong guess here is permanent: a typed record must never + // become BROKER because its plugin merely failed to load. When the name is a shipped + // provider's and that provider is not loaded, its absence explains why nothing claims + // the properties, whatever brokers are registered (a broker may legally be called + // HDFS). Only a name that is not an absent provider's and is a registered broker's is + // read as a broker record. Anything else is kept as it was, so the migration is + // retried at the next start, and every use reports the reason until then. + if (StorageRegistry.Provider.byName(fsName).isPresent() && !StorageAdapter.hasProvider(fsName)) { + unavailableReason = "legacy record of storage type '" + fsName + "' was not migrated:" + + " its filesystem provider is not loaded (" + e.getMessage() + ")"; + } else if (!isRegisteredBroker(fsName)) { + unavailableReason = "legacy record was not migrated: no loaded filesystem provider" + + " claims its properties (" + e.getMessage() + ") and '" + fsName + + "' is not a registered broker"; + } + if (unavailableReason != null) { + errMsg = unavailableReason; LOG.warn("Repository '{}': {}", name, errMsg); return; } @@ -276,40 +305,56 @@ public void gsonPostProcess() { try { StorageAdapter brokerAdapter = StorageAdapter.ofBroker(fsName, props); fileSystemDescriptor = FileSystemDescriptor.fromStorageAdapter(brokerAdapter, fsName); - } catch (RuntimeException e2) { - errMsg = "legacy filesystem metadata was not migrated: " + e2.getMessage(); + } catch (RuntimeException | LinkageError e2) { + unavailableReason = "legacy record was not migrated: " + e2.getMessage(); + errMsg = unavailableReason; LOG.error("Repository '{}': {}", name, errMsg); return; } } fsProps = fileSystemDescriptor.getProperties(); } else { - errMsg = "metadata corrupt: both 'fs' and 'fs_descriptor' fields are missing"; + unavailableReason = "metadata corrupt: both 'fs' and 'fs_descriptor' fields are missing"; + errMsg = unavailableReason; LOG.error("Repository '{}': {}", name, errMsg); return; } - // Initialize SPI filesystem for I/O; broker resolves a live endpoint per I/O call + // Bind the provider once: the normalised location and the SPI filesystem for I/O come from the + // same binding. A broker repository resolves a live endpoint per I/O call instead. if (fileSystemDescriptor.getStorageType() != FsStorageType.BROKER) { try { - this.spiFs = FileSystemFactory.getFileSystem(StorageAdapter.of(fsProps)); + StorageAdapter storageAdapter = StorageAdapter.of(fsProps); + this.normalizedLocation = storageAdapter.validateAndNormalizeUri(location); + this.spiFs = FileSystemFactory.getFileSystem(storageAdapter); } catch (IOException | RuntimeException | LinkageError e) { - // LinkageError included: this runs at image load and edit-log replay, where a plugin - // whose factory loaded but whose bind()/create() first links a jar it lacks must cost - // this repository, not the FE - the same catch CatalogFactory uses for catalogs. - LOG.warn("Failed to initialize SPI filesystem for repository {}: {}", name, e.getMessage()); + // Same reasoning as above: the plugin may be absent (the FE serves without it) or + // half-installed. The repository stays in metadata and reports this on every use. + unavailableReason = "filesystem provider did not bind at load: " + e.getMessage(); + errMsg = unavailableReason; + LOG.warn("Repository '{}': {}", name, errMsg); } } } /** - * Whether the repository has a storage descriptor. False only for a legacy record whose migration - * {@link #gsonPostProcess()} declined because no loaded filesystem provider claimed it; the record - * is then written back unchanged, {@link #getErrorMsg()} says why, and every I/O path fails with it. + * Whether the repository has a storage descriptor. False for a legacy record whose migration + * {@link #gsonPostProcess()} declined (no loaded provider claimed it, or its typed provider is absent) + * and for a corrupt record; the record is then written back unchanged and {@link #getErrorMsg()} + * says why. A repository with a descriptor can still be unusable - see {@link #getUnavailableReason()}. */ public boolean hasFileSystemDescriptor() { return fileSystemDescriptor != null; } + /** + * Why this repository cannot serve I/O right now, or null when it can: an unmigrated or corrupt + * record, or a descriptor whose filesystem provider did not bind at load. Every job and command + * that would use the repository checks this first and fails with it instead of throwing. + */ + public String getUnavailableReason() { + return unavailableReason; + } + private static boolean isRegisteredBroker(String brokerName) { if (Strings.isNullOrEmpty(brokerName)) { return false; @@ -331,12 +376,10 @@ public boolean isReadOnly() { } public String getLocation() { - if (fileSystemDescriptor == null - || fileSystemDescriptor.getStorageType() == FsStorageType.BROKER) { - return location; - } - return StorageAdapter.of(fileSystemDescriptor.getProperties()) - .validateAndNormalizeUri(location); + // Resolved once where the provider was bound; the raw location for a broker repository and + // for one whose provider did not bind (its I/O paths fail with the reason before any path + // assembled here is used). + return normalizedLocation != null ? normalizedLocation : location; } public String getErrorMsg() { @@ -355,8 +398,8 @@ private org.apache.doris.filesystem.FileSystem acquireSpiFs() throws IOException if (spiFs != null) { return spiFs; } - if (fileSystemDescriptor == null) { - throw new IOException("Repository '" + name + "' is not available: " + errMsg); + if (unavailableReason != null || fileSystemDescriptor == null) { + throw new IOException("Repository '" + name + "' is not available: " + unavailableReason); } if (fileSystemDescriptor.getStorageType() != FsStorageType.BROKER) { // spiFs should have been initialized in the constructor or gsonPostProcess. @@ -375,7 +418,8 @@ private org.apache.doris.filesystem.FileSystem acquireSpiFs() throws IOException FrontendOptions.getLocalHostAddress(), Config.edit_log_port); return FileSystemFactory.getBrokerFileSystem(broker.host, broker.port, clientId, brokerAdapter.getBrokerParams()); - } catch (AnalysisException e) { + } catch (AnalysisException | RuntimeException e) { + // RuntimeException: StoragePropertiesException when the broker provider itself is absent. throw new IOException("Failed to acquire broker filesystem for repository " + name + ": " + e.getMessage(), e); } @@ -542,9 +586,9 @@ public boolean ping() { if (FeConstants.runningUnitTest) { return true; } - if (fileSystemDescriptor == null) { - // Not migrated (see gsonPostProcess): errMsg already says why, and the periodic ping must - // not keep rewrapping it. + if (unavailableReason != null) { + // Not migrated, or the provider did not bind at load (see gsonPostProcess): errMsg already + // says why, and the periodic ping must not keep rewrapping it. return false; } // for s3 sdk, the headObject() method does not support list "dir", @@ -572,7 +616,9 @@ public boolean ping() { errMsg = TimeUtils.longToTimeString(System.currentTimeMillis()) + ": Invalid path. " + path + ", error: " + e.getMessage(); return false; - } catch (IOException e) { + } catch (IOException | RuntimeException e) { + // RuntimeException too: ping runs in the RepositoryMgr daemon over every repository, and one + // that throws would leave its errMsg unset and skip the repositories after it. errMsg = TimeUtils.longToTimeString(System.currentTimeMillis()) + ": " + e.getMessage(); return false; } @@ -862,8 +908,12 @@ public Status download(String remoteFilePath, String localFilePath) { } public Status getBrokerAddress(Long beId, Env env, List brokerAddrs) { - if (fileSystemDescriptor == null) { - return new Status(ErrCode.COMMON_ERROR, "Repository '" + name + "' is not available: " + errMsg); + if (unavailableReason != null) { + // The first repository call every backup and restore job makes before it reads the + // descriptor's properties: a job on an unusable repository fails here with a Status and is + // cancelled, instead of throwing out of run() on every tick. + return new Status(ErrCode.COMMON_ERROR, "Repository '" + name + "' is not available: " + + unavailableReason); } // get backend Backend be = Env.getCurrentSystemInfo().getBackend(beId); @@ -941,7 +991,7 @@ public List> getSnapshotInfos(String snapshotName, String timestamp public String getCreateStatement() { if (fileSystemDescriptor == null) { - throw new IllegalStateException("Repository '" + name + "' is not available: " + errMsg); + throw new IllegalStateException("Repository '" + name + "' is not available: " + unavailableReason); } StringBuilder stmtBuilder = new StringBuilder(); stmtBuilder.append("CREATE "); diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/RepositoryMgr.java b/fe/fe-core/src/main/java/org/apache/doris/backup/RepositoryMgr.java index 4526b77286017a..eb157447bd7d88 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/backup/RepositoryMgr.java +++ b/fe/fe-core/src/main/java/org/apache/doris/backup/RepositoryMgr.java @@ -59,8 +59,14 @@ public RepositoryMgr() { @Override protected void runOneCycle() { for (Repository repo : repoNameMap.values()) { - if (!repo.ping()) { - LOG.warn("Failed to connect repository {}. msg: {}", repo.getName(), repo.getErrorMsg()); + try { + if (!repo.ping()) { + LOG.warn("Failed to connect repository {}. msg: {}", repo.getName(), repo.getErrorMsg()); + } + } catch (Exception | LinkageError e) { + // ping() reports through errMsg and never throws by contract; if it ever does, the + // repositories after this one in the cycle must still be checked. + LOG.warn("Failed to ping repository {}", repo.getName(), e); } } } @@ -110,9 +116,15 @@ public Status alterRepo(Repository newRepo, boolean isReplay) { try { Repository repo = repoNameMap.get(newRepo.getName()); if (repo != null) { - if (repo.getFileSystemDescriptor() != null + // On replay the master has already validated the command against its own copy, and + // the replayed record carries its own descriptor. The local copy may differ - an + // unmigrated legacy record on an FE whose plugin is absent - and refusing on that + // basis would silently leave this FE behind the journal, permanently once it + // checkpoints. So a replay is applied as logged; only the command path checks. + if (isReplay + || (repo.getFileSystemDescriptor() != null && (repo.getFileSystemDescriptor().getStorageType() == FsStorageType.S3 - || repo.getFileSystemDescriptor().getStorageType() == FsStorageType.AZURE)) { + || repo.getFileSystemDescriptor().getStorageType() == FsStorageType.AZURE))) { repoNameMap.put(repo.getName(), newRepo); repoIdMap.put(repo.getId(), newRepo); @@ -126,6 +138,11 @@ public Status alterRepo(Repository newRepo, boolean isReplay) { return new Status(ErrCode.COMMON_ERROR, "Only support alter s3 repository"); } } + if (isReplay) { + // EditLog discards this Status: say so here, or the journal is silently not applied. + LOG.error("Replayed ALTER REPOSITORY for '{}' was not applied: the repository does not exist" + + " on this FE", newRepo.getName()); + } return new Status(ErrCode.NOT_FOUND, "repository does not exist"); } finally { lock.unlock(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java index 72b08397614429..9b4a44326dc719 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java @@ -179,6 +179,16 @@ public static StorageAdapter ofProvider(String providerName, Map "Filesystem provider '" + providerName + "' is not available"); } + /** Whether a provider of that name is loaded - the question {@link #ofProvider} answers by throwing. */ + public static boolean hasProvider(String providerName) { + for (FileSystemProvider provider : manager().getProviders()) { + if (provider.name().equalsIgnoreCase(providerName)) { + return true; + } + } + return false; + } + /** * Returns whether the named provider's guess heuristics claim the raw properties — the * facade twin of the legacy per-dialect {@code guessIsMe} statics (guess only, explicit diff --git a/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemDescriptor.java b/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemDescriptor.java index baee1968b03434..7901f5c76492dc 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemDescriptor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemDescriptor.java @@ -80,7 +80,13 @@ public StorageBackend.StorageType getThriftStorageType() { * by BE to initialize its storage client for snapshot/upload/download tasks. */ public Map getBackendConfigProperties() { - return StorageAdapter.of(properties).getBackendConfigProperties(); + // A BROKER descriptor carries the raw WITH BROKER properties, which routing does not claim + // (ofBroker documents that it bypasses routing): bind it the way it was bound, or the first + // BACKUP on a broker repository throws here out of the job loop. + StorageAdapter adapter = storageType == FsStorageType.BROKER + ? StorageAdapter.ofBroker(name, properties) + : StorageAdapter.of(properties); + return adapter.getBackendConfigProperties(); } /** diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateRepositoryCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateRepositoryCommand.java index 7a8586bead2990..aadd9690ec952f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateRepositoryCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateRepositoryCommand.java @@ -76,7 +76,8 @@ public ShowResultSet doRun(ConnectContext ctx, StmtExecutor executor) throws Exc throw new AnalysisException("repository not exist."); } if (!repo.hasFileSystemDescriptor()) { - throw new AnalysisException("repository " + repoName + " is not available: " + repo.getErrorMsg()); + throw new AnalysisException("repository " + repoName + " is not available: " + + repo.getUnavailableReason()); } rows.add(Lists.newArrayList(repoName, repo.getCreateStatement())); return new ShowResultSet(getMetaData(), rows); diff --git a/fe/fe-core/src/test/java/org/apache/doris/backup/BackupHandlerTest.java b/fe/fe-core/src/test/java/org/apache/doris/backup/BackupHandlerTest.java index e055cdb37f7e72..fa5e310681deab 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/backup/BackupHandlerTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/backup/BackupHandlerTest.java @@ -33,7 +33,9 @@ import org.apache.doris.catalog.info.TableNameInfo; import org.apache.doris.common.Config; import org.apache.doris.common.FeConstants; +import org.apache.doris.common.jmockit.Deencapsulation; import org.apache.doris.datasource.InternalCatalog; +import org.apache.doris.foundation.property.StoragePropertiesException; import org.apache.doris.info.TableRefInfo; import org.apache.doris.nereids.trees.plans.commands.BackupCommand; import org.apache.doris.nereids.trees.plans.commands.CancelBackupCommand; @@ -66,6 +68,7 @@ import java.nio.file.Path; import java.nio.file.Paths; import java.util.Comparator; +import java.util.Deque; import java.util.List; import java.util.Map; @@ -138,6 +141,26 @@ public void testInit() { Assertions.assertTrue(backupDir.exists()); } + /** + * One job throwing out of run() - a repository whose provider is absent used to do that on every + * tick - must neither stop the cycle for the jobs after it nor escape the daemon. + */ + @Test + public void testAJobThatThrowsDoesNotStopTheOtherJobsOfTheCycle() { + handler = new BackupHandler(env); + AbstractJob throwing = Mockito.mock(AbstractJob.class); + Mockito.doThrow(new StoragePropertiesException("No supported storage type found")).when(throwing).run(); + AbstractJob next = Mockito.mock(AbstractJob.class); + Map> jobs = Deencapsulation.getField(handler, "dbIdToBackupOrRestoreJobs"); + jobs.put(1L, Lists.newLinkedList(Lists.newArrayList(throwing))); + jobs.put(2L, Lists.newLinkedList(Lists.newArrayList(next))); + + Assertions.assertDoesNotThrow(() -> handler.runAfterCatalogReady()); + + Mockito.verify(throwing).run(); + Mockito.verify(next).run(); + } + @Test public void testCreateAndDropRepository() throws Exception { Mockito.when(brokerMgr.containsBroker(Mockito.anyString())).thenReturn(true); diff --git a/fe/fe-core/src/test/java/org/apache/doris/backup/RepositoryTest.java b/fe/fe-core/src/test/java/org/apache/doris/backup/RepositoryTest.java index 15fac6f75669a4..24d4f1191614b3 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/backup/RepositoryTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/backup/RepositoryTest.java @@ -30,9 +30,11 @@ import org.apache.doris.filesystem.FileEntry; import org.apache.doris.filesystem.FileIterator; import org.apache.doris.filesystem.Location; +import org.apache.doris.filesystem.spi.FileSystemProvider; import org.apache.doris.foundation.fs.FsStorageType; import org.apache.doris.fs.FileSystemDescriptor; import org.apache.doris.fs.FileSystemFactory; +import org.apache.doris.fs.FileSystemPluginManager; import org.apache.doris.fs.TestFileSystemPluginManagers; import org.apache.doris.persist.gson.GsonUtils; import org.apache.doris.service.FrontendOptions; @@ -541,7 +543,9 @@ public void testLegacyRecordIsKeptUnchangedWhileItsProviderIsAbsent() { Assertions.assertFalse(deserialized.hasFileSystemDescriptor(), "a record no provider claims must not be bound to a fallback type"); Assertions.assertNotNull(deserialized.getErrorMsg()); - Assertions.assertTrue(deserialized.getErrorMsg().contains("'HDFS' is not a registered broker"), + Assertions.assertTrue(deserialized.getErrorMsg().contains("'HDFS' was not migrated"), + deserialized.getErrorMsg()); + Assertions.assertTrue(deserialized.getErrorMsg().contains("provider is not loaded"), deserialized.getErrorMsg()); // What the next checkpoint writes: the legacy record, verbatim, and no descriptor. @@ -573,6 +577,221 @@ public void testLegacyRecordIsKeptUnchangedWhileItsProviderIsAbsent() { } } + /** + * A broker may legally be named like a storage type. With that type's plugin absent, the typed + * record still names an absent provider, and that - not the broker registry - explains why nothing + * claims its properties. Persisting BROKER "HDFS" here would be the original defect through a + * narrower door. + */ + @Test + public void testLegacyTypedRecordIsNotMigratedToABrokerOfTheSameNameWhileItsProviderIsAbsent() { + String legacyJson = "{" + + "\"id\":31000," + + "\"n\":\"hdfsRepoBrokerCollision\"," + + "\"iro\":false," + + "\"lo\":\"hdfs://ns/backup\"," + + "\"ct\":-1," + + "\"fs\":{\"n\":\"HDFS\",\"prop\":{\"hdfs.authentication.type\":\"simple\"}}" + + "}"; + Mockito.when(mockedBrokerMgr.containsBroker("HDFS")).thenReturn(true); + StorageAdapter.initPluginManager(TestFileSystemPluginManagers.withoutProviders("HDFS")); + try { + Repository deserialized = GsonUtils.GSON.fromJson(legacyJson, Repository.class); + Assertions.assertFalse(deserialized.hasFileSystemDescriptor(), + "a typed record whose provider is absent must not become BROKER, whatever brokers exist"); + Assertions.assertTrue(deserialized.getUnavailableReason().contains("'HDFS' was not migrated"), + deserialized.getUnavailableReason()); + Assertions.assertTrue(deserialized.getUnavailableReason().contains("provider is not loaded"), + deserialized.getUnavailableReason()); + Assertions.assertFalse(GsonUtils.GSON.toJson(deserialized).contains("fs_descriptor")); + } finally { + StorageAdapter.initPluginManager(null); + } + } + + /** + * The most common shape: a repository that already carries fs_descriptor, on an FE whose provider + * for it is absent. It must report that on every path a job or an operator takes - never re-bind + * per call and throw past the Status/errMsg reporting. + */ + @Test + public void testRepositoryWithDescriptorWhoseProviderIsAbsentReportsInsteadOfThrowing() { + String json = "{" + + "\"id\":32000," + + "\"n\":\"hdfsRepoDescriptor\"," + + "\"iro\":false," + + "\"lo\":\"hdfs://ns/backup\"," + + "\"ct\":-1," + + "\"fs_descriptor\":{\"fs_type\":\"HDFS\",\"fs_name\":\"\"," + + "\"fs_props\":{\"hdfs.authentication.type\":\"simple\"}}" + + "}"; + StorageAdapter.initPluginManager(TestFileSystemPluginManagers.withoutProviders("HDFS")); + try { + Repository deserialized = GsonUtils.GSON.fromJson(json, Repository.class); + Assertions.assertTrue(deserialized.hasFileSystemDescriptor(), "the descriptor is kept as it is"); + Assertions.assertNotNull(deserialized.getUnavailableReason()); + Assertions.assertTrue(deserialized.getUnavailableReason().contains("did not bind at load"), + deserialized.getUnavailableReason()); + Assertions.assertEquals(deserialized.getUnavailableReason(), deserialized.getErrorMsg(), + "SHOW REPOSITORIES shows the reason before the first ping"); + Assertions.assertEquals("hdfs://ns/backup", deserialized.getLocation(), + "the raw location, without re-binding the provider"); + + // The periodic ping keeps the reason as it is and does not throw. + FeConstants.runningUnitTest = false; + try { + Assertions.assertFalse(deserialized.ping()); + } finally { + FeConstants.runningUnitTest = true; + } + Assertions.assertEquals(deserialized.getUnavailableReason(), deserialized.getErrorMsg()); + + Status listing = deserialized.listSnapshots(Lists.newArrayList()); + Assertions.assertFalse(listing.ok()); + Assertions.assertTrue(listing.getErrMsg().contains("did not bind at load"), listing.getErrMsg()); + Status brokerAddress = deserialized.getBrokerAddress(1L, mockedEnv, Lists.newArrayList()); + Assertions.assertFalse(brokerAddress.ok(), "a job's first repository call fails with a Status"); + Assertions.assertTrue(brokerAddress.getErrMsg().contains("did not bind at load"), brokerAddress.getErrMsg()); + List info = deserialized.getInfo(); + Assertions.assertEquals("HDFS", info.get(6)); + Assertions.assertEquals(deserialized.getUnavailableReason(), info.get(7)); + // What it prints and re-serialises is unchanged. + Assertions.assertTrue(deserialized.getCreateStatement().contains("WITH HDFS")); + Assertions.assertTrue(GsonUtils.GSON.toJson(deserialized).contains("\"fs_type\":\"HDFS\"")); + + // The provider is back: the same record binds and is usable. + StorageAdapter.initPluginManager(TestFileSystemPluginManagers.withoutProviders()); + Repository repaired = GsonUtils.GSON.fromJson(json, Repository.class); + Assertions.assertNull(repaired.getUnavailableReason()); + Assertions.assertNull(repaired.getErrorMsg()); + } finally { + StorageAdapter.initPluginManager(null); + } + } + + /** The unmigrated shape's ping guard, executed rather than short-circuited by runningUnitTest. */ + @Test + public void testPingOfAnUnmigratedRecordReportsWithoutRewrappingTheReason() { + String legacyJson = "{" + + "\"id\":33000," + + "\"n\":\"hdfsRepoPing\"," + + "\"iro\":false," + + "\"lo\":\"hdfs://ns/backup\"," + + "\"ct\":-1," + + "\"fs\":{\"n\":\"HDFS\",\"prop\":{\"hdfs.authentication.type\":\"simple\"}}" + + "}"; + StorageAdapter.initPluginManager(TestFileSystemPluginManagers.withoutProviders("HDFS")); + try { + Repository deserialized = GsonUtils.GSON.fromJson(legacyJson, Repository.class); + String reason = deserialized.getErrorMsg(); + Assertions.assertNotNull(reason); + FeConstants.runningUnitTest = false; + try { + Assertions.assertFalse(deserialized.ping()); + Assertions.assertFalse(deserialized.ping()); + } finally { + FeConstants.runningUnitTest = true; + } + Assertions.assertEquals(reason, deserialized.getErrorMsg(), "ping must not rewrap the reason"); + } finally { + StorageAdapter.initPluginManager(null); + } + } + + /** + * The migration binds run plugin code at image load: a provider that fails to link there must cost + * this repository, not the FE. A fake provider that throws NoClassDefFoundError is registered under + * the name the registry consults first, so it is the first one bindPrimary probes. + */ + @Test + public void testLegacyMigrationSurvivesAProviderThatFailsToLink() { + String legacyJson = "{" + + "\"id\":34000," + + "\"n\":\"hdfsRepoLinkage\"," + + "\"iro\":false," + + "\"lo\":\"hdfs://ns/backup\"," + + "\"ct\":-1," + + "\"fs\":{\"n\":\"HDFS\",\"prop\":{\"hdfs.authentication.type\":\"simple\"}}" + + "}"; + FileSystemPluginManager manager = TestFileSystemPluginManagers.withoutProviders(); + manager.registerProvider(new FileSystemProvider() { + @Override + public String name() { + return "JFS"; + } + + @Override + public String description() { + return "a provider whose first call reaches a class it does not carry"; + } + + @Override + public boolean supports(Map properties) { + throw new NoClassDefFoundError("org/example/AbsentDependency"); + } + + @Override + public boolean supportsExplicit(Map properties) { + throw new NoClassDefFoundError("org/example/AbsentDependency"); + } + + @Override + public org.apache.doris.filesystem.FileSystem create(Map properties) { + throw new NoClassDefFoundError("org/example/AbsentDependency"); + } + + @Override + public org.apache.doris.extension.spi.Plugin create() { + return new org.apache.doris.extension.spi.Plugin() { + }; + } + }); + StorageAdapter.initPluginManager(manager); + try { + Repository deserialized = Assertions.assertDoesNotThrow( + () -> GsonUtils.GSON.fromJson(legacyJson, Repository.class)); + Assertions.assertFalse(deserialized.hasFileSystemDescriptor(), "the record is kept, not guessed"); + Assertions.assertTrue(deserialized.getUnavailableReason().contains("AbsentDependency"), + deserialized.getUnavailableReason()); + } finally { + StorageAdapter.initPluginManager(null); + } + } + + /** + * A replayed ALTER REPOSITORY is the master's validated decision; an FE whose copy is the + * unmigrated record must apply it as logged, or it silently diverges from the journal. + */ + @Test + public void testReplayedAlterRepositoryAppliesToAnUnmigratedCopy() { + String legacyJson = "{" + + "\"id\":35000," + + "\"n\":\"alteredRepo\"," + + "\"iro\":false," + + "\"lo\":\"hdfs://ns/backup\"," + + "\"ct\":-1," + + "\"fs\":{\"n\":\"HDFS\",\"prop\":{\"hdfs.authentication.type\":\"simple\"}}" + + "}"; + StorageAdapter.initPluginManager(TestFileSystemPluginManagers.withoutProviders("HDFS")); + try { + Repository unmigrated = GsonUtils.GSON.fromJson(legacyJson, Repository.class); + Assertions.assertFalse(unmigrated.hasFileSystemDescriptor()); + RepositoryMgr mgr = new RepositoryMgr(); + Assertions.assertTrue(mgr.addAndInitRepoIfNotExist(unmigrated, true).ok()); + + // The replayed record was built by the master and carries its descriptor. + Repository altered = new Repository(35000, "alteredRepo", false, "hdfs://ns/backup", + StorageAdapter.ofBroker("broker", Maps.newHashMap())); + Assertions.assertFalse(mgr.alterRepo(altered, false).ok(), + "the command path keeps refusing what it cannot validate locally"); + Assertions.assertTrue(mgr.alterRepo(altered, true).ok(), "a replay is applied as logged"); + Assertions.assertSame(altered, mgr.getRepo("alteredRepo")); + Assertions.assertSame(altered, mgr.getRepo(35000)); + } finally { + StorageAdapter.initPluginManager(null); + } + } + /** The broker fallback is taken on the one positive signal only: a registered broker name. */ @Test public void testLegacyRecordNamingAnUnregisteredBrokerIsNotMigratedToBroker() { diff --git a/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemDescriptorTest.java b/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemDescriptorTest.java new file mode 100644 index 00000000000000..39cf523e13e9c2 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemDescriptorTest.java @@ -0,0 +1,43 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.fs; + +import org.apache.doris.foundation.fs.FsStorageType; + +import com.google.common.collect.ImmutableMap; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Map; + +public class FileSystemDescriptorTest { + + /** + * A BROKER descriptor carries the raw WITH BROKER properties, which routing does not claim (the + * broker name is not a key of the map). Binding them by routing threw out of the first BACKUP on a + * broker repository; they have to be bound the way the repository was created. + */ + @Test + public void testBrokerDescriptorBackendPropertiesAreNotRouted() { + FileSystemDescriptor descriptor = new FileSystemDescriptor(FsStorageType.BROKER, "my_broker", + ImmutableMap.of("username", "u", "password", "p")); + Map backend = Assertions.assertDoesNotThrow(descriptor::getBackendConfigProperties); + Assertions.assertEquals("u", backend.get("username")); + Assertions.assertEquals("p", backend.get("password")); + } +} From 322d938a964bc27320e0d7df20b28cd1f00c7bfc Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 15 Sep 2026 23:58:10 +0800 Subject: [PATCH 13/19] [fix](fe) Decide an Azure resource by the whole of the Azure guess when the plugin is absent The Azure provider's supportsGuess is "provider=azure, or an endpoint whose host carries a recognised Azure Blob/DFS suffix". The plugin-absent fallback added to CREATE RESOURCE evaluated the endpoint leg only, so type=s3 with provider=azure and an endpoint whose host carries no such suffix - a custom domain, a private-link CNAME, a reverse proxy, the Azurite emulator - resolved to AZURE with the plugin present and fell through to an S3Resource with it absent; S3Resource accepts provider=AZURE, and ALTER RESOURCE cannot change a type after the plugin is repaired. The whole guess now lives in the SPI's AzureBlobEndpointSignals, next to the endpoint leg that MinIO already shares for its exclusion: the provider's supportsGuess delegates to it and so does fe-core's fallback, so the two cannot drift again. Tests: the SPI predicate answers both legs and neither; CREATE RESOURCE refuses provider=azure with a custom endpoint while the plugin is absent. Co-Authored-By: Claude Opus 5 --- .../commands/info/CreateResourceInfo.java | 15 +++++++------ .../commands/info/CreateResourceInfoTest.java | 12 ++++++++++- .../azure/AzureFileSystemProvider.java | 15 ++++++------- .../spi/AzureBlobEndpointSignals.java | 16 ++++++++++++++ .../spi/AzureBlobEndpointSignalsTest.java | 21 +++++++++++++++++++ 5 files changed, 61 insertions(+), 18 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateResourceInfo.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateResourceInfo.java index 686096f20e0b3d..c97a08ca167744 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateResourceInfo.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateResourceInfo.java @@ -101,15 +101,14 @@ public void analyzeResourceType() throws UserException { return; } } catch (StoragePropertiesException azureAbsent) { - // The Azure plugin is not loaded, so its guess cannot be asked - but the endpoint - // predicate it shares with the routing layer still can. An Azure-shaped map must be - // refused here rather than fall through to type=s3: the resource type is persisted and - // ALTER RESOURCE cannot change it, so an S3Resource created while the plugin was absent - // would stay one after the plugin is repaired. - if (AzureBlobEndpointSignals.guessIsAzureBlobEndpoint( - FileSystemPluginManager.withProbeContext(properties))) { + // The Azure plugin is not loaded, so its guess cannot be asked - but the predicate it is + // made of (provider=azure, or an Azure Blob endpoint) lives in the SPI and still can. An + // Azure-shaped map must be refused here rather than fall through to type=s3: the resource + // type is persisted and ALTER RESOURCE cannot change it, so an S3Resource created while + // the plugin was absent would stay one after the plugin is repaired. + if (AzureBlobEndpointSignals.guessIsAzure(FileSystemPluginManager.withProbeContext(properties))) { throw new AnalysisException("Cannot create resource '" + resourceName - + "': its endpoint is an Azure Blob endpoint, but " + azureAbsent.getMessage()); + + "': its properties select Azure Blob storage, but " + azureAbsent.getMessage()); } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateResourceInfoTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateResourceInfoTest.java index 04817d7e63b754..c62a546d1ff4a6 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateResourceInfoTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateResourceInfoTest.java @@ -52,9 +52,19 @@ public void testAzureShapedPropertiesAreRefusedWhileTheAzurePluginIsAbsent() thr try { CreateResourceInfo info = new CreateResourceInfo(false, false, "azure_res", AZURE_SHAPED); AnalysisException refused = Assertions.assertThrows(AnalysisException.class, info::analyzeResourceType); - Assertions.assertTrue(refused.getMessage().contains("Azure Blob endpoint"), refused.getMessage()); + Assertions.assertTrue(refused.getMessage().contains("select Azure Blob storage"), refused.getMessage()); Assertions.assertTrue(refused.getMessage().contains("'AZURE' is not available"), refused.getMessage()); + // The other leg of the provider's guess: provider=azure with an endpoint whose host carries + // no recognised suffix (a private link, a proxy, the Azurite emulator). + CreateResourceInfo byProvider = new CreateResourceInfo(false, false, "azure_res", ImmutableMap.of( + "type", "s3", "provider", "azure", "s3.endpoint", "https://storage.internal.example:10000", + "s3.access_key", "ak", "s3.secret_key", "sk")); + AnalysisException refusedByProvider = + Assertions.assertThrows(AnalysisException.class, byProvider::analyzeResourceType); + Assertions.assertTrue(refusedByProvider.getMessage().contains("'AZURE' is not available"), + refusedByProvider.getMessage()); + CreateResourceInfo s3 = new CreateResourceInfo(false, false, "s3_res", ImmutableMap.of( "type", "s3", "s3.endpoint", "s3.us-east-1.amazonaws.com")); s3.analyzeResourceType(); diff --git a/fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystemProvider.java index d76f0917675470..470b3a62d2b7d9 100644 --- a/fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystemProvider.java @@ -41,7 +41,7 @@ public class AzureFileSystemProvider implements FileSystemProvider properties) { @Override public boolean supportsGuess(Map properties) { // Verbatim port of fe-core AzureProperties.guessIsMe: provider=azure, or an endpoint - // alias whose HOST carries a recognised Azure Blob/DFS suffix. The suffix predicate - // (endpoint alias list, host extraction, dot-anchored endsWith, probe-injected live - // suffix list) is shared with the S3-compatible fallback providers via - // AzureBlobEndpointSignals so their mutual exclusion can never drift from this claim. - if ("azure".equalsIgnoreCase(properties.get(PROVIDER_KEY))) { - return true; - } - return AzureBlobEndpointSignals.guessIsAzureBlobEndpoint(properties); + // alias whose HOST carries a recognised Azure Blob/DFS suffix. The whole predicate lives in + // AzureBlobEndpointSignals: its endpoint leg is shared with the S3-compatible fallback + // providers (mutual exclusion), and the whole of it with fe-core, which has to answer the + // same question when this plugin is not loaded - neither may drift from this claim. + return AzureBlobEndpointSignals.guessIsAzure(properties); } @Override diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/AzureBlobEndpointSignals.java b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/AzureBlobEndpointSignals.java index 0faa8ed5d3e654..ff68e09d027131 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/AzureBlobEndpointSignals.java +++ b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/AzureBlobEndpointSignals.java @@ -77,6 +77,22 @@ public final class AzureBlobEndpointSignals { private AzureBlobEndpointSignals() { } + /** + * Explicit provider selection key; {@code provider=azure} claims the map without an endpoint. + */ + public static final String PROVIDER_KEY = "provider"; + + /** + * Azure's whole routing guess over the (probe view of the) properties - {@code provider=azure}, or + * an endpoint alias whose host carries a recognised Azure Blob/DFS suffix. The Azure provider's + * {@code supportsGuess} is exactly this, and so is fe-core's answer when that provider is not + * loaded: a persisted type must be decided by one predicate, or an absent plugin lets an + * Azure-shaped map through as something else. + */ + public static boolean guessIsAzure(Map properties) { + return "azure".equalsIgnoreCase(properties.get(PROVIDER_KEY)) || guessIsAzureBlobEndpoint(properties); + } + /** * The endpoint leg of Azure's routing guess over the full (probe view of the) properties: * takes the first present endpoint alias and matches its host against the live suffix list. diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/AzureBlobEndpointSignalsTest.java b/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/AzureBlobEndpointSignalsTest.java index 030ad017d72f71..64962560549080 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/AzureBlobEndpointSignalsTest.java +++ b/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/AzureBlobEndpointSignalsTest.java @@ -87,4 +87,25 @@ void guessConsultsAzureEndpointAliasOrder() { Assertions.assertFalse(AzureBlobEndpointSignals.guessIsAzureBlobEndpoint(new HashMap<>())); } + + @Test + void wholeGuessIsProviderKeyOrEndpoint() { + // The Azure provider's supportsGuess and fe-core's plugin-absent fallback both answer with + // this predicate; both legs have to be here or the two drift. + Map byProvider = new HashMap<>(); + byProvider.put("provider", "AzUrE"); + byProvider.put("s3.endpoint", "https://storage.internal.example:10000"); + Assertions.assertTrue(AzureBlobEndpointSignals.guessIsAzure(byProvider), + "provider=azure claims the map whatever the endpoint host"); + + Map byEndpoint = new HashMap<>(); + byEndpoint.put("s3.endpoint", "https://acct.blob.core.windows.net"); + Assertions.assertTrue(AzureBlobEndpointSignals.guessIsAzure(byEndpoint)); + + Map neither = new HashMap<>(); + neither.put("provider", "s3"); + neither.put("s3.endpoint", "https://storage.internal.example:10000"); + Assertions.assertFalse(AzureBlobEndpointSignals.guessIsAzure(neither)); + Assertions.assertFalse(AzureBlobEndpointSignals.guessIsAzure(new HashMap<>())); + } } From 4f9bacf75104a97ed7b45427ca11f004e4bdaf28 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 15 Sep 2026 23:58:10 +0800 Subject: [PATCH 14/19] [fix](fe) Refuse a connector provider that answers null engine names, and pin the masking parity gate to the provider list A directory connector plugin whose acceptedCreateTableEngineNames() returned null passed the first-call guard and threw NullPointerException one statement later, out of registerDiscovered and Env.initialize: the FE exited for the one answer the guard did not cover. A null set is refused like a blank type name. The masking parity test asserted that at least ten providers were on the test classpath while fourteen are; a provider leaving fe-core's test-scope dependency list, or a new fe-filesystem module added to the deploy loop but not there, would have narrowed the gate silently. It now names every shipped provider (OBS and COS when their class is on the classpath), and its case-insensitivity check spells its keys differently from the set's entries. Co-Authored-By: Claude Opus 5 --- .../connector/ConnectorPluginManager.java | 5 +++ .../util/DatasourcePrintableMapTest.java | 40 +++++++++++++++---- .../connector/ConnectorPluginManagerTest.java | 21 ++++++++++ 3 files changed, 59 insertions(+), 7 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java index 15f065411c295b..511c5d3ab24106 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java @@ -296,6 +296,11 @@ private static String typeNameProblem(String type) { * matter — mirroring how a duplicate catalog type is handled. */ private String createTableEngineNameProblem(Set engineNames) { + if (engineNames == null) { + // The SPI promises "empty if none"; a null answer is the one shape the loop below cannot + // take, and for a directory plugin it is that plugin's problem, not the FE's. + return "acceptedCreateTableEngineNames() returned null"; + } for (String engineName : engineNames) { if (engineName == null || engineName.trim().isEmpty()) { return "acceptedCreateTableEngineNames() returned a blank engine name"; diff --git a/fe/fe-core/src/test/java/org/apache/doris/common/util/DatasourcePrintableMapTest.java b/fe/fe-core/src/test/java/org/apache/doris/common/util/DatasourcePrintableMapTest.java index 5d083b26a96c1f..c79b27477c617e 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/common/util/DatasourcePrintableMapTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/common/util/DatasourcePrintableMapTest.java @@ -23,6 +23,7 @@ import org.junit.jupiter.api.Test; import java.util.ArrayList; +import java.util.Arrays; import java.util.HashMap; import java.util.List; import java.util.Map; @@ -89,24 +90,49 @@ public void testShippedFilesystemPluginAliasesAreMaskedWithoutThePlugin() { } } } - Assertions.assertTrue(providerNames.size() >= 10, - () -> "the shipped filesystem providers must be on the test classpath, found only " - + providerNames); + // Every shipped provider by name, so a module leaving fe-core's test-scope dependency list - + // or a new one added to the deploy loop but not here - fails this gate rather than silently + // narrowing it. OBS and COS are dropped from the test classpath by -Ddisable.obs / -Ddisable.cos. + List expected = new ArrayList<>(Arrays.asList("LOCAL", "HDFS", "OSS_HDFS", "JFS", "S3", "OSS", + "GCS", "MINIO", "OZONE", "AZURE", "HTTP", "Broker")); + if (isOnClasspath("org.apache.doris.filesystem.obs.ObsFileSystemProvider")) { + expected.add("OBS"); + } + if (isOnClasspath("org.apache.doris.filesystem.cos.CosFileSystemProvider")) { + expected.add("COS"); + } + Set found = new TreeSet<>(String.CASE_INSENSITIVE_ORDER); + found.addAll(providerNames); + Set missing = new TreeSet<>(String.CASE_INSENSITIVE_ORDER); + missing.addAll(expected); + missing.removeAll(found); + Assertions.assertTrue(missing.isEmpty(), + () -> "shipped filesystem providers missing from the test classpath: " + missing); Assertions.assertTrue(unmasked.isEmpty(), () -> "sensitive aliases printed in clear when the plugin declaring them is not loaded: " + unmasked); - // And the set is consulted case-insensitively by the printer itself, plugin or no plugin. + // And the set is consulted case-insensitively by the printer itself, plugin or no plugin: + // the keys below are spelled differently from the set's entries. Map map = new HashMap<>(); - map.put("cos.session_token", "sts-token"); - map.put("AZURE_CLIENT_SECRET", "client-secret"); - map.put("ozone.secret_key", "ozone-secret"); + map.put("COS.Session_Token", "sts-token"); + map.put("azure_client_secret", "client-secret"); + map.put("Ozone.Secret_Key", "ozone-secret"); String printed = new DatasourcePrintableMap<>(map, " = ", true, false, true).toString(); Assertions.assertFalse(printed.contains("sts-token"), printed); Assertions.assertFalse(printed.contains("client-secret"), printed); Assertions.assertFalse(printed.contains("ozone-secret"), printed); } + private static boolean isOnClasspath(String className) { + try { + Class.forName(className, false, DatasourcePrintableMapTest.class.getClassLoader()); + return true; + } catch (ClassNotFoundException e) { + return false; + } + } + @Test public void testBasicConstructor() { Map testMap = new HashMap<>(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorPluginManagerTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorPluginManagerTest.java index 282cc54c45c08e..23f55ff227c557 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorPluginManagerTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorPluginManagerTest.java @@ -358,6 +358,27 @@ public Connector create(Map properties, ConnectorContext context // Built-ins keep failing loudly: a classpath provider that cannot link is a broken FE build. Assertions.assertThrows(NoClassDefFoundError.class, () -> manager.registerDiscovered(unlinkable, true)); + + // The null answer is the other way a first call can go wrong, one statement later. + ConnectorProvider nullEngines = new ConnectorProvider() { + @Override + public String getType() { + return "null_engines"; + } + + @Override + public Set acceptedCreateTableEngineNames() { + return null; + } + + @Override + public Connector create(Map properties, ConnectorContext context) { + return null; + } + }; + Assertions.assertFalse(Assertions.assertDoesNotThrow(() -> manager.registerDiscovered(nullEngines, false)), + "a provider answering null engine names must be refused, not thrown"); + Assertions.assertTrue(manager.getRegisteredTypes().isEmpty(), "nothing may have been registered"); } private static ConnectorProvider createProviderWithEngines(String type, String... engineNames) { From 7881f7e5b2f7225d18de71711eae29a05c99d426 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 16 Sep 2026 02:06:53 +0800 Subject: [PATCH 15/19] [fix](fe) Keep the nameless BrokerDesc broker fallback ba673af70c2 made the two-argument BrokerDesc constructor rethrow the binding error when no broker name was given, so that OUTFILE on an absent provider reports the loaded providers instead of "Unknown broker name(null)". That message is part of the contract a regression test pins: export_p0/test_export_table_with_label_retry exports to a file:// path that routes nowhere and asserts exactly that text in the cancelled job's ErrorMsg, and cloud_p0 went red on it. The wording of that refusal is not this PR's concern; the fallback is restored as it was at the base, and the deserialisation-time tolerance in StorageDesc - the part of that commit the review asked for - stays. Co-Authored-By: Claude Opus 5 --- .../org/apache/doris/analysis/BrokerDesc.java | 10 ++----- .../analysis/StorageDescPersistTest.java | 27 ------------------- 2 files changed, 2 insertions(+), 35 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/BrokerDesc.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/BrokerDesc.java index 8b71ea73510b1d..dad70399e5fe7f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/analysis/BrokerDesc.java +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/BrokerDesc.java @@ -88,14 +88,8 @@ public BrokerDesc(String name, Map properties) { // Override the storage type based on property configuration this.storageType = StorageBackend.StorageType.valueOfIgnoreCase(storageAdapter.getStorageName()); } catch (StoragePropertiesException e) { - // A map no provider claims is a broker's own configuration when the statement named - // a broker: keep the storage type as BROKER and bind a broker adapter. Without a - // broker name (OUTFILE, the REST file API) nothing could serve it as BROKER either, - // and "Unknown broker name(null)" at dispatch would hide the real reason - typically - // that the filesystem provider for these properties is not loaded - so report that. - if (StringUtils.isBlank(name)) { - throw e; - } + // Currently ignored: these properties might be broker-specific. + // Just keep the storage type as BROKER, and bind a broker adapter this.storageAdapter = StorageAdapter.ofBroker(name, this.properties); this.storageType = StorageBackend.StorageType.BROKER; } diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java index 85815b0c1ae9be..73350836a2fdd5 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java @@ -134,33 +134,6 @@ public void testBrokerDescRoundTripSurvivesAnAbsentBrokerProvider() throws Excep } } - /** - * The two-argument constructor's broker fallback is for a statement that named a broker. Without - * one (OUTFILE, the REST file API) a map no provider claims is not a broker's, and the real reason - * - the provider is not loaded - must reach the user instead of "Unknown broker name(null)". - */ - @Test - public void testNamelessBrokerDescReportsTheAbsentProviderInsteadOfABroker() { - Map properties = Maps.newHashMap(); - properties.put("s3.endpoint", "s3.us-east-1.amazonaws.com"); - properties.put("s3.region", "us-east-1"); - properties.put("s3.access_key", "ak"); - properties.put("s3.secret_key", "sk"); - StorageAdapter.initPluginManager(TestFileSystemPluginManagers.withoutProviders("S3")); - try { - StoragePropertiesException refused = Assertions.assertThrows(StoragePropertiesException.class, - () -> new BrokerDesc(null, properties)); - Assertions.assertTrue(refused.getMessage().contains("Loaded filesystem providers"), refused.getMessage()); - - // A named broker keeps its fallback: its properties are the broker's own configuration. - BrokerDesc named = new BrokerDesc("my_broker", properties); - Assertions.assertEquals(StorageBackend.StorageType.BROKER, named.getStorageType()); - Assertions.assertEquals("my_broker", named.getStorageAdapter().getBrokerName()); - } finally { - StorageAdapter.initPluginManager(null); - } - } - private static void setField(Class clazz, Object target, String fieldName, Object value) throws ReflectiveOperationException { Field field = clazz.getDeclaredField(fieldName); From c1fc199b3a4ef7760ce1dd9fb71ee2888ca6d5e4 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 16 Sep 2026 04:46:15 +0800 Subject: [PATCH 16/19] [fix](fe) Never read a storage-type name as a broker's, let ALTER repair an unbound repository, and roll back a refused lineage plugin Follow-ups to the review-bot findings on the previous head. A typed legacy record whose provider is loaded but threw or rejected the properties during migration, with a broker of the same name registered, still took the broker fallback: the rule only recognised an absent provider as the explanation, so a provider failure plus a same-name broker persisted the record as BROKER - the original defect through one more door. A name that is a storage type's (a shipped provider other than BROKER, which is a family a broker may well be named after) is now never read as a broker's: the record is kept with the reason, absent or failed. Only a name that is no storage type's and is a registered broker's is a broker record. ALTER REPOSITORY refused every unavailable repository, including one whose descriptor merely did not bind at load because a formerly accepted property is now rejected - the case ALTER exists to repair. It now refuses only a record with no descriptor to merge into, binds the merged properties itself and reports the binding's reason as the DDL error when they still do not. A broker repository binds per call and nothing checks its provider at load, so a resumed backup or restore passed getBrokerAddress() and threw from StorageAdapter.ofBroker() while constructing its task. getBrokerAddress() binds the broker descriptor and answers with a Status instead. A lineage plugin refused at create()/initialize() left its factory in the map (retaining the discarded classloader) and an instance whose initialize() had failed unclosed. Both go with the plugin now; the inventory row stays, since the registry has no removal and a row for a plugin that was found but refused is what an operator reading information_schema.extensions after the WARN needs. A legacy record whose broker is not registered says to restart after ADD BROKER: the migration is retried at load only. Tests: a same-name broker with a throwing provider keeps the typed record; a repository unbound at load is repaired by ALTER and a still-unbindable correction is refused with the reason; a lineage plugin whose initialize() throws is closed and its factory dropped. Co-Authored-By: Claude Opus 5 --- .../apache/doris/backup/BackupHandler.java | 14 +++- .../org/apache/doris/backup/Repository.java | 44 ++++++++--- .../lineage/LineageEventProcessor.java | 22 +++++- .../doris/backup/BackupHandlerTest.java | 45 +++++++++++ .../apache/doris/backup/RepositoryTest.java | 64 ++++++++++++---- .../lineage/LineageEventProcessorTest.java | 76 +++++++++++++++++++ 6 files changed, 238 insertions(+), 27 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java b/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java index cce3939e729da7..e2c21978f3f1e4 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java +++ b/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java @@ -245,16 +245,26 @@ public void alterRepository(String repoName, Map newProps) if (oldRepo == null) { throw new DdlException("Repository does not exist"); } - if (oldRepo.getUnavailableReason() != null) { + if (!oldRepo.hasFileSystemDescriptor()) { + // Nothing to merge into: an unmigrated legacy or corrupt record. A repository whose + // descriptor merely did not bind at load is exactly what ALTER is for - the corrected + // properties are bound below and refused with their reason if they still do not. throw new DdlException("Repository " + repoName + " is not available: " + oldRepo.getUnavailableReason()); } // Merge new properties with the existing repository's properties Map mergedProps = mergeProperties(oldRepo, newProps); + StorageAdapter mergedStorage; + try { + mergedStorage = StorageAdapter.of(mergedProps); + } catch (RuntimeException | LinkageError e) { + throw new DdlException("Failed to alter repository " + repoName + + ": the merged properties do not bind a filesystem provider: " + e.getMessage()); + } // Create new Repository instance with merged properties Repository newRepo = new Repository( oldRepo.getId(), oldRepo.getName(), oldRepo.isReadOnly(), - oldRepo.getLocation(), StorageAdapter.of(mergedProps) + oldRepo.getLocation(), mergedStorage ); // Verify the repository can be connected with new settings if (!newRepo.ping()) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java b/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java index bfc7d38cb3ecd8..b68792c5884b76 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java +++ b/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java @@ -281,19 +281,23 @@ public void gsonPostProcess() { // A typed legacy record names its storage type ("S3", "HDFS", "AZURE", ...); a broker // record names its broker. The descriptor carries an explicit type and is persisted by // the next checkpoint, so a wrong guess here is permanent: a typed record must never - // become BROKER because its plugin merely failed to load. When the name is a shipped - // provider's and that provider is not loaded, its absence explains why nothing claims - // the properties, whatever brokers are registered (a broker may legally be called - // HDFS). Only a name that is not an absent provider's and is a registered broker's is - // read as a broker record. Anything else is kept as it was, so the migration is - // retried at the next start, and every use reports the reason until then. - if (StorageRegistry.Provider.byName(fsName).isPresent() && !StorageAdapter.hasProvider(fsName)) { - unavailableReason = "legacy record of storage type '" + fsName + "' was not migrated:" - + " its filesystem provider is not loaded (" + e.getMessage() + ")"; + // become BROKER because its plugin failed to load, threw, or rejected the properties. + // So a name that is a shipped provider's is never read as a broker's, whatever brokers + // are registered (a broker may legally be called HDFS): the record is kept, with the + // reason - the provider is absent, or it is loaded and the binding failed. Only a name + // that is not a provider's and is a registered broker's is read as a broker record. + // Anything else is kept as it was, so the migration is retried at the next start, and + // every use reports the reason until then. + if (isStorageTypeName(fsName)) { + unavailableReason = "legacy record of storage type '" + fsName + "' was not migrated: " + + (StorageAdapter.hasProvider(fsName) + ? "its filesystem provider is loaded but did not bind the properties (" + : "its filesystem provider is not loaded (") + + e.getMessage() + "). Repair the plugin or the properties and restart the FE."; } else if (!isRegisteredBroker(fsName)) { unavailableReason = "legacy record was not migrated: no loaded filesystem provider" + " claims its properties (" + e.getMessage() + ") and '" + fsName - + "' is not a registered broker"; + + "' is not a registered broker. After ADD BROKER, restart the FE."; } if (unavailableReason != null) { errMsg = unavailableReason; @@ -355,6 +359,17 @@ public String getUnavailableReason() { return unavailableReason; } + /** + * Whether a legacy record's name is a storage type's - the name of a shipped provider other than + * BROKER, which is a family, not a type: a broker record names its broker, and a broker may be + * called "broker". + */ + private static boolean isStorageTypeName(String fsName) { + return StorageRegistry.Provider.byName(fsName) + .filter(provider -> provider != StorageRegistry.Provider.BROKER) + .isPresent(); + } + private static boolean isRegisteredBroker(String brokerName) { if (Strings.isNullOrEmpty(brokerName)) { return false; @@ -926,6 +941,15 @@ public Status getBrokerAddress(Long beId, Env env, List brokerAddrs) { brokerAddrs.add(new FsBroker("127.0.0.1", 0)); return Status.OK; } + // A broker repository binds per call and nothing checked its provider at load: bind here, so + // a missing or failing broker provider is this job's Status rather than a throw from the + // task construction that follows. + try { + fileSystemDescriptor.getBackendConfigProperties(); + } catch (RuntimeException | LinkageError e) { + return new Status(ErrCode.COMMON_ERROR, "Repository '" + name + "' is not available: " + + e.getMessage()); + } // get proper broker for this backend FsBroker brokerAddr = null; diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/lineage/LineageEventProcessor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/lineage/LineageEventProcessor.java index 5138cda08e5db2..d57c9d80dac8a9 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/lineage/LineageEventProcessor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/lineage/LineageEventProcessor.java @@ -209,12 +209,13 @@ pluginRoots, getClass().getClassLoader(), LOG.info("Skip lineage plugin not in activate_lineage_plugin: {}", pluginName); continue; } + LineagePlugin plugin = null; try { Map props = new HashMap<>(); props.put("plugin.path", resolvePluginPath(pluginName)); props.put("plugin.name", pluginName); PluginContext context = new PluginContext(props); - LineagePlugin plugin = entry.getValue().create(context); + plugin = entry.getValue().create(context); if (plugin != null) { plugin.initialize(context); plugins.add(plugin); @@ -223,14 +224,31 @@ pluginRoots, getClass().getClassLoader(), } catch (Exception | LinkageError e) { // create() and initialize() are the first calls into the plugin implementation (the // factory only named it): a dependency it lacks arrives here as NoClassDefFoundError, - // which must cost this plugin alone, not FE startup. Its classloader goes with it. + // which must cost this plugin alone, not FE startup. Everything published for it goes + // with it: the factory (which would otherwise retain the classloader), the classloader, + // and an instance whose initialize() failed. The inventory row stays: the registry has + // no removal, and a row for a plugin that was found but refused is what an operator + // reading information_schema.extensions after this WARN needs to see. LOG.warn("Failed to create/initialize lineage plugin: {}", pluginName, e); + factories.remove(pluginName); + closeQuietly(plugin, pluginName); runtimeManager.discard(pluginName); } } initPlugins(plugins); } + private static void closeQuietly(LineagePlugin plugin, String pluginName) { + if (plugin == null) { + return; + } + try { + plugin.close(); + } catch (Exception | LinkageError e) { + LOG.warn("Failed to close lineage plugin {} after its initialization failed", pluginName, e); + } + } + private String safeFactoryName(LineagePluginFactory factory) { if (factory == null) { return ""; diff --git a/fe/fe-core/src/test/java/org/apache/doris/backup/BackupHandlerTest.java b/fe/fe-core/src/test/java/org/apache/doris/backup/BackupHandlerTest.java index fa5e310681deab..6c840d550e6182 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/backup/BackupHandlerTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/backup/BackupHandlerTest.java @@ -32,6 +32,7 @@ import org.apache.doris.catalog.TabletInvertedIndex; import org.apache.doris.catalog.info.TableNameInfo; import org.apache.doris.common.Config; +import org.apache.doris.common.DdlException; import org.apache.doris.common.FeConstants; import org.apache.doris.common.jmockit.Deencapsulation; import org.apache.doris.datasource.InternalCatalog; @@ -43,6 +44,7 @@ import org.apache.doris.nereids.trees.plans.commands.RestoreCommand; import org.apache.doris.nereids.trees.plans.commands.info.LabelNameInfo; import org.apache.doris.persist.EditLog; +import org.apache.doris.persist.gson.GsonUtils; import org.apache.doris.task.DirMoveTask; import org.apache.doris.task.DownloadTask; import org.apache.doris.task.SnapshotTask; @@ -141,6 +143,49 @@ public void testInit() { Assertions.assertTrue(backupDir.exists()); } + /** + * A repository whose descriptor did not bind at load (properties the provider now rejects) is what + * ALTER REPOSITORY exists to repair: the corrected properties bind, and the replacement is usable. + * Only a record with no descriptor at all - unmigrated or corrupt - has nothing to merge into. + */ + @Test + public void testAlterRepairsARepositoryWhoseDescriptorDidNotBindAtLoad() throws Exception { + handler = new BackupHandler(env); + String json = "{" + + "\"id\":50000," + + "\"n\":\"s3RepoToRepair\"," + + "\"iro\":false," + + "\"lo\":\"s3://my-bucket/backup\"," + + "\"ct\":-1," + + "\"fs_descriptor\":{\"fs_type\":\"S3\",\"fs_name\":\"\"," + + "\"fs_props\":{\"s3.access_key\":\"ak\",\"s3.secret_key\":\"sk\"}}" + + "}"; + Repository broken = GsonUtils.GSON.fromJson(json, Repository.class); + Assertions.assertTrue(broken.hasFileSystemDescriptor()); + Assertions.assertNotNull(broken.getUnavailableReason(), "no endpoint: the S3 provider cannot bind this"); + Assertions.assertTrue(handler.getRepoMgr().addAndInitRepoIfNotExist(broken, true).ok()); + + Map correction = Maps.newHashMap(); + correction.put("s3.endpoint", "s3.us-east-1.amazonaws.com"); + correction.put("s3.region", "us-east-1"); + handler.alterRepository("s3RepoToRepair", correction); + + Repository repaired = handler.getRepoMgr().getRepo("s3RepoToRepair"); + Assertions.assertNotSame(broken, repaired); + Assertions.assertNull(repaired.getUnavailableReason()); + Assertions.assertEquals("ak", repaired.getFileSystemDescriptor().getProperties().get("s3.access_key")); + Assertions.assertEquals("us-east-1", repaired.getFileSystemDescriptor().getProperties().get("s3.region")); + + // Still not bindable: refused with the binding's reason, and the record is left as it was. + Map stillBroken = Maps.newHashMap(); + stillBroken.put("s3.endpoint", ""); + stillBroken.put("s3.region", ""); + DdlException refused = Assertions.assertThrows(DdlException.class, + () -> handler.alterRepository("s3RepoToRepair", stillBroken)); + Assertions.assertTrue(refused.getMessage().contains("do not bind a filesystem provider"), refused.getMessage()); + Assertions.assertSame(repaired, handler.getRepoMgr().getRepo("s3RepoToRepair")); + } + /** * One job throwing out of run() - a repository whose provider is absent used to do that on every * tick - must neither stop the cycle for the jobs after it nor escape the daemon. diff --git a/fe/fe-core/src/test/java/org/apache/doris/backup/RepositoryTest.java b/fe/fe-core/src/test/java/org/apache/doris/backup/RepositoryTest.java index 24d4f1191614b3..de9261ab79daa4 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/backup/RepositoryTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/backup/RepositoryTest.java @@ -698,6 +698,38 @@ public void testPingOfAnUnmigratedRecordReportsWithoutRewrappingTheReason() { } } + /** + * The other half of the collision: the typed record's provider is loaded but its binding throws + * (a half-installed plugin, or properties it now rejects), and a broker of the same name exists. + * A provider failure is not "this is a broker record" either: the record stays untouched. + */ + @Test + public void testLegacyTypedRecordIsNotMigratedToABrokerOfTheSameNameWhenItsProviderFails() { + String legacyJson = "{" + + "\"id\":31500," + + "\"n\":\"hdfsRepoBrokerCollisionThrowing\"," + + "\"iro\":false," + + "\"lo\":\"hdfs://ns/backup\"," + + "\"ct\":-1," + + "\"fs\":{\"n\":\"HDFS\",\"prop\":{\"hdfs.authentication.type\":\"simple\"}}" + + "}"; + Mockito.when(mockedBrokerMgr.containsBroker("HDFS")).thenReturn(true); + FileSystemPluginManager manager = TestFileSystemPluginManagers.withoutProviders(); + manager.registerProvider(throwingProvider("HDFS")); + StorageAdapter.initPluginManager(manager); + try { + Repository deserialized = Assertions.assertDoesNotThrow( + () -> GsonUtils.GSON.fromJson(legacyJson, Repository.class)); + Assertions.assertFalse(deserialized.hasFileSystemDescriptor(), + "a provider that threw is not a refusal, and a same-name broker is not a signal"); + Assertions.assertTrue(deserialized.getUnavailableReason().contains("loaded but did not bind"), + deserialized.getUnavailableReason()); + Assertions.assertFalse(GsonUtils.GSON.toJson(deserialized).contains("fs_descriptor")); + } finally { + StorageAdapter.initPluginManager(null); + } + } + /** * The migration binds run plugin code at image load: a provider that fails to link there must cost * this repository, not the FE. A fake provider that throws NoClassDefFoundError is registered under @@ -714,10 +746,26 @@ public void testLegacyMigrationSurvivesAProviderThatFailsToLink() { + "\"fs\":{\"n\":\"HDFS\",\"prop\":{\"hdfs.authentication.type\":\"simple\"}}" + "}"; FileSystemPluginManager manager = TestFileSystemPluginManagers.withoutProviders(); - manager.registerProvider(new FileSystemProvider() { + manager.registerProvider(throwingProvider("JFS")); + StorageAdapter.initPluginManager(manager); + try { + Repository deserialized = Assertions.assertDoesNotThrow( + () -> GsonUtils.GSON.fromJson(legacyJson, Repository.class)); + Assertions.assertFalse(deserialized.hasFileSystemDescriptor(), "the record is kept, not guessed"); + Assertions.assertTrue(deserialized.getUnavailableReason().contains("AbsentDependency"), + deserialized.getUnavailableReason()); + } finally { + StorageAdapter.initPluginManager(null); + } + } + + /** A provider registered under {@code name} whose every call reaches a class it does not carry. */ + private static FileSystemProvider throwingProvider( + String name) { + return new FileSystemProvider() { @Override public String name() { - return "JFS"; + return name; } @Override @@ -745,17 +793,7 @@ public org.apache.doris.extension.spi.Plugin create() { return new org.apache.doris.extension.spi.Plugin() { }; } - }); - StorageAdapter.initPluginManager(manager); - try { - Repository deserialized = Assertions.assertDoesNotThrow( - () -> GsonUtils.GSON.fromJson(legacyJson, Repository.class)); - Assertions.assertFalse(deserialized.hasFileSystemDescriptor(), "the record is kept, not guessed"); - Assertions.assertTrue(deserialized.getUnavailableReason().contains("AbsentDependency"), - deserialized.getUnavailableReason()); - } finally { - StorageAdapter.initPluginManager(null); - } + }; } /** diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/lineage/LineageEventProcessorTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/lineage/LineageEventProcessorTest.java index 2a0191aff257ae..49482995e5e8bf 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/lineage/LineageEventProcessorTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/lineage/LineageEventProcessorTest.java @@ -18,6 +18,7 @@ package org.apache.doris.nereids.lineage; import org.apache.doris.common.Config; +import org.apache.doris.common.jmockit.Deencapsulation; import org.apache.doris.extension.loader.ApiVersionGate; import org.apache.doris.extension.spi.PluginContext; import org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand; @@ -35,6 +36,7 @@ import java.util.Arrays; import java.util.Collections; import java.util.List; +import java.util.Map; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; @@ -74,6 +76,80 @@ public LineagePlugin create() { } } + /** Its factory and create() succeed; initialize() reaches a class the jar lacks. */ + public static class FailingInitializeLineagePluginFactory implements LineagePluginFactory { + static final AtomicInteger CLOSED = new AtomicInteger(); + + @Override + public String name() { + return "failing-initialize-lineage-test"; + } + + @Override + public String description() { + return "initialize() links a class this plugin does not carry"; + } + + @Override + public LineagePlugin create() { + return new LineagePlugin() { + @Override + public String name() { + return "failing-initialize-lineage-test"; + } + + @Override + public void initialize(PluginContext context) { + throw new NoClassDefFoundError("org/example/AbsentHttpClient"); + } + + @Override + public void close() { + CLOSED.incrementAndGet(); + } + + @Override + public boolean eventFilter() { + return false; + } + + @Override + public boolean exec(LineageInfo lineageInfo) { + return false; + } + }; + } + } + + /** + * A plugin admitted by the loader and refused one step later must leave nothing behind: the + * instance whose initialize() failed is closed, its factory is dropped (it would retain the + * classloader), and nothing is active. + */ + @Test + public void testAPluginWhoseInitializeFailsIsRolledBack(@TempDir Path tempDir) throws IOException { + writeLineagePluginJar(tempDir.resolve("lineage").resolve("failing-init").resolve("failing-init.jar"), + FailingInitializeLineagePluginFactory.class); + String savedPluginDir = Config.plugin_dir; + String[] savedActive = Config.activate_lineage_plugin; + Config.plugin_dir = tempDir.toString(); + Config.activate_lineage_plugin = new String[0]; + int closedBefore = FailingInitializeLineagePluginFactory.CLOSED.get(); + try { + LineageEventProcessor processor = new LineageEventProcessor(); + Assertions.assertDoesNotThrow(processor::start); + Assertions.assertFalse(processor.hasActivePlugins()); + Assertions.assertEquals(closedBefore + 1, FailingInitializeLineagePluginFactory.CLOSED.get(), + "the created instance is closed"); + Map factories = Deencapsulation.getField(processor, "factories"); + Assertions.assertFalse(factories.containsKey("failing-initialize-lineage-test"), + "the factory is dropped with its classloader"); + } finally { + Config.plugin_dir = savedPluginDir; + Config.activate_lineage_plugin = savedActive; + } + } + /** * {@code create()} / {@code initialize()} are the first calls into the plugin implementation, one * step after the loader handed back the factory. A {@code NoClassDefFoundError} there is an Error From 05b472131ef889d8cefbaa0846e3c683f8ea99d7 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 16 Sep 2026 07:23:02 +0800 Subject: [PATCH 17/19] [fix](fe) Contain a filesystem that fails to link at its first I/O call inside the repository A provider can bind and construct its filesystem and only link a missing class when exists() or list() first runs - the plugin-absent case one step later than the binding. The repository's I/O paths contained IOException (and ping() a RuntimeException) there but let the LinkageError through: ping() left errMsg stale so SHOW REPOSITORIES showed the repository healthy, and listSnapshots(), upload() and download() threw past their Status instead of reporting it, into the job loop. Each of those boundaries now returns or records the error like an IOException. Test: a filesystem whose exists() and list() throw NoClassDefFoundError makes ping() false with the reason in errMsg and listSnapshots() an error Status, without a throw. Co-Authored-By: Claude Opus 5 --- .../org/apache/doris/backup/Repository.java | 18 +++++++------ .../apache/doris/backup/RepositoryTest.java | 27 +++++++++++++++++++ 2 files changed, 37 insertions(+), 8 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java b/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java index b68792c5884b76..93cc130196e63c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java +++ b/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java @@ -532,7 +532,7 @@ public Status initRepository() { } return Status.OK; } - } catch (IOException e) { + } catch (IOException | LinkageError e) { return new Status(ErrCode.COMMON_ERROR, "Failed to init repository: " + e.getMessage()); } finally { releaseSpiFs(fs); @@ -631,9 +631,11 @@ public boolean ping() { errMsg = TimeUtils.longToTimeString(System.currentTimeMillis()) + ": Invalid path. " + path + ", error: " + e.getMessage(); return false; - } catch (IOException | RuntimeException e) { - // RuntimeException too: ping runs in the RepositoryMgr daemon over every repository, and one - // that throws would leave its errMsg unset and skip the repositories after it. + } catch (IOException | RuntimeException | LinkageError e) { + // RuntimeException and LinkageError too: ping runs in the RepositoryMgr daemon over every + // repository, and one that throws would leave its errMsg unset and skip the repositories + // after it. A LinkageError is a filesystem that bound but links a missing class only at its + // first call - the same plugin-absent case, one step later. errMsg = TimeUtils.longToTimeString(System.currentTimeMillis()) + ": " + e.getMessage(); return false; } @@ -694,7 +696,7 @@ public Status listSnapshots(List snapshotNames) { } snapshotNames.addAll(ssNameSet); return Status.OK; - } catch (IOException e) { + } catch (IOException | LinkageError e) { return new Status(ErrCode.COMMON_ERROR, "Failed to list snapshots: " + e.getMessage()); } finally { releaseSpiFs(fs); @@ -818,7 +820,7 @@ public Status upload(String localFilePath, String remoteFilePath) { fs.delete(Location.of(finalRemotePath), false); spiUploadFile(fs, localFilePath, finalRemotePath); } - } catch (IOException e) { + } catch (IOException | LinkageError e) { return new Status(ErrCode.COMMON_ERROR, "Failed to upload " + localFilePath + ": " + e.getMessage()); } finally { releaseSpiFs(fs); @@ -915,7 +917,7 @@ public Status download(String remoteFilePath, String localFilePath) { return Status.OK; } catch (FileNotFoundException e) { return new Status(ErrCode.NOT_FOUND, "file " + localFilePath + " does not exist"); - } catch (IOException e) { + } catch (IOException | LinkageError e) { return new Status(ErrCode.COMMON_ERROR, "Failed to download file: " + e.getMessage()); } finally { releaseSpiFs(fs); @@ -1088,7 +1090,7 @@ private List getSnapshotInfo(String snapshotName, String timestamp) { info.add(FeConstants.null_string); info.add("ERROR: No info file found"); } - } catch (IOException e) { + } catch (IOException | LinkageError e) { info.add(snapshotName); info.add(FeConstants.null_string); info.add("ERROR: Failed to get info: " + e.getMessage()); diff --git a/fe/fe-core/src/test/java/org/apache/doris/backup/RepositoryTest.java b/fe/fe-core/src/test/java/org/apache/doris/backup/RepositoryTest.java index de9261ab79daa4..44a0a134da893b 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/backup/RepositoryTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/backup/RepositoryTest.java @@ -830,6 +830,33 @@ public void testReplayedAlterRepositoryAppliesToAnUnmigratedCopy() { } } + /** + * A provider can bind and build its filesystem, then link a missing class only at the first I/O + * call. That LinkageError must be contained where an IOException is: ping() records it in errMsg + * and the Status-returning paths return it, instead of throwing past the job loop. + */ + @Test + public void testAFilesystemThatFailsToLinkAtFirstIoIsReportedNotThrown() throws IOException { + Mockito.when(mockFs.exists(Mockito.any(Location.class))) + .thenThrow(new NoClassDefFoundError("org/example/AbsentSdkClass")); + Mockito.when(mockFs.list(Mockito.any(Location.class))) + .thenThrow(new NoClassDefFoundError("org/example/AbsentSdkClass")); + repo = new Repository(10000, "repo", false, location, testProps); + + FeConstants.runningUnitTest = false; + try { + Assertions.assertFalse(Assertions.assertDoesNotThrow(() -> repo.ping())); + } finally { + FeConstants.runningUnitTest = true; + } + Assertions.assertNotNull(repo.getErrorMsg()); + Assertions.assertTrue(repo.getErrorMsg().contains("AbsentSdkClass"), repo.getErrorMsg()); + + Status listing = Assertions.assertDoesNotThrow(() -> repo.listSnapshots(Lists.newArrayList())); + Assertions.assertFalse(listing.ok()); + Assertions.assertTrue(listing.getErrMsg().contains("AbsentSdkClass"), listing.getErrMsg()); + } + /** The broker fallback is taken on the one positive signal only: a registered broker name. */ @Test public void testLegacyRecordNamingAnUnregisteredBrokerIsNotMigratedToBroker() { From b273abf0be9bc1088ba4336b4234384846665a57 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 16 Sep 2026 09:58:50 +0800 Subject: [PATCH 18/19] [fix](fe) Contain ServiceConfigurationError at every plugin boundary, and walk a plugin's engine-name set inside the guard ServiceConfigurationError is an Error but not a LinkageError: a plugin whose static initializer, name(), probe or bind runs a nested ServiceLoader lookup over one of its own malformed service files - its driver discovery - threw it through every guard this branch added, which caught RuntimeException and LinkageError only. In the loader that aborted the rest of the family load and leaked the classloader; in StorageDesc.gsonPostProcess it took image load or a follower's journal replay down; in the family managers, the repository binds and I/O boundaries and the job and ping loops it escaped the same way. Every such guard now names ServiceConfigurationError too. The connector manager's first-call guard covered calling acceptedCreateTableEngineNames() but not walking its answer: the plugin-owned set was first traversed in the problem checks and again while claiming, both outside the guard, so a lazy set whose iterator links a missing class threw out of registerDiscovered() - after the type name had been claimed on the second pass. The set is copied into a host-owned snapshot inside the guard, and the checks and claims walk the copy. Tests: a plugin whose initializer runs a nested lookup over a service file naming a class that does not exist is one LoadFailure whose cause is the ServiceConfigurationError; a provider whose probe throws it leaves a persisted load job's descriptor unbound at replay instead of failing the replay; a connector provider whose engine-name set throws when walked is refused with no type claimed. Co-Authored-By: Claude Opus 5 --- .../apache/doris/analysis/StorageDesc.java | 3 +- .../apache/doris/backup/BackupHandler.java | 5 +- .../org/apache/doris/backup/Repository.java | 21 +++--- .../apache/doris/backup/RepositoryMgr.java | 3 +- .../connector/ConnectorPluginManager.java | 7 +- .../doris/fs/FileSystemPluginManager.java | 3 +- .../lineage/LineageEventProcessor.java | 4 +- .../analysis/StorageDescPersistTest.java | 70 +++++++++++++++++++ .../connector/ConnectorPluginManagerTest.java | 32 +++++++++ .../loader/DirectoryPluginRuntimeManager.java | 9 +-- ...ectoryPluginRuntimeManagerLinkageTest.java | 30 ++++++++ ...lformedServiceLookupTestPluginFactory.java | 52 ++++++++++++++ .../loader/testplugins/NestedService.java | 22 ++++++ 13 files changed, 238 insertions(+), 23 deletions(-) create mode 100644 fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/MalformedServiceLookupTestPluginFactory.java create mode 100644 fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/NestedService.java diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/StorageDesc.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/StorageDesc.java index 683fd3a4a675f3..3a4b493b7ccfee 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/analysis/StorageDesc.java +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/StorageDesc.java @@ -27,6 +27,7 @@ import java.io.IOException; import java.util.HashMap; import java.util.Map; +import java.util.ServiceConfigurationError; /** @@ -122,7 +123,7 @@ public void gsonPostProcess() throws IOException { // initStorageAdapter() lazily, so the job binds at use and fails there with a Status. try { initStorageAdapter(); - } catch (RuntimeException | LinkageError e) { + } catch (RuntimeException | LinkageError | ServiceConfigurationError e) { LOG.warn("Storage descriptor (name={}, type={}) could not bind its filesystem provider at" + " load; the binding is retried at use: {}", name, storageType, e.getMessage()); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java b/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java index e2c21978f3f1e4..f2696950a6d366 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java +++ b/fe/fe-core/src/main/java/org/apache/doris/backup/BackupHandler.java @@ -72,6 +72,7 @@ import java.util.LinkedList; import java.util.List; import java.util.Map; +import java.util.ServiceConfigurationError; import java.util.Set; import java.util.concurrent.TimeUnit; import java.util.concurrent.locks.ReadWriteLock; @@ -195,7 +196,7 @@ protected void runAfterCatalogReady() { job.setEnv(env); try { job.run(); - } catch (Exception | LinkageError e) { + } catch (Exception | LinkageError | ServiceConfigurationError e) { // One job's throw must not skip every job after it in this cycle, nor go unrecorded: // a job that keeps throwing here is retried each cycle until its timeout, so the log // is the only place its cause shows up. @@ -257,7 +258,7 @@ public void alterRepository(String repoName, Map newProps) StorageAdapter mergedStorage; try { mergedStorage = StorageAdapter.of(mergedProps); - } catch (RuntimeException | LinkageError e) { + } catch (RuntimeException | LinkageError | ServiceConfigurationError e) { throw new DdlException("Failed to alter repository " + repoName + ": the merged properties do not bind a filesystem provider: " + e.getMessage()); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java b/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java index 93cc130196e63c..c96a063ae4f166 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java +++ b/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java @@ -72,6 +72,7 @@ import java.util.LinkedHashSet; import java.util.List; import java.util.Map; +import java.util.ServiceConfigurationError; import java.util.Set; import java.util.UUID; @@ -277,7 +278,7 @@ public void gsonPostProcess() { try { StorageAdapter storageAdapter = StorageAdapter.of(props); fileSystemDescriptor = FileSystemDescriptor.fromStorageAdapter(storageAdapter, ""); - } catch (RuntimeException | LinkageError e) { + } catch (RuntimeException | LinkageError | ServiceConfigurationError e) { // A typed legacy record names its storage type ("S3", "HDFS", "AZURE", ...); a broker // record names its broker. The descriptor carries an explicit type and is persisted by // the next checkpoint, so a wrong guess here is permanent: a typed record must never @@ -309,7 +310,7 @@ public void gsonPostProcess() { try { StorageAdapter brokerAdapter = StorageAdapter.ofBroker(fsName, props); fileSystemDescriptor = FileSystemDescriptor.fromStorageAdapter(brokerAdapter, fsName); - } catch (RuntimeException | LinkageError e2) { + } catch (RuntimeException | LinkageError | ServiceConfigurationError e2) { unavailableReason = "legacy record was not migrated: " + e2.getMessage(); errMsg = unavailableReason; LOG.error("Repository '{}': {}", name, errMsg); @@ -330,7 +331,7 @@ public void gsonPostProcess() { StorageAdapter storageAdapter = StorageAdapter.of(fsProps); this.normalizedLocation = storageAdapter.validateAndNormalizeUri(location); this.spiFs = FileSystemFactory.getFileSystem(storageAdapter); - } catch (IOException | RuntimeException | LinkageError e) { + } catch (IOException | RuntimeException | LinkageError | ServiceConfigurationError e) { // Same reasoning as above: the plugin may be absent (the FE serves without it) or // half-installed. The repository stays in metadata and reports this on every use. unavailableReason = "filesystem provider did not bind at load: " + e.getMessage(); @@ -532,7 +533,7 @@ public Status initRepository() { } return Status.OK; } - } catch (IOException | LinkageError e) { + } catch (IOException | LinkageError | ServiceConfigurationError e) { return new Status(ErrCode.COMMON_ERROR, "Failed to init repository: " + e.getMessage()); } finally { releaseSpiFs(fs); @@ -631,7 +632,7 @@ public boolean ping() { errMsg = TimeUtils.longToTimeString(System.currentTimeMillis()) + ": Invalid path. " + path + ", error: " + e.getMessage(); return false; - } catch (IOException | RuntimeException | LinkageError e) { + } catch (IOException | RuntimeException | LinkageError | ServiceConfigurationError e) { // RuntimeException and LinkageError too: ping runs in the RepositoryMgr daemon over every // repository, and one that throws would leave its errMsg unset and skip the repositories // after it. A LinkageError is a filesystem that bound but links a missing class only at its @@ -696,7 +697,7 @@ public Status listSnapshots(List snapshotNames) { } snapshotNames.addAll(ssNameSet); return Status.OK; - } catch (IOException | LinkageError e) { + } catch (IOException | LinkageError | ServiceConfigurationError e) { return new Status(ErrCode.COMMON_ERROR, "Failed to list snapshots: " + e.getMessage()); } finally { releaseSpiFs(fs); @@ -820,7 +821,7 @@ public Status upload(String localFilePath, String remoteFilePath) { fs.delete(Location.of(finalRemotePath), false); spiUploadFile(fs, localFilePath, finalRemotePath); } - } catch (IOException | LinkageError e) { + } catch (IOException | LinkageError | ServiceConfigurationError e) { return new Status(ErrCode.COMMON_ERROR, "Failed to upload " + localFilePath + ": " + e.getMessage()); } finally { releaseSpiFs(fs); @@ -917,7 +918,7 @@ public Status download(String remoteFilePath, String localFilePath) { return Status.OK; } catch (FileNotFoundException e) { return new Status(ErrCode.NOT_FOUND, "file " + localFilePath + " does not exist"); - } catch (IOException | LinkageError e) { + } catch (IOException | LinkageError | ServiceConfigurationError e) { return new Status(ErrCode.COMMON_ERROR, "Failed to download file: " + e.getMessage()); } finally { releaseSpiFs(fs); @@ -948,7 +949,7 @@ public Status getBrokerAddress(Long beId, Env env, List brokerAddrs) { // task construction that follows. try { fileSystemDescriptor.getBackendConfigProperties(); - } catch (RuntimeException | LinkageError e) { + } catch (RuntimeException | LinkageError | ServiceConfigurationError e) { return new Status(ErrCode.COMMON_ERROR, "Repository '" + name + "' is not available: " + e.getMessage()); } @@ -1090,7 +1091,7 @@ private List getSnapshotInfo(String snapshotName, String timestamp) { info.add(FeConstants.null_string); info.add("ERROR: No info file found"); } - } catch (IOException | LinkageError e) { + } catch (IOException | LinkageError | ServiceConfigurationError e) { info.add(snapshotName); info.add(FeConstants.null_string); info.add("ERROR: Failed to get info: " + e.getMessage()); diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/RepositoryMgr.java b/fe/fe-core/src/main/java/org/apache/doris/backup/RepositoryMgr.java index eb157447bd7d88..f8a050702f5acb 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/backup/RepositoryMgr.java +++ b/fe/fe-core/src/main/java/org/apache/doris/backup/RepositoryMgr.java @@ -36,6 +36,7 @@ import java.io.DataOutput; import java.io.IOException; import java.util.List; +import java.util.ServiceConfigurationError; import java.util.concurrent.ConcurrentMap; import java.util.concurrent.locks.ReentrantLock; @@ -63,7 +64,7 @@ protected void runOneCycle() { if (!repo.ping()) { LOG.warn("Failed to connect repository {}. msg: {}", repo.getName(), repo.getErrorMsg()); } - } catch (Exception | LinkageError e) { + } catch (Exception | LinkageError | ServiceConfigurationError e) { // ping() reports through errMsg and never throws by contract; if it ever does, the // repositories after this one in the cycle must still be checked. LOG.warn("Failed to ping repository {}", repo.getName(), e); diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java index 511c5d3ab24106..31f024db726578 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java @@ -243,8 +243,11 @@ boolean registerDiscovered(ConnectorProvider provider, boolean failFast) { Set engineNames; try { type = provider.getType(); - engineNames = provider.acceptedCreateTableEngineNames(); - } catch (RuntimeException | LinkageError e) { + // A host-owned copy: the plugin's set is traversed here, under the guard, not later while + // the problem checks and the claims walk it (a lazy set may link a missing class then). + Set answered = provider.acceptedCreateTableEngineNames(); + engineNames = answered == null ? null : new HashSet<>(answered); + } catch (RuntimeException | LinkageError | ServiceConfigurationError e) { // The first calls into plugin code after loading: a getType() that touches a class the // plugin neither bundles nor inherits arrives here as a LinkageError the loader never saw. // For a directory plugin that is one plugin's problem, not the FE's - same guard as diff --git a/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java b/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java index db404f7a8061fb..8fe181448e7041 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java @@ -43,6 +43,7 @@ import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.ServiceConfigurationError; import java.util.ServiceLoader; import java.util.Set; import java.util.concurrent.CopyOnWriteArrayList; @@ -182,7 +183,7 @@ public void loadPlugins(List pluginRoots) { Set sensitiveKeys; try { sensitiveKeys = provider.sensitivePropertyKeys(); - } catch (RuntimeException | LinkageError e) { + } catch (RuntimeException | LinkageError | ServiceConfigurationError e) { runtimeManager.discard(handle.getPluginName()); LOG.warn("Skip filesystem plugin '{}' from {}: sensitivePropertyKeys() failed", handle.getPluginName(), handle.getPluginDir(), e); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/lineage/LineageEventProcessor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/lineage/LineageEventProcessor.java index d57c9d80dac8a9..19171177256284 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/lineage/LineageEventProcessor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/lineage/LineageEventProcessor.java @@ -221,7 +221,7 @@ pluginRoots, getClass().getClassLoader(), plugins.add(plugin); LOG.info("Loaded lineage plugin: {}, pluginPath={}", pluginName, props.get("plugin.path")); } - } catch (Exception | LinkageError e) { + } catch (Exception | LinkageError | ServiceConfigurationError e) { // create() and initialize() are the first calls into the plugin implementation (the // factory only named it): a dependency it lacks arrives here as NoClassDefFoundError, // which must cost this plugin alone, not FE startup. Everything published for it goes @@ -244,7 +244,7 @@ private static void closeQuietly(LineagePlugin plugin, String pluginName) { } try { plugin.close(); - } catch (Exception | LinkageError e) { + } catch (Exception | LinkageError | ServiceConfigurationError e) { LOG.warn("Failed to close lineage plugin {} after its initialization failed", pluginName, e); } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java index 73350836a2fdd5..79668bb847d090 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java @@ -18,7 +18,12 @@ package org.apache.doris.analysis; import org.apache.doris.datasource.storage.StorageAdapter; +import org.apache.doris.extension.spi.Plugin; +import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.filesystem.properties.FileSystemProperties; +import org.apache.doris.filesystem.spi.FileSystemProvider; import org.apache.doris.foundation.property.StoragePropertiesException; +import org.apache.doris.fs.FileSystemPluginManager; import org.apache.doris.fs.TestFileSystemPluginManagers; import org.apache.doris.load.EtlJobType; import org.apache.doris.load.loadv2.BrokerLoadJob; @@ -30,6 +35,7 @@ import java.lang.reflect.Field; import java.util.Map; +import java.util.ServiceConfigurationError; public class StorageDescPersistTest { @@ -118,6 +124,70 @@ public void testBrokerLoadJobRoundTripSurvivesAnAbsentProvider() throws Exceptio } } + /** + * Post-admission: a loaded provider's probe or bind may itself run a nested ServiceLoader lookup and + * fail with ServiceConfigurationError, which is an Error but no LinkageError. At replay that has to + * be contained like the absent-provider case, or the journal kills the follower. + */ + @Test + public void testBrokerLoadJobRoundTripSurvivesAProviderThrowingServiceConfigurationError() throws Exception { + Map properties = Maps.newHashMap(); + properties.put("s3.endpoint", "s3.us-east-1.amazonaws.com"); + properties.put("s3.region", "us-east-1"); + properties.put("s3.access_key", "ak"); + properties.put("s3.secret_key", "sk"); + BrokerDesc brokerDesc = new BrokerDesc("S3", StorageBackend.StorageType.S3, properties); + BrokerLoadJob job = new BrokerLoadJob(); + setField(BrokerLoadJob.class.getSuperclass(), job, "brokerDesc", brokerDesc); + String json = GsonUtils.GSON.toJson(job); + + // Registered under the name the registry consults first, so it is the first one bindPrimary probes. + FileSystemPluginManager manager = TestFileSystemPluginManagers.withoutProviders(); + manager.registerProvider(new FileSystemProvider() { + @Override + public String name() { + return "JFS"; + } + + @Override + public String description() { + return "a provider whose probe runs a malformed nested service lookup"; + } + + @Override + public boolean supports(Map props) { + throw new ServiceConfigurationError("org.example.Driver: Provider org.example.Missing not found"); + } + + @Override + public boolean supportsExplicit(Map props) { + throw new ServiceConfigurationError("org.example.Driver: Provider org.example.Missing not found"); + } + + @Override + public FileSystem create(Map props) { + throw new ServiceConfigurationError("org.example.Driver: Provider org.example.Missing not found"); + } + + @Override + public Plugin create() { + return new Plugin() { + }; + } + }); + StorageAdapter.initPluginManager(manager); + try { + BrokerLoadJob restored = Assertions.assertDoesNotThrow( + () -> GsonUtils.GSON.fromJson(json, BrokerLoadJob.class)); + BrokerDesc restoredDesc = (BrokerDesc) getField(BrokerLoadJob.class.getSuperclass(), restored, "brokerDesc"); + Assertions.assertNull(getField(StorageDesc.class, restoredDesc, "storageAdapter")); + Assertions.assertThrows(ServiceConfigurationError.class, restoredDesc::getStorageAdapter, + "the first use reports the provider's own failure"); + } finally { + StorageAdapter.initPluginManager(null); + } + } + @Test public void testBrokerDescRoundTripSurvivesAnAbsentBrokerProvider() throws Exception { Map properties = Maps.newHashMap(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorPluginManagerTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorPluginManagerTest.java index 23f55ff227c557..67e463799bc0bb 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorPluginManagerTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorPluginManagerTest.java @@ -379,6 +379,38 @@ public Connector create(Map properties, ConnectorContext context Assertions.assertFalse(Assertions.assertDoesNotThrow(() -> manager.registerDiscovered(nullEngines, false)), "a provider answering null engine names must be refused, not thrown"); Assertions.assertTrue(manager.getRegisteredTypes().isEmpty(), "nothing may have been registered"); + + // The set is the plugin's: a lazy one may link a missing class only when it is first walked, + // which has to happen inside the guard rather than in the checks and claims that follow. + ConnectorProvider lazyEngines = new ConnectorProvider() { + @Override + public String getType() { + return "lazy_engines"; + } + + @Override + public Set acceptedCreateTableEngineNames() { + return new java.util.AbstractSet() { + @Override + public java.util.Iterator iterator() { + throw new NoClassDefFoundError("org/example/OptionalEngineCatalog"); + } + + @Override + public int size() { + return 1; + } + }; + } + + @Override + public Connector create(Map properties, ConnectorContext context) { + return null; + } + }; + Assertions.assertFalse(Assertions.assertDoesNotThrow(() -> manager.registerDiscovered(lazyEngines, false)), + "a set that fails when walked must be refused, not thrown"); + Assertions.assertTrue(manager.getRegisteredTypes().isEmpty(), "no type name may be left claimed"); } private static ConnectorProvider createProviderWithEngines(String type, String... engineNames) { diff --git a/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManager.java b/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManager.java index 4ed4d023970144..df7cfd8df2fb43 100644 --- a/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManager.java +++ b/fe/fe-extension-loader/src/main/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManager.java @@ -35,6 +35,7 @@ import java.util.List; import java.util.Objects; import java.util.Optional; +import java.util.ServiceConfigurationError; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentMap; @@ -281,7 +282,7 @@ private PluginHandle loadFromPluginDir(Path pluginDir, ClassLoader parent, Cl Class discoveredClass; try { discoveredClass = classLoader.loadClass(factoryClassName); - } catch (ReflectiveOperationException | RuntimeException | LinkageError e) { + } catch (ReflectiveOperationException | RuntimeException | LinkageError | ServiceConfigurationError e) { // LinkageError included: defining the factory class resolves its supertypes, so a // dependency the plugin neither bundles nor inherits from its parent surfaces here as // NoClassDefFoundError - an Error, not a ReflectiveOperationException. Left uncaught it @@ -316,7 +317,7 @@ private PluginHandle loadFromPluginDir(Path pluginDir, ClassLoader parent, Cl @SuppressWarnings("unchecked") Class factoryClass = (Class) discoveredClass.asSubclass(factoryType); factory = factoryClass.getDeclaredConstructor().newInstance(); - } catch (ReflectiveOperationException | RuntimeException | LinkageError e) { + } catch (ReflectiveOperationException | RuntimeException | LinkageError | ServiceConfigurationError e) { // newInstance() is where the factory class is first initialized. A static initializer // that fails arrives either as the Error itself - NoClassDefFoundError for a missing // dependency - or, for a non-Error, wrapped in ExceptionInInitializerError; both are @@ -342,7 +343,7 @@ private PluginHandle loadFromPluginDir(Path pluginDir, ClassLoader parent, Cl String pluginName; try { pluginName = factory.name(); - } catch (RuntimeException | LinkageError e) { + } catch (RuntimeException | LinkageError | ServiceConfigurationError e) { closeClassLoader(classLoader); throw new PluginLoadException( normalizedDir, @@ -366,7 +367,7 @@ private PluginHandle loadFromPluginDir(Path pluginDir, ClassLoader parent, Cl String description; try { description = factory.description(); - } catch (RuntimeException | LinkageError e) { + } catch (RuntimeException | LinkageError | ServiceConfigurationError e) { closeClassLoader(classLoader); throw new PluginLoadException( normalizedDir, diff --git a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerLinkageTest.java b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerLinkageTest.java index 28a1e9688817a1..75921b7ae73e6a 100644 --- a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerLinkageTest.java +++ b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/DirectoryPluginRuntimeManagerLinkageTest.java @@ -20,7 +20,9 @@ import org.apache.doris.extension.loader.testplugins.AbsentDependencyProbe; import org.apache.doris.extension.loader.testplugins.AbsentStaticInitTestPluginFactory; import org.apache.doris.extension.loader.testplugins.AbsentSuperclassTestPluginFactory; +import org.apache.doris.extension.loader.testplugins.MalformedServiceLookupTestPluginFactory; import org.apache.doris.extension.loader.testplugins.MetadataTestPluginFactory; +import org.apache.doris.extension.loader.testplugins.NestedService; import org.apache.doris.extension.loader.testplugins.NotAFactory; import org.apache.doris.extension.loader.testplugins.ThrowingStaticInitTestPluginFactory; import org.apache.doris.extension.spi.PluginFactory; @@ -39,6 +41,7 @@ import java.util.Collections; import java.util.LinkedHashMap; import java.util.Map; +import java.util.ServiceConfigurationError; import java.util.jar.Attributes; import java.util.jar.JarEntry; import java.util.jar.JarOutputStream; @@ -160,6 +163,33 @@ void testAServiceFileNamingANonFactoryIsAFailureNotAThrow() throws IOException { + failure.getMessage()); } + @Test + void testAMalformedNestedServiceLookupIsAFailureNotAThrow() throws IOException { + // A plugin's own driver discovery: its service file for NestedService names a class that does + // not exist, so the lookup in its static initializer throws ServiceConfigurationError - an + // Error that is neither a LinkageError nor a RuntimeException, and that used to escape both + // catch sites, abort the rest of the family and leak the classloader. + Map entries = new LinkedHashMap<>(); + entries.put(MalformedServiceLookupTestPluginFactory.class.getName().replace('.', '/') + ".class", + classBytes(MalformedServiceLookupTestPluginFactory.class)); + entries.put("META-INF/services/" + NestedService.class.getName(), + "org.example.MissingNestedProvider\n".getBytes(StandardCharsets.UTF_8)); + Path root = tempDir.resolve("plugins"); + writeJar(root.resolve("malformed-service").resolve("malformed-service.jar"), + MalformedServiceLookupTestPluginFactory.class.getName(), entries); + + LoadReport report = load(root); + Assertions.assertTrue(report.getSuccesses().isEmpty(), "the plugin cannot have loaded"); + Assertions.assertEquals(1, report.getFailures().size()); + LoadFailure failure = report.getFailures().get(0); + Assertions.assertEquals(LoadFailure.STAGE_INSTANTIATE, failure.getStage()); + Assertions.assertInstanceOf(ServiceConfigurationError.class, failure.getCause(), + () -> "the nested lookup's own error is the cause, got " + failure.getCause()); + Assertions.assertTrue(failure.getMessage().contains("; caused by java.util.ServiceConfigurationError"), + failure.getMessage()); + Assertions.assertTrue(failure.getMessage().contains("MissingNestedProvider"), failure.getMessage()); + } + @Test void testACyclicCauseChainIsWalkedOnce() { // Legal Java: a's cause is still the sentinel when initCause runs, so the chain is a -> b -> a. diff --git a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/MalformedServiceLookupTestPluginFactory.java b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/MalformedServiceLookupTestPluginFactory.java new file mode 100644 index 00000000000000..bfe441e3b820c6 --- /dev/null +++ b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/MalformedServiceLookupTestPluginFactory.java @@ -0,0 +1,52 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.extension.loader.testplugins; + +import org.apache.doris.extension.spi.Plugin; +import org.apache.doris.extension.spi.PluginFactory; + +import java.util.ServiceLoader; + +/** + * This class defines cleanly and its static initializer runs a nested {@link ServiceLoader} lookup + * against its own classloader. The test jar registers a provider class that does not exist, so the + * lookup fails with {@link java.util.ServiceConfigurationError} - an {@link Error} that is neither a + * {@link LinkageError} nor a {@link RuntimeException}, and the shape a plugin's own driver discovery + * produces when one of its service files is broken. + */ +public class MalformedServiceLookupTestPluginFactory implements PluginFactory { + + private static final Object FIRST_PROVIDER = ServiceLoader.load(NestedService.class, + MalformedServiceLookupTestPluginFactory.class.getClassLoader()).iterator().next(); + + @Override + public String name() { + return "malformed-service-lookup-test"; + } + + @Override + public String description() { + return "Plugin whose static initializer runs a malformed nested service lookup: " + FIRST_PROVIDER; + } + + @Override + public Plugin create() { + return new Plugin() { + }; + } +} diff --git a/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/NestedService.java b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/NestedService.java new file mode 100644 index 00000000000000..d4ccfdff5d1c7d --- /dev/null +++ b/fe/fe-extension-loader/src/test/java/org/apache/doris/extension/loader/testplugins/NestedService.java @@ -0,0 +1,22 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.extension.loader.testplugins; + +/** A service a plugin looks up through its own ServiceLoader; the plugin's registration for it is malformed. */ +public interface NestedService { +} From 7467c22861a7bd6a9330bd33c9597acfe05fb3d9 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 16 Sep 2026 12:34:23 +0800 Subject: [PATCH 19/19] [fix](fe) Decide a legacy repository's storage identity by its name, and walk a provider's sensitive keys before publishing it The legacy migration routed the record's properties first and consulted its name only when no provider claimed them. A WITH BROKER repository commonly stores fs.defaultFS and hadoop.username, which the HDFS provider claims, so such a record was migrated as HDFS, the next checkpoint made that permanent, and every later backup bypassed the broker it was created on. The record's name is the one identity it carries - a storage type for a typed record, the broker for a broker record - and now decides: a storage type's name binds that type (or is kept with the reason, which for a broker that happens to be named after a storage type says to DROP and re-CREATE); a registered broker's name is a broker record, whatever its properties route to; only a name that is neither - one from before the type names, or a broker since dropped - is routed as before. FileSystemPluginManager guarded the call to sensitivePropertyKeys() but not the walk over its answer, which happened in the masking set's addAll after the provider was already routable (and, for a built-in, after its inventory row): a plugin-owned lazy set failing when iterated, or a null alias failing the case-insensitive comparator, aborted the family load with the provider active and unmasked. The set is copied and checked inside the guard, before anything is published, on all three admission paths. Tests: a legacy broker record with HDFS-shaped properties stays a broker repository through a checkpoint round trip; a provider whose sensitive-key set fails when walked, or carries a null alias, is refused before it is published. Co-Authored-By: Claude Opus 5 --- .../org/apache/doris/backup/Repository.java | 73 ++++++++++--------- .../doris/fs/FileSystemPluginManager.java | 34 ++++++++- .../apache/doris/backup/RepositoryTest.java | 35 +++++++++ .../doris/fs/FileSystemPluginManagerTest.java | 57 +++++++++++++++ 4 files changed, 160 insertions(+), 39 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java b/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java index c96a063ae4f166..64f6240908dd22 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java +++ b/fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java @@ -272,50 +272,53 @@ public void gsonPostProcess() { LOG.info("Repository '{}': migrating legacy 'fs' field to 'fs_descriptor'", name); Map props = legacyFs.properties != null ? legacyFs.properties : new HashMap<>(); String fsName = legacyFs.name != null ? legacyFs.name : ""; - // Both binds run plugin code (bindPrimary probes every loaded provider) at image load and - // edit-log replay: LinkageError included, so a half-installed plugin costs this - // repository, not the FE - the same catch CatalogFactory uses for catalogs. - try { - StorageAdapter storageAdapter = StorageAdapter.of(props); - fileSystemDescriptor = FileSystemDescriptor.fromStorageAdapter(storageAdapter, ""); - } catch (RuntimeException | LinkageError | ServiceConfigurationError e) { - // A typed legacy record names its storage type ("S3", "HDFS", "AZURE", ...); a broker - // record names its broker. The descriptor carries an explicit type and is persisted by - // the next checkpoint, so a wrong guess here is permanent: a typed record must never - // become BROKER because its plugin failed to load, threw, or rejected the properties. - // So a name that is a shipped provider's is never read as a broker's, whatever brokers - // are registered (a broker may legally be called HDFS): the record is kept, with the - // reason - the provider is absent, or it is loaded and the binding failed. Only a name - // that is not a provider's and is a registered broker's is read as a broker record. - // Anything else is kept as it was, so the migration is retried at the next start, and - // every use reports the reason until then. - if (isStorageTypeName(fsName)) { + // The record's name is the one identity it carries: a typed legacy record names its storage + // type ("S3", "HDFS", "AZURE", ...), a broker record names its broker. The descriptor written + // here carries an explicit type and is persisted by the next checkpoint, so it has to be + // decided by that identity, not by whichever provider happens to claim the properties: a + // WITH BROKER repository commonly stores fs.defaultFS and the HDFS provider would claim it, + // and a typed repository must never become BROKER because its plugin failed to load, threw, + // or rejected the properties. So: a storage type's name binds that type, or is kept with the + // reason; a registered broker's name is a broker record; anything else - a name from before + // the type names, or a broker since dropped - is routed as it always was. A kept record is + // retried at the next start, and every use reports the reason until then. Every bind runs + // plugin code at image load and edit-log replay, so a LinkageError or a plugin's own + // ServiceConfigurationError costs this repository, not the FE - the catch CatalogFactory uses. + if (isStorageTypeName(fsName)) { + try { + StorageAdapter storageAdapter = StorageAdapter.of(props); + fileSystemDescriptor = FileSystemDescriptor.fromStorageAdapter(storageAdapter, ""); + } catch (RuntimeException | LinkageError | ServiceConfigurationError e) { unavailableReason = "legacy record of storage type '" + fsName + "' was not migrated: " + (StorageAdapter.hasProvider(fsName) ? "its filesystem provider is loaded but did not bind the properties (" : "its filesystem provider is not loaded (") - + e.getMessage() + "). Repair the plugin or the properties and restart the FE."; - } else if (!isRegisteredBroker(fsName)) { - unavailableReason = "legacy record was not migrated: no loaded filesystem provider" - + " claims its properties (" + e.getMessage() + ") and '" + fsName - + "' is not a registered broker. After ADD BROKER, restart the FE."; - } - if (unavailableReason != null) { - errMsg = unavailableReason; - LOG.warn("Repository '{}': {}", name, errMsg); - return; + + e.getMessage() + "). Repair the plugin or the properties and restart the FE;" + + " if this repository was created WITH BROKER \"" + fsName + "\", DROP and" + + " re-CREATE it."; } - LOG.warn("Repository '{}': primary storage migration failed ({}), migrating as a broker" - + " repository on broker '{}'", name, e.getMessage(), fsName); + } else if (isRegisteredBroker(fsName)) { try { StorageAdapter brokerAdapter = StorageAdapter.ofBroker(fsName, props); fileSystemDescriptor = FileSystemDescriptor.fromStorageAdapter(brokerAdapter, fsName); - } catch (RuntimeException | LinkageError | ServiceConfigurationError e2) { - unavailableReason = "legacy record was not migrated: " + e2.getMessage(); - errMsg = unavailableReason; - LOG.error("Repository '{}': {}", name, errMsg); - return; + } catch (RuntimeException | LinkageError | ServiceConfigurationError e) { + unavailableReason = "legacy record on broker '" + fsName + "' was not migrated: " + + e.getMessage(); } + } else { + try { + StorageAdapter storageAdapter = StorageAdapter.of(props); + fileSystemDescriptor = FileSystemDescriptor.fromStorageAdapter(storageAdapter, ""); + } catch (RuntimeException | LinkageError | ServiceConfigurationError e) { + unavailableReason = "legacy record was not migrated: no loaded filesystem provider" + + " claims its properties (" + e.getMessage() + ") and '" + fsName + + "' is not a registered broker. After ADD BROKER, restart the FE."; + } + } + if (unavailableReason != null) { + errMsg = unavailableReason; + LOG.warn("Repository '{}': {}", name, errMsg); + return; } fsProps = fileSystemDescriptor.getProperties(); } else { diff --git a/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java b/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java index 8fe181448e7041..71b8afcd277c38 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java @@ -41,6 +41,7 @@ import java.util.Arrays; import java.util.Collections; import java.util.HashMap; +import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.ServiceConfigurationError; @@ -117,8 +118,10 @@ public void loadBuiltins() { // Snapshot all self-reported metadata (sensitive keys included) // before mutating any store, so one throwing implementation is // rejected cleanly instead of aborting startup or leaving an - // inventory row for a provider that never became active. - Set sensitiveKeys = p.sensitivePropertyKeys(); + // inventory row for a provider that never became active. The + // snapshot walks the provider's set here, so a lazy set that fails + // when walked fails before anything is published. + Set sensitiveKeys = snapshotSensitiveKeys(p); PluginRegistry.getInstance().registerBuiltin(PLUGIN_FAMILY, p); DatasourcePrintableMap.registerSensitiveKeys(sensitiveKeys); providers.add(p); @@ -182,7 +185,7 @@ public void loadPlugins(List pluginRoots) { // without its inventory row (or vice versa). Set sensitiveKeys; try { - sensitiveKeys = provider.sensitivePropertyKeys(); + sensitiveKeys = snapshotSensitiveKeys(provider); } catch (RuntimeException | LinkageError | ServiceConfigurationError e) { runtimeManager.discard(handle.getPluginName()); LOG.warn("Skip filesystem plugin '{}' from {}: sensitivePropertyKeys() failed", @@ -224,8 +227,31 @@ public FileSystem createFileSystem(Map properties) throws IOExce /** Registers a provider at highest priority. For testing overrides. */ public void registerProvider(FileSystemProvider provider) { + Set sensitiveKeys = snapshotSensitiveKeys(provider); providers.add(0, provider); - DatasourcePrintableMap.registerSensitiveKeys(provider.sensitivePropertyKeys()); + DatasourcePrintableMap.registerSensitiveKeys(sensitiveKeys); + } + + /** + * A host-owned copy of the provider's sensitive aliases, walked and checked here - inside the + * caller's guard and before anything is published - because the set is the plugin's: a lazy one + * may link a missing class only when iterated, and a null element would fail the masking set's + * case-insensitive comparator later, with the provider already routable. + */ + private static Set snapshotSensitiveKeys(FileSystemProvider provider) { + Set answered = provider.sensitivePropertyKeys(); + if (answered == null) { + return Collections.emptySet(); + } + Set copy = new HashSet<>(); + for (String key : answered) { + if (key == null) { + throw new IllegalArgumentException("sensitivePropertyKeys() of provider '" + provider.name() + + "' contains a null alias"); + } + copy.add(key); + } + return copy; } /** Returns an unmodifiable view of the loaded providers, in registration order. */ diff --git a/fe/fe-core/src/test/java/org/apache/doris/backup/RepositoryTest.java b/fe/fe-core/src/test/java/org/apache/doris/backup/RepositoryTest.java index 44a0a134da893b..5c0b6a0ba0ec2f 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/backup/RepositoryTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/backup/RepositoryTest.java @@ -577,6 +577,41 @@ public void testLegacyRecordIsKeptUnchangedWhileItsProviderIsAbsent() { } } + /** + * A WITH BROKER repository commonly stores the properties the HDFS provider would claim + * (fs.defaultFS, hadoop.username). Its identity is its broker name, not whichever provider claims + * the map: routing first would persist it as HDFS and every later backup would bypass the broker. + */ + @Test + public void testLegacyBrokerRecordKeepsItsBrokerIdentityWhateverItsPropertiesRouteTo() { + String legacyJson = "{" + + "\"id\":36000," + + "\"n\":\"brokerRepoHdfsProps\"," + + "\"iro\":false," + + "\"lo\":\"hdfs://ns/backup\"," + + "\"ct\":-1," + + "\"fs\":{\"n\":\"broker_0\",\"prop\":{\"fs.defaultFS\":\"hdfs://ns\"," + + "\"hadoop.username\":\"hadoop\",\"hdfs.authentication.type\":\"simple\"}}" + + "}"; + Mockito.when(mockedBrokerMgr.containsBroker("broker_0")).thenReturn(true); + + Repository deserialized = GsonUtils.GSON.fromJson(legacyJson, Repository.class); + + Assertions.assertEquals(FsStorageType.BROKER, deserialized.getFileSystemDescriptor().getStorageType(), + "the registered broker name decides, not the HDFS provider that claims the properties"); + Assertions.assertEquals("broker_0", deserialized.getFileSystemDescriptor().getName()); + Assertions.assertEquals("hdfs://ns", deserialized.getFileSystemDescriptor().getProperties().get("fs.defaultFS"), + "the broker's pass-through properties are kept"); + Assertions.assertNull(deserialized.getUnavailableReason()); + + // What the checkpoint writes reloads as the same broker repository. + String checkpointed = GsonUtils.GSON.toJson(deserialized); + Assertions.assertTrue(checkpointed.contains("\"fs_type\":\"BROKER\""), checkpointed); + Repository reloaded = GsonUtils.GSON.fromJson(checkpointed, Repository.class); + Assertions.assertEquals(FsStorageType.BROKER, reloaded.getFileSystemDescriptor().getStorageType()); + Assertions.assertEquals("broker_0", reloaded.getFileSystemDescriptor().getName()); + } + /** * A broker may legally be named like a storage type. With that type's plugin absent, the typed * record still names an absent provider, and that - not the broker registry - explains why nothing diff --git a/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java b/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java index 29feec06d54b2e..2f10280d8c58ee 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java @@ -59,6 +59,63 @@ public Set sensitivePropertyKeys() { DatasourcePrintableMap.SENSITIVE_KEY.contains("PLUGIN_MANAGER_TEST_SECRET_ALIAS")); } + /** + * The sensitive-key set is the plugin's: it is walked and checked before the provider is published, + * so a lazy set that fails when iterated, or one carrying a null alias, refuses the provider instead + * of failing later in the masking set with the provider already routable. + */ + @Test + public void registerProvider_walksTheSensitiveKeySetBeforePublishingTheProvider() { + FileSystemPluginManager manager = new FileSystemPluginManager(); + FileSystemProvider lazy = new FileSystemProvider() { + @Override + public boolean supports(Map properties) { + return false; + } + + @Override + public FileSystem create(Map properties) { + return null; + } + + @Override + public Set sensitivePropertyKeys() { + return new java.util.AbstractSet() { + @Override + public java.util.Iterator iterator() { + throw new NoClassDefFoundError("org/example/OptionalAliasCatalog"); + } + + @Override + public int size() { + return 1; + } + }; + } + }; + Assertions.assertThrows(NoClassDefFoundError.class, () -> manager.registerProvider(lazy)); + Assertions.assertTrue(manager.getProviders().isEmpty(), "a provider whose set fails when walked is not published"); + + FileSystemProvider nullAlias = new FileSystemProvider() { + @Override + public boolean supports(Map properties) { + return false; + } + + @Override + public FileSystem create(Map properties) { + return null; + } + + @Override + public Set sensitivePropertyKeys() { + return Collections.singleton(null); + } + }; + Assertions.assertThrows(IllegalArgumentException.class, () -> manager.registerProvider(nullAlias)); + Assertions.assertTrue(manager.getProviders().isEmpty()); + } + // ---- bindAll (P0-T02 / D-009): raw map -> List ---- @Test