Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@ The existing `driver_url` paths (jdbc / iceberg-jdbc / paimon-jdbc catalogs) rem

When a feature genuinely needs an operator-supplied artifact, read it from a local, operator-controlled directory (the `jdbc_drivers_dir` / plugin-directory pattern), not from a URL. The ADBC catalog's `AdbcDriverPathResolver` is the model: remote schemes rejected outright, a scheme-less value restricted to a bare file name so no path separator can escape the configured directory, and the check placed where the artifact is loaded rather than only at CREATE.

Every consumer of the **jdbc-flavored** `driver_url` (the jdbc / iceberg-jdbc / paimon-jdbc catalogs, which do accept a URL) must route the raw value through `JdbcDriverUrlSecurity.check` (`fe-foundation`, `foundation.security`) from its property holder's statement-time validation — `JdbcCatalogProperties.checkCreateTimeOnlyRules` and the iceberg/paimon JDBC metastore holders' `validate()` — which the engine reaches on both CREATE and ALTER CATALOG and never on replay or a catalog rebuild. That class is the single source of truth for the rule; do not fork or re-derive it per connector, and never call it from a holder's `of()`. Such a property must also get a row in the key table of `PluginDrivenExternalCatalog#driverUrlKeysOf` (one table drives both the gate trigger set and the checked values), which is how the operator's `jdbc_driver_secure_path` / `jdbc_driver_url_white_list` policy reaches the ALTER CATALOG path — ALTER never runs `Connector.preCreateValidation`, where CREATE applies that policy, so a key missing from that table lets an ALTER repoint the jar past the operator's configuration.
Every consumer of the **jdbc-flavored** `driver_url` (the jdbc / iceberg-jdbc / paimon-jdbc catalogs, which do accept a URL) must route the raw value through `JdbcDriverUrlSecurity.check` (`fe-foundation`, `foundation.security`) from its property holder's statement-time validation — `JdbcCatalogProperties.checkCreateTimeOnlyRules` and the iceberg/paimon JDBC metastore holders' `validate()` — which the engine reaches on both CREATE and ALTER CATALOG and never on replay or a catalog rebuild. That class is the single source of truth for the rule; do not fork or re-derive it per connector, and never call it from a holder's `of()`. The connector must also declare those `driver_url` property keys (with their flavor condition) in its `ConnectorProvider.securitySurface()` via `ConnectorSecuritySurface.ofJdbcDriverUrls(...)`: the engine looks that surface up by catalog type (without initializing a connector) and applies the operator's `jdbc_driver_secure_path` / `jdbc_driver_url_white_list` policy from `PluginDrivenExternalCatalog`'s CREATE and ALTER validation, never on replay. ALTER never runs `Connector.preCreateValidation`, where CREATE applies that policy, so a key missing from the surface lets an ALTER repoint the jar past the operator's configuration.

Separately, any new outbound HTTP request the FE or BE issues to a host named by a non-SUPER user is an SSRF surface. Call it out explicitly in the PR description together with the privilege required to reach it.
Separately, any new outbound HTTP request the FE or BE issues to a host named by a non-SUPER user is an SSRF surface. Call it out explicitly in the PR description together with the privilege required to reach it. A catalog connector whose user-supplied property names a host the FE connects to also declares that key in its `ConnectorProvider.securitySurface()` via `ConnectorSecuritySurface.ofSsrfEndpoints(...)`; the engine's CREATE / ALTER CATALOG SSRF gate — active only when `security_checker_class_name` is configured — probes those endpoints through it, so the engine never hard-codes a catalog type's property names.

## Thrift and Protobuf Field Ids

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,18 @@ public void startSSRFChecking(String originUri) throws Exception {
return;
}

@Override
public void startSSRFChecking() throws Exception {
return;
}

@Override
public void stopSSRFChecking() {
return;
}

@Override
public boolean isEnabled() {
return false;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -18,22 +18,47 @@
package org.apache.doris.cloud.security;

import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;

import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Strings;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Socket;

/**
* SecurityChecker is used to check if the url is safe
*/
public abstract class SecurityChecker {
private static class SingletonHolder {
private static final SecurityChecker INSTANCE = Strings.isNullOrEmpty(Config.security_checker_class_name)
? new DummySecurityChecker() : new UrlSecurityChecker();
}
private static final Logger LOG = LogManager.getLogger(SecurityChecker.class);

// Upper bound for one SSRF connectivity probe. The probe only needs the net-hook to observe the
// resolved IP before the TCP handshake completes; a short timeout keeps CREATE/ALTER responsive when
// the endpoint is a public-but-unreachable host (that outcome is ignored, see checkEndpointForSsrf).
private static final int SSRF_PROBE_CONNECT_TIMEOUT_MS = 5000;

private static volatile SecurityChecker instance;

protected SecurityChecker() {}

public static SecurityChecker getInstance() {
return SingletonHolder.INSTANCE;
if (instance == null) {
synchronized (SecurityChecker.class) {
if (instance == null) {
instance = Strings.isNullOrEmpty(Config.security_checker_class_name)
? new DummySecurityChecker() : new UrlSecurityChecker();
}
}
}
return instance;
}

@VisibleForTesting
public static void setInstanceForTest(SecurityChecker checker) {
instance = checker;
}

/**
Expand All @@ -45,12 +70,76 @@ public static SecurityChecker getInstance() {
public abstract String getSafeJdbcUrl(String originJdbcUrl) throws Exception;

/**
* Check if the uri is safe, avoid SSRF attack
* Only handle http:// https://
* Install the net-hook for an http(s) request on the current thread, avoid SSRF attack.
* Only handle http:// https://. Must bracket a real request: startSSRFChecking before issuing it,
* {@link #stopSSRFChecking()} after it, and no further request may be issued once stopped.
* @param originUri
* @throws Exception
*/
public abstract void startSSRFChecking(String originUri) throws Exception;

/**
* Install the net-hook for a raw TCP connection on the current thread, avoid SSRF attack.
* Handles any protocol that connects through a JDK socket (thrift, jdbc, ...), not only http.
* Same bracketing contract as {@link #startSSRFChecking(String)}.
* @throws Exception
*/
public abstract void startSSRFChecking() throws Exception;

public abstract void stopSSRFChecking();

/**
* Whether a real SSRF checker (the third-party SDK) is installed. When false every
* {@link #checkEndpointForSsrf(String, int)} is a no-op, so the default open-source build behaves
* exactly as before.
*/
public abstract boolean isEnabled();

/**
* Point-in-time SSRF gate for a single user-supplied endpoint: brackets a lightweight TCP connect
* with the no-arg net-hook so the SDK resolves the host and rejects an intranet target before the
* handshake completes.
*
* <p>The SDK signals an intranet target by throwing {@code SSRFUnsafeConnectionException} (a
* {@link SecurityException}) from inside the connect; that — and only that — is the rejection. A plain
* connectivity failure (refused / timeout / unknown host) is NOT an SSRF signal and is ignored, so a
* catalog that points at a currently-unreachable but public endpoint can still be created.
*/
public final void checkEndpointForSsrf(String host, int port) throws DdlException {
if (!isEnabled()) {
return;
}
try {
startSSRFChecking();
} catch (Exception e) {
// isEnabled() is true but the SDK net-hook could not be installed: surface the misconfiguration
// rather than silently skipping the security gate.
throw new DdlException("Failed to initialize SSRF checking for endpoint " + host + ":" + port
+ ": " + e.getMessage(), e);
}
try {
doConnect(host, port, SSRF_PROBE_CONNECT_TIMEOUT_MS);
} catch (SecurityException e) {
throw new DdlException("Endpoint " + host + ":" + port
+ " is rejected by the SSRF policy (intranet address not allowed): " + e.getMessage());
} catch (IOException e) {
if (LOG.isDebugEnabled()) {
LOG.debug("SSRF probe to {}:{} did not complete, ignored (not an SSRF rejection): {}",
host, port, e.toString());
}
} finally {
stopSSRFChecking();
}
}

/**
* The connection seam of {@link #checkEndpointForSsrf(String, int)}: a single real TCP connect the
* net-hook can observe. Overridable so unit tests can drive the SSRF / unreachable / success branches
* without the third-party SDK or a live server.
*/
protected void doConnect(String host, int port, int connectTimeoutMs) throws IOException {
try (Socket socket = new Socket()) {
socket.connect(new InetSocketAddress(host, port), connectTimeoutMs);
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,19 @@ public class UrlSecurityChecker extends SecurityChecker {
private static final Logger LOG = LogManager.getLogger(UrlSecurityChecker.class);
private static Method jdbcUrlCheckMethod = null;
private static Method urlSecurityCheckMethod = null;
private static Method tcpSecurityCheckMethod = null;
private static Method urlSecurityStopCheckMethod = null;

static {
try {
Class clazz = Class.forName(Config.security_checker_class_name);
jdbcUrlCheckMethod = clazz.getMethod("filterJdbcConnectionSource", String.class);
urlSecurityCheckMethod = clazz.getMethod("startSSRFNetHookChecking", String.class);
urlSecurityStopCheckMethod = clazz.getMethod("stopSSRFNetHookChecking", String.class);
// No-arg overload: hooks a raw TCP connection (thrift / jdbc / ...), not only http.
tcpSecurityCheckMethod = clazz.getMethod("startSSRFNetHookChecking");
// The SDK's stop is no-arg. Binding it with String.class (then invoking with no argument)
// meant stop never fired and the net-hook leaked onto the pooled thread.
urlSecurityStopCheckMethod = clazz.getMethod("stopSSRFNetHookChecking");
} catch (Exception e) {
LOG.warn("security_checker_class_name:{} exception:", Config.security_checker_class_name, e);
LogUtils.stderr("Failed to find com.aliyun.securitysdk.SecurityUtil's method");
Expand All @@ -63,6 +68,20 @@ public void startSSRFChecking(String originUri) throws Exception {
throw new Exception("SecurityUtil.startSSRFNetHookChecking not found");
}

@Override
public void startSSRFChecking() throws Exception {
if (tcpSecurityCheckMethod != null) {
tcpSecurityCheckMethod.invoke(null);
return;
}
throw new Exception("SecurityUtil.startSSRFNetHookChecking (no-arg) not found");
}

@Override
public boolean isEnabled() {
return true;
}

@Override
public void stopSSRFChecking() {
if (urlSecurityStopCheckMethod != null) {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

package org.apache.doris.cloud.security;

import org.apache.doris.common.DdlException;

import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

/**
* {@link SecurityChecker#checkEndpointForSsrf} is the single primitive the catalog SSRF gate delegates to.
* The real third-party SDK is operator-supplied and absent in CI, so these tests drive its branches through
* the {@code doConnect} seam: the SDK signals an intranet target by throwing {@link SecurityException} from
* inside the connect, a plain connectivity failure is {@link IOException}, and when no checker is installed
* the whole probe is a no-op.
*/
public class SecurityCheckerTest {

private enum ConnectBehavior { SUCCESS, SSRF, IO }

/** A SecurityChecker whose net-hook bracket and connect are recorded, not real. */
private static final class RecordingChecker extends SecurityChecker {
private boolean enabled = true;
private boolean startThrows = false;
private ConnectBehavior connectBehavior = ConnectBehavior.SUCCESS;
private final List<String> events = new ArrayList<>();

@Override
public String getSafeJdbcUrl(String originJdbcUrl) {
return originJdbcUrl;
}

@Override
public void startSSRFChecking(String originUri) throws Exception {
start();
}

@Override
public void startSSRFChecking() throws Exception {
start();
}

private void start() throws Exception {
events.add("start");
if (startThrows) {
throw new Exception("cannot install net-hook");
}
}

@Override
public void stopSSRFChecking() {
events.add("stop");
}

@Override
public boolean isEnabled() {
return enabled;
}

@Override
protected void doConnect(String host, int port, int connectTimeoutMs) throws IOException {
events.add("connect");
switch (connectBehavior) {
case SSRF:
throw new SecurityException("simulated SSRFUnsafeConnectionException for " + host);
case IO:
throw new IOException("connection refused");
default:
break;
}
}
}

@Test
public void disabledCheckerSkipsProbeEntirely() throws DdlException {
RecordingChecker checker = new RecordingChecker();
checker.enabled = false;
// Intranet-looking host, but a disabled checker must not even connect.
checker.checkEndpointForSsrf("10.0.0.1", 9083);
Assertions.assertTrue(checker.events.isEmpty(), "disabled checker must not touch the network");
}

@Test
public void successfulProbeBracketsConnectAndReturns() throws DdlException {
RecordingChecker checker = new RecordingChecker();
checker.connectBehavior = ConnectBehavior.SUCCESS;
checker.checkEndpointForSsrf("public.example.com", 9083);
Assertions.assertEquals(Arrays.asList("start", "connect", "stop"), checker.events);
}

@Test
public void intranetTargetIsRejectedAsDdlException() {
RecordingChecker checker = new RecordingChecker();
checker.connectBehavior = ConnectBehavior.SSRF;
DdlException e = Assertions.assertThrows(DdlException.class,
() -> checker.checkEndpointForSsrf("10.0.0.1", 9083));
Assertions.assertTrue(e.getMessage().contains("SSRF policy"), e.getMessage());
// stop still runs in the finally block even when the connect throws.
Assertions.assertEquals(Arrays.asList("start", "connect", "stop"), checker.events);
}

@Test
public void unreachableEndpointIsNotAnSsrfRejection() throws DdlException {
RecordingChecker checker = new RecordingChecker();
checker.connectBehavior = ConnectBehavior.IO;
// A refused/timed-out connect is a reachability fact, not an SSRF signal: must not throw.
checker.checkEndpointForSsrf("public-but-down.example.com", 9083);
Assertions.assertEquals(Arrays.asList("start", "connect", "stop"), checker.events);
}

@Test
public void failedHookInstallSurfacesAsDdlExceptionWithoutConnecting() {
RecordingChecker checker = new RecordingChecker();
checker.startThrows = true;
DdlException e = Assertions.assertThrows(DdlException.class,
() -> checker.checkEndpointForSsrf("some.host", 9083));
Assertions.assertTrue(e.getMessage().contains("Failed to initialize SSRF checking"), e.getMessage());
// No connect attempt, and nothing to stop since the hook was never installed.
Assertions.assertEquals(Arrays.asList("start"), checker.events);
}

@Test
public void dummyCheckerIsDisabled() {
Assertions.assertFalse(new DummySecurityChecker().isEnabled());
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,13 @@ public final class HiveCatalogProperties {
*/
public static final String URI = "uri";

/**
* Hive 3's {@code MetastoreConf} name for {@link #HIVE_METASTORE_URIS}. This holder does not bind it, but
* the raw property passthrough hands it to the HMS client, which reads it BEFORE
* {@link #HIVE_METASTORE_URIS}; when set, it is the endpoint the client actually connects to.
*/
public static final String METASTORE_THRIFT_URIS = "metastore.thrift.uris";

/**
* Selects the metastore implementation. Read by {@code HmsClientConfig} (which owns the key and
* its accepted values); named here only so {@link #of} can reject a removed value by its message.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import org.apache.doris.connector.spi.Connector;
import org.apache.doris.connector.spi.ConnectorContext;
import org.apache.doris.connector.spi.ConnectorProvider;
import org.apache.doris.connector.spi.ConnectorSecuritySurface;

import java.util.Collections;
import java.util.HashMap;
Expand Down Expand Up @@ -80,4 +81,13 @@ public void validatePropertiesForUpdate(
candidate.putAll(updatedProperties);
HiveCatalogProperties.of(candidate).checkCreateTimeOnlyRules(updatedProperties);
}

@Override
public ConnectorSecuritySurface securitySurface() {
// Every key the HMS client may take its thrift endpoint from: hive.metastore.uris, its documented alias
// uri, and Hive 3's metastore.thrift.uris, which the client reads first. The engine's SSRF gate rejects
// an intranet target there at CREATE / ALTER CATALOG.
return ConnectorSecuritySurface.ofSsrfEndpoints(HiveCatalogProperties.HIVE_METASTORE_URIS,
HiveCatalogProperties.URI, HiveCatalogProperties.METASTORE_THRIFT_URIS);
}
}
Loading
Loading