Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions docker/thirdparties/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Thirdparty Docker environments

Run a component with Bash 4 or newer and Docker Compose v2:

```bash
bash run-thirdparties-docker.sh -c iceberg
bash run-thirdparties-docker.sh -c iceberg --stop
```

## Rootless Docker on Linux

Install and start the current user's rootless Docker daemon first. Its setup
requires subordinate UID/GID ranges, `newuidmap`/`newgidmap`, and enabled
unprivileged user namespaces. Once configured, launching these environments
does not require host `sudo` or membership in the host `docker` group.

On Linux, the launcher requires a rootless daemon and rejects rootful daemons.
It detects the user's `$XDG_RUNTIME_DIR/docker.sock` socket unless an endpoint
or context was explicitly selected. Docker commands always run as the current
user; there is no automatic elevation or `DOCKER_USE_SUDO` switch.
The endpoint can also be selected explicitly:

```bash
export DOCKER_HOST="unix://${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/docker.sock"
docker info --format '{{json .SecurityOptions}}' # must include name=rootless
bash run-thirdparties-docker.sh -c iceberg
```

On macOS, the launcher uses the current user's Docker Desktop endpoint.

Keep the working directory writable by the current user. The launcher cleans
fixture files owned by subordinate container UIDs from inside the same rootless
user namespace, using `alpine:3.20`. It does not change host ownership to root.

Hive, Kerberos and Iceberg REST's built-in HDFS require Docker Engine 29.5 or
newer for rootless host networking. Hive defaults to `HIVE_HOST_ALIAS=localhost`
and uses loopback for its same-host services. Iceberg REST HDFS uses
host ports 20020, 20870, 20864, 20866 and
20867, so its DataNode block addresses are reachable by host clients.

Kerberos client configuration and keytabs remain under
`docker-compose/kerberos/two-kerberos-hives/`; use the printed `KRB5_CONFIG`
path for host clients. The rootless launcher does not write `/etc/hosts`,
`/etc/krb5.conf` or `/keytabs`. The DB2 image retains its container-level
`privileged` setting within the rootless daemon's user namespace.

`--reserve-ports` changes a host sysctl and is unavailable in rootless mode.
It is optional; an operator can reserve ports separately when required.
Image pulls and fixture downloads still require access to their registries
and artifact repositories. `--no-load-data` skips Hive fixture refresh;
combine it with `--hive-mode rebuild` to validate a fresh Hive installation
without downloading a regression-data baseline.
92 changes: 92 additions & 0 deletions docker/thirdparties/docker-command.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

configure_docker_access() {
local rootless_runtime_dir
local rootless_socket
local security_options

rootless_runtime_dir="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}"
rootless_socket="${rootless_runtime_dir}/docker.sock"

# An explicit endpoint or context takes precedence over socket discovery.
if [[ "${HOST_OS}" != "Darwin" && -z "${DOCKER_HOST+x}" \
&& -z "${DOCKER_CONTEXT:-}" && -S "${rootless_socket}" ]]; then
export DOCKER_HOST="unix://${rootless_socket}"
fi

if ! security_options="$(docker info --format '{{json .SecurityOptions}}')"; then
echo "Unable to access Docker as the current user; start the user's rootless daemon or select its endpoint" >&2
return 1
fi
# Docker Desktop on macOS runs its daemon in a VM. Linux requires the
# user's rootless daemon; an accessible system daemon is not accepted.
if [[ "${HOST_OS}" != "Darwin" && "${security_options}" != *'"name=rootless"'* ]]; then
echo "This launcher requires rootless Docker on Linux; the selected daemon is rootful" >&2
return 1
fi
}

require_rootless_host_network() {
local server_version
local major
local minor

if [[ "${HOST_OS}" == "Darwin" ]]; then
return 0
fi
server_version="$(docker_cli info --format '{{.ServerVersion}}')" || return
major="${server_version%%.*}"
minor="${server_version#*.}"
minor="${minor%%.*}"
# Older rootless host networks cannot reach ports published on the real
# host, which these fixtures use for PostgreSQL and HDFS RPC.
if ((major < 29 || (major == 29 && minor < 5))); then
echo "Rootless host networking requires Docker Engine 29.5 or newer (found ${server_version})" >&2
return 1
fi
}

rootless_fixture_cmd() {
local fixture_dir="$1"
shift

# Stay in the daemon's user namespace when images have chowned bind mounts
# to subordinate UIDs. Mount only the fixture directory being operated on.
docker_cli run --rm --network none --user 0:0 \
--security-opt no-new-privileges \
--mount "type=bind,source=${fixture_dir},target=/fixture" \
"${THIRDPARTY_FILESYSTEM_IMAGE:-alpine:3.20}" "$@"
}

reset_rootless_data_dir() {
local data_dir="$1"

mkdir -p "${data_dir}"
rootless_fixture_cmd "${data_dir}" sh -ec '
find /fixture -mindepth 1 -maxdepth 1 -exec rm -rf -- {} +
chown 0:0 /fixture
chmod 755 /fixture
'
}

chmod_fixture_dir() {
local mode="$1"
local fixture_dir="$2"
rootless_fixture_cmd "${fixture_dir}" chmod -R "${mode}" /fixture
}
5 changes: 3 additions & 2 deletions docker/thirdparties/docker-compose/db2/db2.yaml.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,14 @@ services:
- ${DOCKER_DB2_EXTERNAL_PORT}:50000
privileged: true
healthcheck:
test: ["CMD-SHELL", "su - db2inst1 -c \"source ~/.bash_profile; db2 connect to doris && db2 'select 1 from sysibm.sysdummy1'\""]
test: ["CMD-SHELL", "test -f /tmp/doris_db2_initialized && su - db2inst1 -c \"source ~/.bash_profile; db2 connect to doris && db2 'select count(*) from doris_test.sample_table'\""]
interval: 20s
timeout: 60s
retries: 30
start_period: 300s
volumes:
- ./init:/docker-entrypoint-initdb.d
- ./init:/docker-entrypoint-initdb.d:ro
- ./scripts/init-db2.sh:/var/custom/init-db2.sh:ro
environment:
- LICENSE=accept
- DBNAME=doris
Expand Down
38 changes: 38 additions & 0 deletions docker/thirdparties/docker-compose/db2/scripts/init-db2.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

set -euo pipefail
rm -f /tmp/doris_db2_initialized

# The IBM image executes /var/custom scripts after creating DBNAME. It does
# not execute SQL files mounted at /docker-entrypoint-initdb.d by itself.
su - "${DB2INSTANCE}" -c 'bash -se' <<'EOF'
. "${HOME}/sqllib/db2profile"
db2 connect to doris
schema_count=$(db2 -x "SELECT COUNT(*) FROM SYSCAT.SCHEMATA WHERE SCHEMANAME='DORIS_TEST'")
if ((schema_count == 0)); then
db2 -stvf /docker-entrypoint-initdb.d/02-create-schema.sql
db2 -stvf /docker-entrypoint-initdb.d/03-create-table.sql
db2 -stvf /docker-entrypoint-initdb.d/04-insert.sql
fi
rows=$(db2 -x 'SELECT COUNT(*) FROM doris_test.sample_table')
[[ "${rows}" -eq 3 ]]
db2 connect reset
EOF

touch /tmp/doris_db2_initialized
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,8 @@ services:
ES_8_HOST: "doris--es_8"
command: [ "sh", "-c", "/mnt/scripts/es_init.sh" ]
depends_on:
doris--es_5:
condition: service_healthy
doris--es_6:
condition: service_healthy
doris--es_7:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@
# specific language governing permissions and limitations
# under the License.

set -eo pipefail

generate_bulk_request() {
local index_name=$1
local type_value=$2
Expand Down Expand Up @@ -234,3 +236,4 @@ generate_bulk_request "test_keyword_array" "" "item_" "$keyword_array_data_file"
curl -X POST "http://${ES_8_HOST}:9200/_bulk" --data-binary "@$keyword_array_bulk_file" -H "Content-Type: application/json"

touch /tmp/SUCCESS
exec tail -f /dev/null
4 changes: 2 additions & 2 deletions docker/thirdparties/docker-compose/hive/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,11 +116,11 @@ Relevant env vars:
After bootstrapping a clean Hive stack, stop the containers and run:

```bash
sudo docker compose -p "${CONTAINER_UID}hive3" \
docker compose -p "${CONTAINER_UID}hive3" \
-f docker/thirdparties/docker-compose/hive/hive-3x.yaml down

bash docker/thirdparties/docker-compose/hive/scripts/snapshot-hive-baseline.sh \
"${CONTAINER_UID}hive3" /tmp/hive3-baseline.tar.gz
"doris-shared-hive3" /tmp/hive3-baseline.tar.gz
```

Then upload the resulting tarball to OSS at `oss://<s3BucketName>/regression/datalake/pipeline_data/hive_baseline/hive3-baseline-<version>.tar.gz` (same convention for `hive2`).
Expand Down
4 changes: 2 additions & 2 deletions docker/thirdparties/docker-compose/hive/README_ZH.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,11 +116,11 @@ Hive 运行态(HDFS 数据、Postgres Metastore、模块 SHA 记录)存放
在一次完整 bootstrap 成功后,停止容器并运行:

```bash
sudo docker compose -p "${CONTAINER_UID}hive3" \
docker compose -p "${CONTAINER_UID}hive3" \
-f docker/thirdparties/docker-compose/hive/hive-3x.yaml down

bash docker/thirdparties/docker-compose/hive/scripts/snapshot-hive-baseline.sh \
"${CONTAINER_UID}hive3" /tmp/hive3-baseline.tar.gz
"doris-shared-hive3" /tmp/hive3-baseline.tar.gz
```

然后把得到的 tarball 上传到 `oss://<s3BucketName>/regression/datalake/pipeline_data/hive_baseline/hive3-baseline-<version>.tar.gz`(`hive2` 同理)。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ set -eo pipefail
# snapshot-hive-baseline.sh <volume_prefix> <output_path>
#
# Example:
# bash snapshot-hive-baseline.sh doris--syt--hive3 /tmp/hive3-baseline.tar.gz
# bash snapshot-hive-baseline.sh doris-shared-hive3 /tmp/hive3-baseline.tar.gz
#
# Prerequisites:
# - All Hive containers must be STOPPED before running this script.
Expand All @@ -33,35 +33,48 @@ set -eo pipefail
VOLUME_PREFIX="${1:?Usage: $0 <volume_prefix> <output_path>}"
OUTPUT_PATH="${2:?Usage: $0 <volume_prefix> <output_path>}"

THIRDPARTY_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." &>/dev/null && pwd)"
HOST_OS="$(uname -s)"
# shellcheck source=../../../docker-command.sh
. "${THIRDPARTY_ROOT}/docker-command.sh"
configure_docker_access

docker_cli() {
docker "$@"
}

echo "[snapshot] volume prefix: ${VOLUME_PREFIX}"
echo "[snapshot] output: ${OUTPUT_PATH}"

# Verify all 4 volumes exist
for suffix in namenode datanode pgdata state; do
vol="${VOLUME_PREFIX}-${suffix}"
if ! sudo docker volume inspect "${vol}" >/dev/null 2>&1; then
if ! docker_cli volume inspect "${vol}" >/dev/null 2>&1; then
echo "ERROR: volume ${vol} does not exist" >&2
exit 1
fi
done

_t0=$(date +%s)
tmp_output="$(mktemp "${OUTPUT_PATH}.XXXXXX")"
trap 'rm -f "${tmp_output}"' EXIT

# Mount all 4 volumes read-only into a single alpine container and tar them
# in one pass. Copy only the small state volume to ephemeral container storage
# so legacy baseline.version files can be dropped from newly exported tarballs.
sudo docker run --rm \
-v "${VOLUME_PREFIX}-namenode:/snapshot/namenode:ro" \
-v "${VOLUME_PREFIX}-datanode:/snapshot/datanode:ro" \
-v "${VOLUME_PREFIX}-pgdata:/snapshot/pgdata:ro" \
docker_cli run --rm --network none --security-opt no-new-privileges \
-v "${VOLUME_PREFIX}-namenode:/work/namenode:ro" \
-v "${VOLUME_PREFIX}-datanode:/work/datanode:ro" \
-v "${VOLUME_PREFIX}-pgdata:/work/pgdata:ro" \
-v "${VOLUME_PREFIX}-state:/snapshot/state:ro" \
alpine sh -c '
"${THIRDPARTY_FILESYSTEM_IMAGE:-alpine:3.20}" sh -ec '
mkdir -p /work/state
cp -a /snapshot/state/. /work/state/
rm -f /work/state/baseline.version
tar czf - -C /snapshot namenode datanode pgdata -C /work state
tar czf - -C /work namenode datanode pgdata state
' \
> "${OUTPUT_PATH}"
> "${tmp_output}"
mv "${tmp_output}" "${OUTPUT_PATH}"

size=$(du -h "${OUTPUT_PATH}" | cut -f1)
echo "[snapshot] done took=$(( $(date +%s) - _t0 ))s size=${size}"
Expand Down
2 changes: 1 addition & 1 deletion docker/thirdparties/docker-compose/hudi/hudi.env.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# limitations under the License.
#

CONTAINER_UID=doris--
CONTAINER_UID=${CONTAINER_UID}
HUDI_NETWORK=${CONTAINER_UID}hudi-network

# Ports exposed to host
Expand Down
Loading
Loading