diff --git a/.agents/skills/test-coverage/SKILL.md b/.agents/skills/test-coverage/SKILL.md new file mode 100644 index 0000000000..2c66f80f4f --- /dev/null +++ b/.agents/skills/test-coverage/SKILL.md @@ -0,0 +1,141 @@ +--- +name: test-coverage +description: Measure C++ and Java test coverage of origin/main across every Linux platform (glibc/musl × x64/aarch64) and test JVM CI tests there (HotSpot, OpenJ9, GraalVM), one worktree and subagent per libc×arch, then merge everything into one report. Use when asked for full, merged or cross-platform coverage. +--- + +# Test coverage across Linux platforms + +Each cell runs `:ddprof-lib:coverageReport -PenableCoverage` in a container through +`utils/run-containers-tests.sh --coverage-out`; `utils/merge-coverage.sh` merges the +cells. Read [doc/building/CodeCoverage.md](../../../doc/building/CodeCoverage.md) +first: the C++ numbers cover the `-O0 -DDEBUG` build only. macOS is not part of the +matrix. Optional arguments: `ref=` (default `origin/main`) and +`tests=` (passed as `--tests`, for a quick smoke run). + +## 1. Preflight + +Stop and report on the first failure: +- A container runtime answers: `podman info`, else `docker info`. Use it as `RT` for + every cell. Docker on macOS needs `colima start` first. +- The host has `java`, `python3` and `curl` on `PATH`, and a gcovr at least as new as the one in the base + images (checked below): an older gcovr rejects the cells' tracefiles ("Wrong format + version"). If the host's is older or missing, do not change the user's Python; + after step 2 create `python3 -m venv "$RUN/tools/venv"`, install + `gcovr==` into it, and put `$RUN/tools/venv/bin` first on `PATH` + in step 4. Both are needed only for step 4, but check now. +- Free disk: about 3 GB per worktree plus the container images. +- Memory: `RT info --format '{{.Host.MemTotal}}'` (podman) or `'{{.MemTotal}}'` + (docker), in bytes. Gradle, the `-O0` compile and the test JVMs of one cell need + about 8 GB; with less, the Gradle daemon is killed ("Gradle build daemon + disappeared unexpectedly"). Set `SLOTS` = memory / 8 GB, at most 4. If it is 0, + stop and ask the user to give the VM more memory (`podman machine set --memory`, + `colima start --memory`). +- Other containers: list `RT ps`. Running containers share that memory; ask the + user before stopping any. +- Base images: for each `java-profiler-base:-` that exists, run + `RT run --rm --platform linux/ gcovr --version`; note the + newest version for the host check above. If it fails, that agent's first cell must + pass `--rebuild-base`; the script then rebuilds the JDK images on top of it. + +The non-native architecture runs emulated (QEMU or Rosetta) and takes several times +longer than the ~30 minutes of a native cell. Tell the user the slot count and that +estimate before starting. + +## 2. Worktrees + +```bash +git fetch origin main +COMMIT=$(git rev-parse ) +RUN=$(pwd)/build/test-coverage/$(git rev-parse --short=12 $COMMIT)-$(date -u +%Y%m%dT%H%M%SZ) +for c in glibc-x64 glibc-aarch64 musl-x64 musl-aarch64; do + git worktree add --detach "$RUN/wt-$c" $COMMIT +done +``` + +On macOS, check that `RUN` starts with `$HOME/` and stop if it does not: colima and +podman machine share `$HOME` with their VM by default, and a cell directory the VM +cannot see makes every cell `blocked`. +Check that the worktree's `utils/run-containers-tests.sh` accepts `--coverage-out` +(`grep -q -- --coverage-out`); if it does not, the ref predates coverage mode — stop. + +## 3. Subagents + +Run at most `SLOTS` subagents at a time, in the background (general-purpose): launch +the first `SLOTS` in one message, and launch the next one when an agent sends its +final per-cell report (interim "waiting" notifications do not free a slot). Start +with the emulated architecture's agents: they take longest. +Cells per agent: the JVMs CI tests on each platform (`test_workflow.yml` matrices). +CI's `8-orcl` and `8-ibm` (glibc x64) have no container image and are not covered; +say so in the report. Keep this table in sync when the CI matrices change. + +| Agent | JDKs | +|---|---| +| glibc-x64 | 8 8-j9 11 11-j9 17 17-j9 17-graal 21 21-graal 25 25-graal | +| glibc-aarch64 | 8-j9 11-j9 17 17-j9 17-graal 21 21-graal 25 25-graal | +| musl-x64 | 8 11 17 21 25 | +| musl-aarch64 | 8 11 17 21 25 | + +Give each agent this brief, filled in: + +> Run java-profiler coverage cells for libc=``, arch=``, one at a time, +> in worktree `/wt--`. JDKs, in order: ``. Runtime: ``. +> For each JDK ``, let `` be `/cells/--` and `` +> be `/logs/--.log` (absolute paths; `mkdir -p /logs`). +> Before every start, retries included, run `rm -rf .exit` so the wait +> below cannot end on an earlier attempt's result. Start the cell detached so no tool +> timeout kills it: +> `cd && nohup bash -c './utils/run-containers-tests.sh --container= +> --libc= --arch= --jdk= --coverage-out= [--tests=]; +> echo $? > .exit' > 2>&1 &`. +> Then wait until `.exit` exists (Monitor with an until-loop, or a background +> `while [ ! -f .exit ]; do sleep 60; done`, re-armed if it times out). Do not +> start the next cell before the current one finishes. +> Classify each cell from the exit code, `/cell.properties` and the log: +> `passed` (exit 0); `failed` (a task failed on its own terms: compile error, test +> failures, gtest failure — name the task and tests); `blocked` (no +> `cell.properties`: image build, download or container failure; or Gradle was +> killed: "Gradle build daemon disappeared unexpectedly", exit 137, `Killed`; or exit +> 70: the export to `` failed). If +> the log says a coverage tool is missing (`checkCoverageTools`), the cached base +> image predates it: rerun that cell once with `--rebuild-base`. Retry any other +> `blocked` cell once. Do not edit code, change tests or retry `failed` cells. Never +> print more than 30 log lines. Never put the word "gradlew" in a shell command, not +> even in a grep pattern: a hook blocks it — search logs for `What went wrong` or +> `FAILED` instead. +> Report one line per cell: jdk, status, gradle_exit, failing task/tests or the +> blocking error, log path. + +While they run, answer the user from their notifications only; if asked for +progress, list which `$RUN/cells/*.exit` files exist. + +## 4. Merge + +After all four agents send their final reports, merge every cell that has +`cell.properties`. A `failed` cell still holds the Java coverage of the tests that ran, +but a failed gtest stops the build before the C++ report, so such a cell has no C++ +tracefile. `summary.md` marks, per cell, whether it is in each report. + +```bash +WT=$(ls -d "$RUN"/wt-* | head -1) # any of the run's worktrees +cells=(); for p in "$RUN"/cells/*/cell.properties; do cells+=("${p%/cell.properties}"); done +"$WT/utils/merge-coverage.sh" --source-root="$WT" --out="$RUN/merged" "${cells[@]}" +``` + +If gcovr rejects a tracefile, report the gcovr error and the cell; do not drop the +cell silently. + +## 5. Report + +- The commit, and a per-cell table: libc, arch, jdk, `passed`/`failed`/`blocked`, + failing tests or blocking error. +- The merged totals from `$RUN/merged/summary.md`, and paths to + `merged/cpp/index.html` and `merged/java/html/index.html`. +- Which cells are missing from the merge or from one of its reports (the C++ and + Java columns of `summary.md`) and why, and that every `failed` or missing cell + lowers the numbers. Say so when `tests=` filtered the run. The C++ report is the + debug build only. + +Leave `$RUN` in place. Offer to remove the worktrees +(`git worktree remove "$RUN"/wt-*`); `cells/`, `logs/` and `merged/` stay until the +user deletes them. On Linux with rootful Docker the container writes `cells/` as +root; remove them with `sudo` or from a container. diff --git a/AGENTS.md b/AGENTS.md index 4ccd3e6aaf..8901cacb2e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -87,7 +87,8 @@ dependency-free leak-demo app; not published). Configurations: `release`, `debug`, plus `asan`/`tsan`/`fuzzer` when the sanitizer runtime is found for the selected compiler (inactive on macOS and musl), plus opt-in `coverage` with `-PenableCoverage` (`:ddprof-lib:coverageReport`; the C++ report covers only the -`-O0 -DDEBUG` build — see [doc/building/CodeCoverage.md](doc/building/CodeCoverage.md)). +`-O0 -DDEBUG` build — see [doc/building/CodeCoverage.md](doc/building/CodeCoverage.md); +merged coverage across the CI-tested Linux platforms and JVMs: the `test-coverage` skill). ```bash ./.claude/commands/build-and-summarize buildRelease # release library + JAR diff --git a/ddprof-lib/build.gradle.kts b/ddprof-lib/build.gradle.kts index bb865ed6ca..4c7303fc2b 100644 --- a/ddprof-lib/build.gradle.kts +++ b/ddprof-lib/build.gradle.kts @@ -427,6 +427,8 @@ if (project.hasProperty("enableCoverage")) { "--gcov-executable", gcovExecutable, "--html", "--html-details", "-o", "${outDir.absolutePath}/index.html", + // Tracefile for merging runs from several platforms (gcovr --add-tracefile). + "--json", "${outDir.absolutePath}/coverage.json", "--print-summary", // Safety net only: -fprofile-update=atomic (see configureCoverage) should keep // counters consistent, but a stray negative count must not discard a diff --git a/ddprof-test/build.gradle.kts b/ddprof-test/build.gradle.kts index 828cb50fdc..37e30a3742 100644 --- a/ddprof-test/build.gradle.kts +++ b/ddprof-test/build.gradle.kts @@ -148,6 +148,11 @@ tasks.withType().named { it in coverageTaskNames }.configureEach { environment("JACOCO_CHILD_DIR", childJacocoDir.get().asFile.absolutePath) environment("JACOCO_INCLUDES", jacocoIncludes) } + // -Ptests filters both coverage tasks, and a pattern usually matches tests of only one + // (slow or not); an empty match must not stop the build before the reports. + if (project.hasProperty("tests")) { + filter.isFailOnNoMatchingTests = false + } } // On musl the profiler-test plugin creates the test tasks as Exec tasks (running its own diff --git a/doc/building/CodeCoverage.md b/doc/building/CodeCoverage.md index dcd86ef5e3..059c6172fe 100644 --- a/doc/building/CodeCoverage.md +++ b/doc/building/CodeCoverage.md @@ -6,7 +6,7 @@ | Report | Tool | Sources | Output | |--------|------|---------|--------| -| C++ | gcovr | gtest binaries + Java integration tests | `ddprof-lib/build/reports/coverage/cpp/index.html` | +| C++ | gcovr | gtest binaries + Java integration tests | `ddprof-lib/build/reports/coverage/cpp/index.html`, tracefile `.../cpp/coverage.json` | | Java | JaCoCo | Java integration tests (incl. child JVMs) | `ddprof-lib/build/reports/coverage/java/html/index.html`, `.../java/jacoco.xml` | Coverage is opt-in. Pass `-PenableCoverage`, or none of the coverage tasks exist. @@ -105,15 +105,27 @@ failing gtest stops the build before the report. `utils/run-containers-tests.sh` base images include gcovr from PyPI, and `llvm` on glibc for the unversioned `llvm-cov`. The script reuses cached images, so a base image -cached before these packages were added lacks them, and so does every JDK test image -built on it. Pass `--rebuild-base` once: it rebuilds the base image and the JDK image -on top of it. +cached before these packages were added lacks them. Pass `--rebuild-base` once per +libc and architecture. Each JDK image records the ID of the base image it was built +on, and the script rebuilds it when that base image changes. -The script's `--config` does not accept `coverage`. Open a shell in the container and -run the task there: +`--coverage-out=DIR` runs the report for one cell and copies the reports, the JaCoCo +`.exec` files, the classes they were recorded against and a `cell.properties` file to +`DIR` (see [utils/README.md](../../utils/README.md)). A failing gtest stops the build +before the report, so that cell exports no C++ tracefile. ```bash -./utils/run-containers-tests.sh --libc=glibc --jdk=21 --rebuild-base --shell -# inside the container: -./gradlew :ddprof-lib:coverageReport -PenableCoverage +./utils/run-containers-tests.sh --libc=musl --arch=aarch64 --jdk=21 --coverage-out=build/cov/musl-aarch64-21 ``` + +## Merging Platforms + +One run covers one libc, architecture and test JVM. Code that only runs on another +platform (aarch64 stack walking, musl paths, J9 support) shows as not covered. +`utils/merge-coverage.sh` merges `--coverage-out` directories built from one commit: +gcovr merges the `coverage.json` tracefiles, and the JaCoCo CLI merges the `.exec` +files and reports against one cell's classes. Every container builds with JDK 21, so +the class files match across cells; the script warns when they differ. The host's +gcovr must be at least as new as the containers': an older one rejects their +tracefiles with "Wrong format version". The +`test-coverage` skill runs the full Linux matrix and merges it. diff --git a/doc/building/TestingGuide.md b/doc/building/TestingGuide.md index 8c025ad493..779a05539f 100644 --- a/doc/building/TestingGuide.md +++ b/doc/building/TestingGuide.md @@ -335,8 +335,9 @@ The C++ report covers only the `-O0 -DDEBUG` build. Prerequisites and limits are (ddtool) fails outside the CI network and aborts container commands. Run `./utils/run-containers-tests.sh --help` for the full flag list (`--tests`, `--gtest`, `--gtest-task`, `--mount`, `--rebuild`, `--matrix`, ...). Cached images are reused until `--rebuild-base` (or `--rebuild`) is passed. Use it -after the base image's package list changes, e.g. to pick up gcovr for coverage runs: -it rebuilds the base image and the JDK image built on it. +after the base image's package list changes, e.g. to pick up gcovr for coverage runs. +A JDK image is also rebuilt when it was built on another base image than the current +one (the `java-profiler.base-id` label), so every JDK picks up the rebuilt base. ```bash # Matches the nightly configuration diff --git a/utils/README.md b/utils/README.md index 0fc0f25253..5e0876d2b8 100644 --- a/utils/README.md +++ b/utils/README.md @@ -175,12 +175,13 @@ Runs tests in containers across various OS/libc/JDK combinations, mirroring the --arch=x64|aarch64|all[,..] (default: auto-detect) --config=debug|release|asan|tsan|all[,..] (default: debug) --container=podman|docker (default: podman) - --tests="TestPattern" (optional) + --tests="TestPattern" (optional; letters, digits and _ . * # , -) --gtest (enable C++ gtests) --gtest-task=Task (run one C++ gtest task) --shell (drop to shell instead of running tests) --mount (mount local repo instead of cloning) --rebuild (force rebuild of container images) + --coverage-out=DIR (single cell: coverageReport, exported to DIR) --matrix (preview a full matrix) --run (execute an inferred matrix without prompting) --fail-fast (stop matrix execution on first failure) @@ -206,6 +207,16 @@ Examples: Single-value commands run one container test configuration immediately. When any dimension expands to multiple cells, the script prints a compact status table first; interactive terminals ask for confirmation, while non-interactive runs require `--run` to execute. Matrix execution prints the status table again after all cells finish and writes summaries to `build/reports/container-matrix/summary.md` and `build/reports/container-matrix/summary.json`. Matrix gtest runs require a short `--gtest-task` name so the task follows each cell's configuration. Cells not run because of `--fail-fast` are reported as cancelled separately from unsupported cells that are skipped. +`--coverage-out=DIR` runs `:ddprof-lib:coverageReport -PenableCoverage` for one cell and replaces in `DIR` the reports (`reports/`, including the gcovr tracefile `reports/cpp/coverage.json`), the JaCoCo `.exec` files (`jacoco/`), the classes they were recorded against (`classes/`) and `cell.properties` (libc, arch, jdk, tests filter, commit, Gradle exit code). The export also runs when Gradle fails; the script still exits with Gradle's code, or with 70 when Gradle passed but the export failed. The script can run from a git worktree: it also mounts the main repository's git dir, read-only. + +### `merge-coverage.sh` + +Merges `--coverage-out` directories from several cells into one C++ report (gcovr `--add-tracefile`) and one Java report (JaCoCo CLI `merge` + `report`), plus `summary.md`. All cells and `--source-root` must be at the same commit. Needs `java`, `python3` and a gcovr at least as new as the containers' (an older one rejects their tracefiles); the JaCoCo CLI comes from `$JACOCO_CLI` or is downloaded from Maven Central with `curl` and checked against a pinned SHA-256. `--out` is replaced on each run, and `summary.md` marks which cells each report includes. The `test-coverage` skill drives the full Linux matrix and this merge. + +```bash +./utils/merge-coverage.sh --source-root=. --out=build/coverage-merged cells/glibc-x64-21 cells/musl-aarch64-17 +``` + ### `check-signal-reachability.py` Static async-signal-safety check. Builds a call graph from the relocations of the debug object files (`objdump -dr`) and reports every function reachable from a signal handler or malloc hook that calls an unsafe symbol (heap allocation, locks, stdio, dynamic-loader calls, dynamic TLS, logging). The entry points, unsafe symbols and known findings are data lists at the top of the script. ELF objects only (Linux x64/aarch64); needs GNU `objdump` and `c++filt`. diff --git a/utils/merge-coverage.sh b/utils/merge-coverage.sh new file mode 100755 index 0000000000..ff766e8853 --- /dev/null +++ b/utils/merge-coverage.sh @@ -0,0 +1,220 @@ +#!/usr/bin/env bash +# Copyright 2026, Datadog, Inc +# +# Merge coverage exported by `run-containers-tests.sh --coverage-out=DIR` from several +# platform cells into one C++ (gcovr) and one Java (JaCoCo) report. +# +# Usage: ./utils/merge-coverage.sh --source-root=DIR --out=DIR CELL_DIR... +# --source-root=DIR checkout of the commit every cell was built from (report sources) +# --out=DIR output directory (cpp/, java/, summary.md); replaced on each run +# CELL_DIR a --coverage-out directory; needs cell.properties +# +# Requires java, python3 and a gcovr at least as new as the containers' (older ones +# reject their tracefiles). The JaCoCo CLI is $JACOCO_CLI, else downloaded with curl. + +set -euo pipefail + +# Must match jacoco.toolVersion in ddprof-test/build.gradle.kts. +JACOCO_VERSION="0.8.14" +JACOCO_CLI_SHA256="811c7f8c6b358c5d68a8973cfa867f6892be7a671b697a4b13c4b447e6daf75c" +SOURCE_ROOT="" +OUT="" +CELLS=() + +usage() { + awk 'NR >= 4 && NR <= 13 { sub(/^# ?/, ""); print }' "$0" + exit "${1:-0}" +} + +die() { + echo "Error: $*" >&2 + exit 1 +} + +prop() { + sed -n "s/^$2=//p" "$1/cell.properties" +} + +for arg in "$@"; do + case "$arg" in + --source-root=*) SOURCE_ROOT="${arg#*=}" ;; + --out=*) OUT="${arg#*=}" ;; + --help|-h) usage 0 ;; + -*) echo "Unknown option: $arg" >&2; usage 1 ;; + *) CELLS+=("$arg") ;; + esac +done + +[[ -n "$SOURCE_ROOT" ]] || die "--source-root is required" +[[ -n "$OUT" ]] || die "--out is required" +(( ${#CELLS[@]} > 0 )) || die "no cell directories given" +CPP_SRC="$SOURCE_ROOT/ddprof-lib/src/main/cpp" +JAVA_SRC="$SOURCE_ROOT/ddprof-lib/src/main/java" +[[ -d "$CPP_SRC" && -d "$JAVA_SRC" ]] || die "$SOURCE_ROOT is not a java-profiler checkout" + +# Every cell must come from one commit: gcovr matches lines by number, and JaCoCo +# ignores execution data recorded against different class bytes. +COMMIT="" +for cell in "${CELLS[@]}"; do + [[ -f "$cell/cell.properties" ]] || die "$cell has no cell.properties" + cell_commit=$(prop "$cell" commit) + [[ -n "$cell_commit" ]] || die "$cell has no commit in cell.properties" + if [[ -z "$COMMIT" ]]; then + COMMIT=$cell_commit + elif [[ "$cell_commit" != "$COMMIT" ]]; then + die "$cell was built from $cell_commit, other cells from $COMMIT" + fi +done +source_commit=$(git -C "$SOURCE_ROOT" rev-parse HEAD 2>/dev/null || true) +if [[ "$source_commit" != "$COMMIT" ]]; then + die "--source-root is at ${source_commit:-}, cells were built from $COMMIT" +fi + +mkdir -p "$OUT" +OUT="$(cd "$OUT" && pwd)" +# --out is replaced, so it must be empty or carry the marker an earlier merge wrote. +MARKER=".merge-coverage-output" +if [[ -n "$(ls -A "$OUT")" && ! -f "$OUT/$MARKER" ]]; then + die "--out $OUT is neither empty nor an earlier merge output" +fi +touch "$OUT/$MARKER" +# Keep tools/ (the downloaded CLI); a report left from an earlier merge would be linked +# from this run's summary. +rm -rf "$OUT/cpp" "$OUT/java" "$OUT/summary.md" +mkdir -p "$OUT/cpp" "$OUT/java" + +# Per-cell inclusion, indexed like CELLS (bash 3.2 has no associative arrays). +CELL_CPP=() +CELL_JAVA=() + +# ---- C++ ---- +tracefiles=() +for i in "${!CELLS[@]}"; do + cell=${CELLS[$i]} + if [[ -f "$cell/reports/cpp/coverage.json" ]]; then + tracefiles+=(--add-tracefile "$cell/reports/cpp/coverage.json") + CELL_CPP[$i]=yes + else + echo "Warning: $cell has no C++ tracefile; it is left out of the C++ report" >&2 + CELL_CPP[$i]="no (no tracefile)" + fi +done +CPP_SUMMARY="(no C++ tracefiles)" +if (( ${#tracefiles[@]} > 0 )); then + # Compilers differ between cells (glibc vs musl toolchains), so a function can be + # reported at different lines; keep those entries apart instead of failing. + gcovr --root "$CPP_SRC" "${tracefiles[@]}" \ + --merge-mode-functions=separate \ + --html --html-details -o "$OUT/cpp/index.html" \ + --json "$OUT/cpp/coverage.json" \ + --print-summary | tee "$OUT/cpp/summary.txt" + CPP_SUMMARY=$(cat "$OUT/cpp/summary.txt") +fi + +# ---- Java ---- +execs=() +CLASSES="" +for i in "${!CELLS[@]}"; do + cell=${CELLS[$i]} + cell_execs=0 + if [[ -d "$cell/jacoco" ]]; then + while IFS= read -r f; do + execs+=("$f") + cell_execs=$((cell_execs + 1)) + done < <(find "$cell/jacoco" -name '*.exec' -type f | sort) + fi + if (( cell_execs == 0 )); then + CELL_JAVA[$i]="no (no JaCoCo data)" + elif [[ ! -d "$cell/classes" ]]; then + CELL_JAVA[$i]=yes + elif [[ -z "$CLASSES" ]]; then + CLASSES="$cell/classes" + CELL_JAVA[$i]=yes + elif diff -rq "$CLASSES" "$cell/classes" >/dev/null; then + CELL_JAVA[$i]=yes + else + echo "Warning: classes in $cell differ from $CLASSES; JaCoCo drops its data for the differing classes" >&2 + CELL_JAVA[$i]="partial (classes differ)" + fi +done +JAVA_SUMMARY="(no JaCoCo execution data)" +if (( ${#execs[@]} > 0 )) && [[ -z "$CLASSES" ]]; then + JAVA_SUMMARY="(no cell exported classes)" + for i in "${!CELLS[@]}"; do + [[ "${CELL_JAVA[$i]}" != yes ]] || CELL_JAVA[$i]="no (no classes in any cell)" + done +fi +if (( ${#execs[@]} > 0 )) && [[ -n "$CLASSES" ]]; then + cli="${JACOCO_CLI:-}" + if [[ -z "$cli" ]]; then + cli="$OUT/tools/org.jacoco.cli-$JACOCO_VERSION-nodeps.jar" + if [[ ! -f "$cli" ]]; then + mkdir -p "$OUT/tools" + url="https://repo1.maven.org/maven2/org/jacoco/org.jacoco.cli/$JACOCO_VERSION/org.jacoco.cli-$JACOCO_VERSION-nodeps.jar" + curl -fsSL -o "$cli.tmp" "$url" + actual=$( (shasum -a 256 "$cli.tmp" 2>/dev/null || sha256sum "$cli.tmp") | cut -c1-64) + if [[ "$actual" != "$JACOCO_CLI_SHA256" ]]; then + rm -f "$cli.tmp" + die "SHA-256 mismatch for $url" + fi + mv "$cli.tmp" "$cli" + fi + fi + java -jar "$cli" merge "${execs[@]}" --destfile "$OUT/java/merged.exec" + java -jar "$cli" report "$OUT/java/merged.exec" \ + --classfiles "$CLASSES" --sourcefiles "$JAVA_SRC" \ + --html "$OUT/java/html" --xml "$OUT/java/jacoco.xml" + JAVA_SUMMARY=$(python3 -I - "$OUT/java/jacoco.xml" <<'EOF' +import sys +import xml.etree.ElementTree as ET + +root = ET.parse(sys.argv[1]).getroot() +for c in root.findall("counter"): + covered, missed = int(c.get("covered")), int(c.get("missed")) + total = covered + missed + pct = 100.0 * covered / total if total else 0.0 + print(f"{c.get('type').lower()}: {pct:.1f}% ({covered} out of {total})") +EOF +) +fi + +# ---- Summary ---- +FILTERED=false +for cell in "${CELLS[@]}"; do + [[ -z "$(prop "$cell" tests)" ]] || FILTERED=true +done +{ + echo "# Merged coverage" + echo + echo "- Commit: \`$COMMIT\`" + if [[ -f "$OUT/cpp/index.html" ]]; then + echo "- C++ report: \`cpp/index.html\`" + fi + if [[ -f "$OUT/java/html/index.html" ]]; then + echo "- Java report: \`java/html/index.html\`" + fi + if $FILTERED; then + echo "- Filtered run: some cells ran only the tests matching their \`tests\` pattern" + fi + echo + echo "| libc | arch | jdk | tests | gradle exit | in C++ report | in Java report |" + echo "|---|---|---|---|---|---|---|" + for i in "${!CELLS[@]}"; do + cell=${CELLS[$i]} + echo "| $(prop "$cell" libc) | $(prop "$cell" arch) | $(prop "$cell" jdk) | $(prop "$cell" tests) | $(prop "$cell" gradle_exit) | ${CELL_CPP[$i]} | ${CELL_JAVA[$i]} |" + done + echo + echo "## C++ (debug build, cells marked in the C++ column)" + echo + echo '```text' + echo "$CPP_SUMMARY" + echo '```' + echo + echo "## Java (cells marked in the Java column)" + echo + echo '```text' + echo "$JAVA_SUMMARY" + echo '```' +} > "$OUT/summary.md" + +echo ">>> Merged coverage written to $OUT (summary.md, cpp/index.html, java/html/index.html)" diff --git a/utils/run-containers-tests.sh b/utils/run-containers-tests.sh index feae25bc58..b808e3aeef 100755 --- a/utils/run-containers-tests.sh +++ b/utils/run-containers-tests.sh @@ -21,6 +21,8 @@ # --mount (mount local repo instead of cloning - faster but may have stale artifacts) # --rebuild (force rebuild of container images) # --rebuild-base (force rebuild of the base image, and of the JDK image built on it) +# --coverage-out=DIR (single cell: run :ddprof-lib:coverageReport -PenableCoverage and +# export reports, raw JaCoCo data and classes to DIR) # --matrix (preview a full matrix; unset dimensions expand to all) # --run (execute an inferred matrix without prompting) # --fail-fast (stop matrix execution on first failure) @@ -37,6 +39,8 @@ ARCH="" CONFIG="debug" TESTS="" GTEST_TASK="" +COVERAGE_OUT="" +COVERAGE_OUT_SET=false SHELL_MODE=false MOUNT_MODE=false GTEST_ENABLED=false @@ -767,7 +771,7 @@ run_matrix() { } usage() { - awk 'NR >= 4 && NR <= 27 { sub(/^# ?/, ""); print }' "$0" + awk 'NR >= 4 && NR <= 29 { sub(/^# ?/, ""); print }' "$0" exit 0 } @@ -834,6 +838,11 @@ while [[ $# -gt 0 ]]; do PASS_THROUGH_ARGS+=("$1") shift ;; + --coverage-out=*) + COVERAGE_OUT="${1#*=}" + COVERAGE_OUT_SET=true + shift + ;; --matrix) MATRIX_MODE=true shift @@ -886,6 +895,26 @@ if [[ -n "$GTEST_TASK" && -n "$TESTS" ]]; then exit 1 fi +# The pattern is spliced unquoted into a command that the host evals and the container +# shell runs again (and into cell.properties), so only test-pattern characters are safe. +if [[ "$TESTS" =~ [^A-Za-z0-9_.*#,-] ]]; then + echo "Error: --tests may contain only letters, digits and _ . * # , -" + exit 1 +fi + +if $COVERAGE_OUT_SET; then + if [[ -z "$COVERAGE_OUT" ]]; then + echo "Error: --coverage-out needs a directory" + exit 1 + fi + if $SHELL_MODE || $MOUNT_MODE || $MATRIX_MODE || $CONFIG_SET || [[ -n "$GTEST_TASK" ]]; then + echo "Error: --coverage-out cannot be combined with --shell, --mount, --matrix, --config or --gtest-task" + exit 1 + fi + # coverageReport depends on gtestCoverage. + GTEST_ENABLED=true +fi + MATRIX_LIBCS_TEXT=$(expand_dimension "$LIBC" "libc" glibc musl) MATRIX_ARCHES_TEXT=$(expand_dimension "$ARCH" "arch" x64 aarch64) MATRIX_JDKS_TEXT=$(expand_jdk_dimension "$JDK_VERSION") @@ -909,6 +938,10 @@ if (( MATRIX_RUNNABLE == 0 )); then fi if (( MATRIX_TOTAL > 1 || MATRIX_RUNNABLE > 1 )) || $MATRIX_MODE; then + if [[ -n "$COVERAGE_OUT" ]]; then + echo "Error: --coverage-out runs a single cell; select one libc, arch and jdk" + exit 1 + fi if $SHELL_MODE; then echo "Error: --shell is interactive and is not supported with matrix runs. Run a single cell without matrix dimensions instead." exit 1 @@ -935,6 +968,16 @@ LIBC="${MATRIX_CELL_LIBCS[0]}" ARCH="${MATRIX_CELL_ARCHES[0]}" JDK_VERSION="${MATRIX_CELL_JDKS[0]}" CONFIG="${MATRIX_CELL_CONFIGS[0]}" +if [[ -n "$COVERAGE_OUT" ]]; then + CONFIG="coverage" + mkdir -p "$COVERAGE_OUT" + COVERAGE_OUT="$(cd "$COVERAGE_OUT" && pwd)" + # A run that dies before the in-container export must not leave the previous one. + # Best effort: a rootful container's files are root-owned, and its export, which + # runs as root, clears them again. + rm -rf "$COVERAGE_OUT/reports" "$COVERAGE_OUT/jacoco" "$COVERAGE_OUT/classes" \ + "$COVERAGE_OUT/cell.properties" 2>/dev/null || true +fi # Parse JDK version and variant (e.g., "21-j9" -> version="21", variant="j9") JDK_BASE_VERSION="${JDK_VERSION%%-*}" @@ -1116,11 +1159,17 @@ fi # ========== Build JDK Image (if needed) ========== IMAGE_EXISTS=false # A JDK image cached on top of an older base image would hide the rebuilt base's packages -# (e.g. gcovr), so it is reused only when the base image was reused too. -if [[ "$REBUILD" == "false" && "$BASE_IMAGE_EXISTS" == "true" ]]; then - if "$CONTAINER_RUNTIME" image inspect "$IMAGE_NAME" >/dev/null 2>&1; then +# (e.g. gcovr), so it is reused only when its label names the current base image ID; +# an image without the label is rebuilt. +BASE_IMAGE_ID=$("$CONTAINER_RUNTIME" image inspect --format '{{.Id}}' "$BASE_IMAGE_NAME") +if [[ "$REBUILD" == "false" && "$REBUILD_BASE" == "false" ]]; then + IMAGE_BASE_ID=$("$CONTAINER_RUNTIME" image inspect \ + --format '{{index .Config.Labels "java-profiler.base-id"}}' "$IMAGE_NAME" 2>/dev/null || true) + if [[ -n "$BASE_IMAGE_ID" && "$IMAGE_BASE_ID" == "$BASE_IMAGE_ID" ]]; then IMAGE_EXISTS=true echo ">>> Using cached image: $IMAGE_NAME" + elif [[ -n "$IMAGE_BASE_ID" ]] || "$CONTAINER_RUNTIME" image inspect "$IMAGE_NAME" >/dev/null 2>&1; then + echo ">>> Cached image $IMAGE_NAME was not built on the current $BASE_IMAGE_NAME; rebuilding" fi fi @@ -1199,7 +1248,8 @@ WORKDIR /workspace EOF fi - "$CONTAINER_RUNTIME" build $CONTAINER_PLATFORM -t "$IMAGE_NAME" -f "$DOCKERFILE_DIR/Dockerfile" "$DOCKERFILE_DIR" + "$CONTAINER_RUNTIME" build $CONTAINER_PLATFORM --label "java-profiler.base-id=$BASE_IMAGE_ID" \ + -t "$IMAGE_NAME" -f "$DOCKERFILE_DIR/Dockerfile" "$DOCKERFILE_DIR" echo ">>> JDK image built: $IMAGE_NAME" fi @@ -1209,7 +1259,12 @@ fi # Capitalize first letter for gradle task names (testDebug, testAsan, etc.) # Note: -Ptests property works uniformly across all platforms (glibc, musl, macOS) CONFIG_CAPITALIZED="$(tr '[:lower:]' '[:upper:]' <<< ${CONFIG:0:1})${CONFIG:1}" -if [[ -n "$GTEST_TASK" ]]; then +if [[ -n "$COVERAGE_OUT" ]]; then + GRADLE_CMD="./gradlew -PCI -PenableCoverage :ddprof-lib:coverageReport" + if [[ -n "$TESTS" ]]; then + GRADLE_CMD="$GRADLE_CMD -Ptests=$TESTS" + fi +elif [[ -n "$GTEST_TASK" ]]; then if [[ "$GTEST_TASK" == :* ]]; then GRADLE_CMD="./gradlew -PCI -PkeepJFRs $GTEST_TASK" else @@ -1232,6 +1287,19 @@ if [[ "$CONFIG" == "tsan" ]] && [[ "$ARCH" == "aarch64" ]] && [[ "$LIBC" == "gli fi GRADLE_CMD="$GRADLE_CMD --no-daemon --parallel --build-cache --no-watch-fs" +# Coverage mode: replace DIR's previous export with this run's reports, JaCoCo .exec files +# and their classes, also after a Gradle failure (coverageReport writes both reports +# first). No single quotes: the command runs through eval inside '...'. +if [[ -n "$COVERAGE_OUT" ]]; then + COVERAGE_EXPORT="cd /workspace && rm -rf /coverage-out/reports /coverage-out/jacoco /coverage-out/classes /coverage-out/cell.properties" + COVERAGE_EXPORT="$COVERAGE_EXPORT && { [ ! -d ddprof-lib/build/reports/coverage ] || cp -r ddprof-lib/build/reports/coverage /coverage-out/reports; }" + COVERAGE_EXPORT="$COVERAGE_EXPORT && { [ ! -d ddprof-test/build/jacoco ] || cp -r ddprof-test/build/jacoco /coverage-out/jacoco; }" + COVERAGE_EXPORT="$COVERAGE_EXPORT && { [ ! -d ddprof-lib/build/classes/java/main ] || cp -r ddprof-lib/build/classes/java/main /coverage-out/classes; }" + COVERAGE_EXPORT="$COVERAGE_EXPORT && printf \"libc=%s\\narch=%s\\njdk=%s\\ntests=%s\\ncommit=%s\\ngradle_exit=%s\\n\" $LIBC $ARCH $JDK_VERSION \"$TESTS\" \"\$(git rev-parse HEAD)\" \$rc > /coverage-out/cell.properties" + # A failed export must not pass as a clean cell. + GRADLE_CMD="{ rc=0; $GRADLE_CMD || rc=\$?; { $COVERAGE_EXPORT; } || { [ \$rc -ne 0 ] || rc=70; }; exit \$rc; }" +fi + # On aarch64 glibc TSan: reduce ASLR entropy so TSan's shadow doesn't conflict with # initial library load addresses. Requires --privileged for sysctl in the container. # Do NOT set kernel.randomize_va_space=0: ld-linux-aarch64.so loads at 0x2000000000 @@ -1262,6 +1330,9 @@ CONTAINER_CMD="$CONTAINER_CMD -e LIBC=$LIBC" CONTAINER_CMD="$CONTAINER_CMD -e SANITIZER=$CONFIG" CONTAINER_CMD="$CONTAINER_CMD -e TEST_CONFIGURATION=$LIBC/${JDK_VERSION}-$CONFIG-$ARCH" CONTAINER_CMD="$CONTAINER_CMD -e GRADLE_USER_HOME=/gradle-cache" +if [[ -n "$COVERAGE_OUT" ]]; then + CONTAINER_CMD="$CONTAINER_CMD -v \"$COVERAGE_OUT\":/coverage-out${CONTAINER_VOLUME_RW_OPTIONS}" +fi if $MOUNT_MODE; then # Mount mode: use local repo directly (faster, but may have stale artifacts) @@ -1282,6 +1353,13 @@ else # Clone mode: shallow clone from mounted local repo for clean builds (default) # Mount the local repo as source, then clone from it to /workspace CONTAINER_CMD="$CONTAINER_CMD -v \"$PROJECT_ROOT\":/source${CONTAINER_VOLUME_RO_OPTIONS}" + # A git worktree's .git is a file naming an absolute gitdir inside the main + # repository; mount that repository's git dir at the same path so the clone resolves. + GIT_COMMON_DIR="" + if [[ -f "$PROJECT_ROOT/.git" ]]; then + GIT_COMMON_DIR=$(git -C "$PROJECT_ROOT" rev-parse --path-format=absolute --git-common-dir) + CONTAINER_CMD="$CONTAINER_CMD -v \"$GIT_COMMON_DIR\":\"$GIT_COMMON_DIR\"${CONTAINER_VOLUME_RO_OPTIONS}" + fi CONTAINER_CMD="$CONTAINER_CMD $IMAGE_NAME" # Build clone and test command - clone from local mounted source. @@ -1291,6 +1369,10 @@ else # "-c safe.directory=..." to git clone is not enough (tested with git 2.34.1 # in the glibc image), unlike for the commands in utils/compute-version.sh. SAFE_DIR_CMD="git config --global --add safe.directory /source/.git && " + if [[ -n "$GIT_COMMON_DIR" ]]; then + # The worktree's checkout and its gitdir both fail git's ownership check. + SAFE_DIR_CMD="git config --global --add safe.directory \"*\" && " + fi if $SHELL_MODE; then CLONE_CMD="${SAFE_DIR_CMD}git clone --depth 1 file:///source /workspace && cd /workspace && /bin/bash" else diff --git a/utils/tests/test_coverage_scripts.sh b/utils/tests/test_coverage_scripts.sh new file mode 100644 index 0000000000..9fef6fb4a1 --- /dev/null +++ b/utils/tests/test_coverage_scripts.sh @@ -0,0 +1,426 @@ +#!/usr/bin/env bash + +# Copyright 2026, Datadog, Inc + +# Hermetic tests for the coverage paths of utils/run-containers-tests.sh +# (--coverage-out, worktree sources, JDK image reuse) and for utils/merge-coverage.sh. +# The container runtime, gcovr and java are stubs that record their arguments. +# Run with: bash utils/tests/test_coverage_scripts.sh + +set -euo pipefail + +SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +UTILS_DIR="$SCRIPT_DIR/.." +TESTS=0 +FAILED=0 + +TEMP_DIR=$(mktemp -d) +trap 'rm -rf "$TEMP_DIR"' EXIT + +pass() { TESTS=$((TESTS + 1)); } +fail() { + TESTS=$((TESTS + 1)) + FAILED=$((FAILED + 1)) + echo "FAIL: $*" >&2 +} + +# Usage: expect_contains +expect_contains() { + if [[ "$2" == *"$3"* ]]; then pass; else fail "$1: expected to contain [$3], got: $2"; fi +} +expect_not_contains() { + if [[ "$2" != *"$3"* ]]; then pass; else fail "$1: expected not to contain [$3], got: $2"; fi +} + +# ---- Stubs ---- +STUB_BIN="$TEMP_DIR/bin" +mkdir -p "$STUB_BIN" +# podman: every image exists; the base image ID is ${STUB_BASE_ID-base-1} and a JDK image's +# base-id label is $STUB_LABEL. `run` records one argument per line; with $STUB_EXEC it runs +# the container command on the host, /source, /workspace, /coverage-out mapped to $STUB_*. +cat > "$STUB_BIN/podman" <<'EOF' +#!/usr/bin/env bash +case "$1 $2" in + "run "*) + printf '%s\n' "$@" > "$STUB_LOG" + if [[ -n "${STUB_EXEC:-}" ]]; then + cmd=${*: -1} + cmd=${cmd//file:\/\/\/source/file://$STUB_SRC} + cmd=${cmd//\/workspace/$STUB_WS} + cmd=${cmd//\/coverage-out/$STUB_OUT} + HOME="$STUB_HOME" bash -c "$cmd" + exit $? + fi + exit "${STUB_RUN_RC:-0}" ;; + "build "*) + echo "$*" >> "$STUB_LOG.build" ;; + "image inspect") + case "$*" in + *'{{.Id}}'*) echo "${STUB_BASE_ID-base-1}" ;; + *java-profiler.base-id*) echo "${STUB_LABEL:-}" ;; + esac ;; +esac +exit 0 +EOF +# gcovr and java record their arguments; java's `report` writes a minimal jacoco.xml. +cat > "$STUB_BIN/gcovr" <<'EOF' +#!/usr/bin/env bash +printf '%s\n' "$@" > "$STUB_LOG.gcovr" +while [[ $# -gt 0 ]]; do + if [[ "$1" == "-o" ]]; then touch "$2"; fi + shift +done +echo "lines: 50.0% (1 out of 2)" +EOF +cat > "$STUB_BIN/java" <<'EOF' +#!/usr/bin/env bash +printf '%s\n' "$@" >> "$STUB_LOG.java" +if [[ "$3" == "report" ]]; then + while [[ $# -gt 0 ]]; do + if [[ "$1" == "--html" ]]; then mkdir -p "$2"; touch "$2/index.html"; fi + if [[ "$1" == "--xml" ]]; then + printf '%s\n' "${STUB_XML:-}" > "$2" + fi + shift + done +fi +EOF +chmod +x "$STUB_BIN"/* +export PATH="$STUB_BIN:$PATH" +export STUB_LOG="$TEMP_DIR/stub.log" + +# ---- Fake repository with a worktree ---- +# run-containers-tests.sh derives the mounted source from its own location. +REPO="$TEMP_DIR/repo" +mkdir -p "$REPO/utils" "$REPO/gradle" +cp "$UTILS_DIR/run-containers-tests.sh" "$REPO/utils/" +touch "$REPO/gradle/wrapper.properties" +# Gradle stand-in for executed container commands: writes the outputs the export copies. +cat > "$REPO/gradlew" <<'EOF' +#!/usr/bin/env bash +mkdir -p ddprof-lib/build/reports/coverage/cpp ddprof-test/build/jacoco ddprof-lib/build/classes/java/main +echo '{}' > ddprof-lib/build/reports/coverage/cpp/coverage.json +touch ddprof-test/build/jacoco/testCoverage.exec ddprof-lib/build/classes/java/main/A.class +exit "${STUB_GRADLE_RC:-0}" +EOF +chmod +x "$REPO/gradlew" +git init --quiet "$REPO" +git -C "$REPO" -c user.name=T -c user.email=t@t.invalid -c commit.gpgsign=false \ + commit --quiet --allow-empty -m init +git -C "$REPO" add -A +git -C "$REPO" -c user.name=T -c user.email=t@t.invalid -c commit.gpgsign=false \ + commit --quiet -m files +WORKTREE="$TEMP_DIR/wt" +git -C "$REPO" worktree add --quiet --detach "$WORKTREE" +COMMON_DIR=$(git -C "$WORKTREE" rev-parse --path-format=absolute --git-common-dir) + +run_script() { + local root=$1 + shift + rm -f "$STUB_LOG" + "$root/utils/run-containers-tests.sh" --container=podman --arch=aarch64 "$@" 2>&1 +} + +# ---- run-containers-tests.sh: coverage command and export ---- +out_dir="$TEMP_DIR/cell-out" +output=$(run_script "$REPO" --libc=glibc --jdk=21 --coverage-out="$out_dir") +log=$(cat "$STUB_LOG") +expect_contains "coverage task" "$log" "./gradlew -PCI -PenableCoverage :ddprof-lib:coverageReport --no-daemon" +expect_not_contains "coverage skips no gtests" "$log" "-Pskip-gtest" +expect_contains "coverage out mount" "$log" "$out_dir:/coverage-out" +expect_contains "coverage env" "$log" "SANITIZER=coverage" +expect_contains "export reports" "$log" "cp -r ddprof-lib/build/reports/coverage /coverage-out/reports" +expect_contains "export jacoco" "$log" "cp -r ddprof-test/build/jacoco /coverage-out/jacoco" +expect_contains "export classes" "$log" "cp -r ddprof-lib/build/classes/java/main /coverage-out/classes" +expect_contains "gradle exit preserved" "$log" "|| rc=\$?;" +expect_contains "export clears previous data" "$log" "rm -rf /coverage-out/reports /coverage-out/jacoco /coverage-out/classes /coverage-out/cell.properties" +expect_not_contains "plain repo has no common-dir mount" "$log" "$COMMON_DIR:$COMMON_DIR" +expect_contains "plain repo keeps narrow safe.directory" "$log" "safe.directory /source/.git" +[[ -d "$out_dir" ]] && pass || fail "--coverage-out directory was not created" + +output=$(run_script "$REPO" --libc=glibc --jdk=21 --coverage-out="$out_dir" --tests=FooTest) +expect_contains "coverage honours --tests" "$(cat "$STUB_LOG")" ":ddprof-lib:coverageReport -Ptests=FooTest" + +if STUB_RUN_RC=7 run_script "$REPO" --libc=glibc --jdk=21 --coverage-out="$out_dir" >/dev/null; then + fail "container failure exit code was swallowed" +else + pass +fi + +# ---- run-containers-tests.sh: the container command itself ---- +export STUB_SRC="$REPO" STUB_HOME="$TEMP_DIR/home" +mkdir -p "$STUB_HOME" +exec_out="$TEMP_DIR/exec-out" +exec_cell() { + rm -rf "$TEMP_DIR/ws" + STUB_EXEC=1 STUB_WS="$TEMP_DIR/ws" STUB_OUT="$exec_out" run_script "$REPO" --libc=glibc --jdk=21 \ + --coverage-out="$exec_out" "$@" >/dev/null +} +mkdir -p "$exec_out/reports/stale" +rc=0 +STUB_GRADLE_RC=3 exec_cell --tests=FooTest || rc=$? +[[ $rc -eq 3 ]] && pass || fail "cell exit code $rc is not Gradle's 3" +props=$(cat "$exec_out/cell.properties" 2>/dev/null || true) +expect_contains "properties fields" "$props" $'libc=glibc\narch=aarch64\njdk=21\ntests=FooTest\n' +expect_contains "properties commit" "$props" "commit=$(git -C "$REPO" rev-parse HEAD)" +expect_contains "properties gradle exit" "$props" "gradle_exit=3" +[[ -f "$exec_out/reports/cpp/coverage.json" && -f "$exec_out/jacoco/testCoverage.exec" && -f "$exec_out/classes/A.class" ]] \ + && pass || fail "export did not copy reports, jacoco and classes: $(find "$exec_out" | sort)" +[[ ! -e "$exec_out/reports/stale" && ! -e "$exec_out/reports/coverage" ]] \ + && pass || fail "export kept or nested the previous export: $(find "$exec_out" | sort)" +rc=0 +exec_cell || rc=$? +[[ $rc -eq 0 ]] && pass || fail "clean cell exited $rc" +expect_contains "unfiltered tests" "$(cat "$exec_out/cell.properties")" $'tests=\n' +# The export fails when its target vanished (a missing parent fails for root too). +rc=0 +rm -rf "$TEMP_DIR/ws" +STUB_EXEC=1 STUB_WS="$TEMP_DIR/ws" STUB_OUT="$TEMP_DIR/missing/out" run_script "$REPO" \ + --libc=glibc --jdk=21 --coverage-out="$exec_out" >/dev/null || rc=$? +[[ $rc -eq 70 ]] && pass || fail "failed export exited $rc, expected 70" +# A run that fails before the export leaves no earlier export behind. +STUB_RUN_RC=125 run_script "$REPO" --libc=glibc --jdk=21 --coverage-out="$exec_out" >/dev/null || true +[[ ! -e "$exec_out/cell.properties" && ! -e "$exec_out/reports" ]] \ + && pass || fail "a failed run left the previous export: $(find "$exec_out" | sort)" +for bad in 'A$B' 'A"B' "A'B" 'A`B' 'A\B' 'A;B' 'A B' 'A|B' $'A\nB'; do + for mode in --coverage-out="$exec_out" --gtest; do + if output=$(run_script "$REPO" --libc=glibc --jdk=21 "$mode" --tests="$bad"); then + fail "--tests=$bad was accepted ($mode)" + else + expect_contains "--tests=$bad rejected ($mode)" "$output" "--tests may contain only" + fi + done +done +output=$(run_script "$REPO" --libc=glibc --jdk=21 --tests='com.Foo*Test#run,Bar-1_x') +expect_contains "test pattern characters accepted" "$(cat "$STUB_LOG")" "-Ptests=com.Foo*Test#run,Bar-1_x" + +# The non-coverage command is unchanged. +output=$(run_script "$REPO" --libc=glibc --jdk=21) +log=$(cat "$STUB_LOG") +expect_contains "default task" "$log" "./gradlew -PCI -PkeepJFRs :ddprof-test:testDebug -Pskip-gtest" +expect_not_contains "default has no export" "$log" "/coverage-out" + +# ---- run-containers-tests.sh: worktree sources ---- +output=$(run_script "$WORKTREE" --libc=musl --jdk=17 --coverage-out="$out_dir") +log=$(cat "$STUB_LOG") +expect_contains "worktree common-dir mount" "$log" "$COMMON_DIR:$COMMON_DIR:ro" +expect_contains "worktree safe.directory" "$log" "safe.directory \"*\"" + +# ---- run-containers-tests.sh: rejected combinations ---- +for bad in "--shell" "--mount" "--matrix" "--config=debug" "--gtest-task=foo_ut"; do + if output=$(run_script "$REPO" --libc=glibc --jdk=21 --coverage-out="$out_dir" "$bad"); then + fail "--coverage-out with $bad was accepted" + else + expect_contains "--coverage-out with $bad" "$output" "--coverage-out cannot be combined" + fi +done +if output=$(run_script "$REPO" --libc=glibc --jdk=21 --coverage-out=); then + fail "empty --coverage-out was accepted" +else + expect_contains "empty --coverage-out" "$output" "--coverage-out needs a directory" +fi +if run_script "$REPO" --libc=glibc,musl --jdk=21 --coverage-out="$TEMP_DIR/never" >/dev/null; then + fail "--coverage-out with two cells was accepted" +else + [[ ! -e "$TEMP_DIR/never" ]] && pass || fail "rejected run created its --coverage-out directory" +fi +if output=$(run_script "$REPO" --libc=glibc,musl --jdk=21 --coverage-out="$out_dir"); then + fail "--coverage-out with two cells was accepted" +else + expect_contains "--coverage-out multi-cell" "$output" "--coverage-out runs a single cell" +fi +if output=$(run_script "$REPO" --libc=musl --jdk=21-j9 --coverage-out="$out_dir"); then + fail "musl J9 coverage cell was accepted" +else + expect_contains "musl J9 skipped" "$output" "no runnable cells" +fi + +# ---- run-containers-tests.sh: JDK image reuse follows the base image ID ---- +rm -f "$STUB_LOG.build" +STUB_LABEL=base-1 run_script "$REPO" --libc=glibc --jdk=21 >/dev/null +if [[ -f "$STUB_LOG.build" ]]; then + fail "JDK image built on the current base was rebuilt: $(cat "$STUB_LOG.build")" +else + pass +fi +output=$(STUB_LABEL=base-0 run_script "$REPO" --libc=glibc --jdk=21) +expect_contains "stale JDK image rebuilt" "$(cat "$STUB_LOG.build")" "--label java-profiler.base-id=base-1 -t java-profiler-test:glibc-jdk21-aarch64" +expect_contains "stale JDK image message" "$output" "was not built on the current java-profiler-base:glibc-aarch64" +rm -f "$STUB_LOG.build" +STUB_LABEL= run_script "$REPO" --libc=glibc --jdk=21 >/dev/null +expect_contains "unlabeled JDK image rebuilt" "$(cat "$STUB_LOG.build")" "-t java-profiler-test:glibc-jdk21-aarch64" + +rm -f "$STUB_LOG.build" +STUB_LABEL=base-1 run_script "$REPO" --libc=glibc --jdk=21 --rebuild-base >/dev/null +expect_contains "--rebuild-base rebuilds the JDK image" "$(cat "$STUB_LOG.build" 2>/dev/null)" "-t java-profiler-test:glibc-jdk21-aarch64" +rm -f "$STUB_LOG.build" +STUB_BASE_ID= STUB_LABEL= run_script "$REPO" --libc=glibc --jdk=21 >/dev/null +expect_contains "unknown base ID rebuilds the JDK image" "$(cat "$STUB_LOG.build" 2>/dev/null)" "-t java-profiler-test:glibc-jdk21-aarch64" + +# ---- run-containers-tests.sh: CI-matched cells the test-coverage skill runs ---- +output=$(run_script "$REPO" --libc=glibc --jdk=21-graal --coverage-out="$out_dir") +expect_contains "graal coverage image" "$(cat "$STUB_LOG")" "java-profiler-test:glibc-jdk21-graal-aarch64" +if output=$(run_script "$REPO" --libc=glibc --jdk=8 --coverage-out="$out_dir"); then + fail "glibc aarch64 HotSpot 8 coverage cell was accepted" +else + expect_contains "glibc aarch64 HotSpot 8 skipped" "$output" "no runnable cells" +fi + +# ---- merge-coverage.sh ---- +MERGE="$UTILS_DIR/merge-coverage.sh" +SRC="$TEMP_DIR/src" +mkdir -p "$SRC/ddprof-lib/src/main/cpp" "$SRC/ddprof-lib/src/main/java" +git init --quiet "$SRC" +git -C "$SRC" -c user.name=T -c user.email=t@t.invalid -c commit.gpgsign=false \ + commit --quiet --allow-empty -m src +SRC_COMMIT=$(git -C "$SRC" rev-parse HEAD) + +# Usage: make_cell [classes-content] +make_cell() { + mkdir -p "$1/reports/cpp" "$1/jacoco/children" "$1/classes" + printf 'libc=glibc\narch=aarch64\njdk=%s\ncommit=%s\ngradle_exit=0\n' "$3" "$2" > "$1/cell.properties" + echo '{}' > "$1/reports/cpp/coverage.json" + touch "$1/jacoco/testCoverage.exec" "$1/jacoco/children/child-1.exec" + echo "${4:-same}" > "$1/classes/A.class" +} +make_cell "$TEMP_DIR/c1" "$SRC_COMMIT" 21 +make_cell "$TEMP_DIR/c2" "$SRC_COMMIT" 25-graal +export JACOCO_CLI="$TEMP_DIR/jacococli.jar" +touch "$JACOCO_CLI" + +rm -f "$STUB_LOG.gcovr" "$STUB_LOG.java" +output=$("$MERGE" --source-root="$SRC" --out="$TEMP_DIR/merged" "$TEMP_DIR/c1" "$TEMP_DIR/c2" 2>&1) +gcovr_args=$(cat "$STUB_LOG.gcovr") +java_args=$(cat "$STUB_LOG.java") +expect_contains "gcovr tracefile 1" "$gcovr_args" "$TEMP_DIR/c1/reports/cpp/coverage.json" +expect_contains "gcovr tracefile 2" "$gcovr_args" "$TEMP_DIR/c2/reports/cpp/coverage.json" +expect_contains "gcovr root" "$gcovr_args" "$SRC/ddprof-lib/src/main/cpp" +expect_contains "gcovr merge mode" "$gcovr_args" "--merge-mode-functions=separate" +expect_contains "jacoco merges child exec" "$java_args" "$TEMP_DIR/c2/jacoco/children/child-1.exec" +expect_contains "jacoco merges task exec" "$java_args" "$TEMP_DIR/c1/jacoco/testCoverage.exec" +expect_contains "jacoco report classes" "$java_args" "$TEMP_DIR/c1/classes" +summary=$(cat "$TEMP_DIR/merged/summary.md") +expect_contains "summary cells" "$summary" "| glibc | aarch64 | 25-graal | | 0 | yes | yes |" +expect_not_contains "unfiltered run has no banner" "$summary" "Filtered run" +expect_contains "summary cpp" "$summary" "lines: 50.0% (1 out of 2)" +expect_contains "summary java" "$summary" "line: 75.0% (3 out of 4)" +expect_not_contains "identical classes do not warn" "$output" "differ from" + +make_cell "$TEMP_DIR/c3" "$SRC_COMMIT" 17 other +output=$("$MERGE" --source-root="$SRC" --out="$TEMP_DIR/merged3" "$TEMP_DIR/c1" "$TEMP_DIR/c3" 2>&1) +expect_contains "differing classes warn" "$output" "classes in $TEMP_DIR/c3 differ" +expect_contains "differing classes in summary" "$(cat "$TEMP_DIR/merged3/summary.md")" "| 17 | | 0 | yes | partial (classes differ) |" + +rm -f "$TEMP_DIR/c3/reports/cpp/coverage.json" +output=$("$MERGE" --source-root="$SRC" --out="$TEMP_DIR/merged4" "$TEMP_DIR/c1" "$TEMP_DIR/c3" 2>&1) +expect_contains "missing tracefile warns" "$output" "$TEMP_DIR/c3 has no C++ tracefile" +expect_contains "missing tracefile in summary" "$(cat "$TEMP_DIR/merged4/summary.md")" "| 0 | no (no tracefile) |" + +make_cell "$TEMP_DIR/c4" "0000000000000000000000000000000000000000" 11 +if output=$("$MERGE" --source-root="$SRC" --out="$TEMP_DIR/m" "$TEMP_DIR/c1" "$TEMP_DIR/c4" 2>&1); then + fail "cells from different commits were merged" +else + expect_contains "commit mismatch" "$output" "other cells from $SRC_COMMIT" +fi +if output=$("$MERGE" --source-root="$SRC" --out="$TEMP_DIR/m" "$TEMP_DIR/c4" 2>&1); then + fail "source root at another commit was accepted" +else + expect_contains "source mismatch" "$output" "--source-root is at $SRC_COMMIT" +fi +mkdir -p "$TEMP_DIR/empty" +if output=$("$MERGE" --source-root="$SRC" --out="$TEMP_DIR/m" "$TEMP_DIR/empty" 2>&1); then + fail "cell without cell.properties was accepted" +else + expect_contains "missing properties" "$output" "has no cell.properties" +fi +if output=$("$MERGE" --source-root="$TEMP_DIR" --out="$TEMP_DIR/m" "$TEMP_DIR/c1" 2>&1); then + fail "non-checkout source root was accepted" +else + expect_contains "bad source root" "$output" "is not a java-profiler checkout" +fi + +# Filtered cells, stale output, empty commit, zero counters, zero cells. +make_cell "$TEMP_DIR/c5" "$SRC_COMMIT" 21 +echo "tests=FooTest" >> "$TEMP_DIR/c5/cell.properties" +mkdir -p "$TEMP_DIR/merged5/cpp" +touch "$TEMP_DIR/merged5/cpp/stale.html" "$TEMP_DIR/merged5/.merge-coverage-output" +"$MERGE" --source-root="$SRC" --out="$TEMP_DIR/merged5" "$TEMP_DIR/c5" >/dev/null 2>&1 +summary=$(cat "$TEMP_DIR/merged5/summary.md") +expect_contains "filtered banner" "$summary" "Filtered run" +expect_contains "filter column" "$summary" "| 21 | FooTest |" +[[ ! -e "$TEMP_DIR/merged5/cpp/stale.html" ]] && pass || fail "--out kept an earlier merge's report" + +# Java column: no .exec files, and .exec files but no classes in any cell. +make_cell "$TEMP_DIR/c9" "$SRC_COMMIT" 11 +rm -f "$TEMP_DIR/c9/jacoco/testCoverage.exec" "$TEMP_DIR/c9/jacoco/children/child-1.exec" +"$MERGE" --source-root="$SRC" --out="$TEMP_DIR/merged9" "$TEMP_DIR/c1" "$TEMP_DIR/c9" >/dev/null 2>&1 +expect_contains "no exec column" "$(cat "$TEMP_DIR/merged9/summary.md")" "| 11 | | 0 | yes | no (no JaCoCo data) |" +rm -rf "$TEMP_DIR/c10" +make_cell "$TEMP_DIR/c10" "$SRC_COMMIT" 8 +rm -rf "$TEMP_DIR/c10/classes" +"$MERGE" --source-root="$SRC" --out="$TEMP_DIR/merged10" "$TEMP_DIR/c10" >/dev/null 2>&1 +summary=$(cat "$TEMP_DIR/merged10/summary.md") +expect_contains "no classes column" "$summary" "| 8 | | 0 | yes | no (no classes in any cell) |" +expect_contains "no classes summary" "$summary" "(no cell exported classes)" + +# --out must be empty or an earlier merge output. +mkdir -p "$TEMP_DIR/notmerge/cpp" +touch "$TEMP_DIR/notmerge/cpp/keep.cpp" +if output=$("$MERGE" --source-root="$SRC" --out="$TEMP_DIR/notmerge" "$TEMP_DIR/c1" 2>&1); then + fail "a non-merge --out was accepted" +else + expect_contains "non-merge out" "$output" "neither empty nor an earlier merge output" +fi +[[ -f "$TEMP_DIR/notmerge/cpp/keep.cpp" ]] && pass || fail "a rejected --out lost its files" + +make_cell "$TEMP_DIR/c6" "" 21 +if output=$("$MERGE" --source-root="$SRC" --out="$TEMP_DIR/m" "$TEMP_DIR/c6" "$TEMP_DIR/c1" 2>&1); then + fail "cell without a commit was merged" +else + expect_contains "empty commit" "$output" "$TEMP_DIR/c6 has no commit in cell.properties" +fi + +if output=$("$MERGE" --source-root="$SRC" --out="$TEMP_DIR/m" 2>&1); then + fail "merge without cells was accepted" +else + expect_contains "no cells" "$output" "no cell directories given" +fi + +STUB_XML='' \ + "$MERGE" --source-root="$SRC" --out="$TEMP_DIR/merged7" "$TEMP_DIR/c1" >/dev/null 2>&1 +expect_contains "zero-total counter" "$(cat "$TEMP_DIR/merged7/summary.md")" "branch: 0.0% (0 out of 0)" + +# JaCoCo CLI download: the jar must match the pinned SHA-256. +cat > "$STUB_BIN/curl" <<'EOF' +#!/usr/bin/env bash +while [[ $# -gt 0 ]]; do + if [[ "$1" == "-o" ]]; then printf '%s' "${STUB_JAR:-jar}" > "$2"; fi + shift +done +EOF +chmod +x "$STUB_BIN/curl" +pinned=$(sed -n 's/^JACOCO_CLI_SHA256="\(.*\)"$/\1/p' "$MERGE") +if output=$(env -u JACOCO_CLI "$MERGE" --source-root="$SRC" --out="$TEMP_DIR/merged8" "$TEMP_DIR/c1" 2>&1); then + fail "JaCoCo CLI with a wrong checksum was used" +else + expect_contains "checksum mismatch" "$output" "SHA-256 mismatch" + [[ ! -e "$TEMP_DIR/merged8/tools/org.jacoco.cli-0.8.14-nodeps.jar" ]] && pass || fail "mismatched jar was kept" +fi +[[ ${#pinned} -eq 64 ]] && pass || fail "merge-coverage.sh has no pinned SHA-256" +# A merge that failed midway (merged8, checksum mismatch) can be rerun into the same --out. +if "$MERGE" --source-root="$SRC" --out="$TEMP_DIR/merged8" "$TEMP_DIR/c1" >/dev/null 2>&1; then + pass +else + fail "rerun into a failed merge's --out was rejected" +fi +summary=$(cat "$TEMP_DIR/merged8/summary.md" 2>/dev/null || true) +expect_contains "cpp link when written" "$summary" "- C++ report:" +expect_contains "java link when written" "$summary" "- Java report:" +expect_not_contains "no java link without report" "$(cat "$TEMP_DIR/merged10/summary.md")" "- Java report:" +merge_version=$(sed -n 's/^JACOCO_VERSION="\(.*\)"$/\1/p' "$MERGE") +build_version=$(sed -n 's/^ *toolVersion = "\(.*\)"$/\1/p' "$UTILS_DIR/../ddprof-test/build.gradle.kts") +[[ -n "$merge_version" && "$merge_version" == "$build_version" ]] \ + && pass || fail "JACOCO_VERSION $merge_version differs from ddprof-test toolVersion $build_version" + +echo "$TESTS tests, $FAILED failed" +[[ $FAILED -eq 0 ]]