Skip to content

ci: build, test and stage the release image by digest - #690

Open
SushaanthSrinivasan wants to merge 1 commit into
ci/release-workflowfrom
ci/release-stage
Open

SushaanthSrinivasan wants to merge 1 commit into
ci/release-workflowfrom
ci/release-stage

Conversation

@SushaanthSrinivasan

Copy link
Copy Markdown

Description

Why is this change being made?

  1. Next step towards an automated release. ci: add a release version check and pin integ tests to an image digest #689 checks the version. This PR builds the image once, runs the integ tests on exactly that image, and stages that same image, so what was tested is what gets released. Nothing is published to ECR Public yet.

What is changing?

  1. release.yml gets four more jobs, after validate-version:
    • build: builds once for linux/amd64 + linux/arm64 (arm64 under QEMU) and pushes it to the private staging repo as <version>-testing. Then it checks what it pushed: the digest matches the manifest bytes, the index has exactly those two platforms, and both binaries carry the version (go version -m on a local copy written by the same build).
    • check-digest: fails the run if build produced no digest, so integ can never fall back to a tag.
    • integ: calls integ.yml with that digest.
    • stage: adds the <version> tag to that same digest in the staging repo. Nothing is rebuilt or copied.
  2. Practice runs. A new practice checkbox, ticked by default. A practice run uses the version already on main, tags rehearsal-<run id>-testing / rehearsal-<run id>, and those images are deleted after 14 days. The run name says (practice). The version check skips its git tag and ECR Public checks in a practice run.
  3. Every run gets its own digest. The build adds labels with the commit, the run id + attempt and the mode. A practice image and a real release can never be the same image, so cleaning up practice images can't delete a release. As a backup, build and stage refuse a digest that already has a tag of the other mode.
  4. AWS credentials reach steps only through step outputs, never the job environment; a step checks that. Tool setup runs before any credentials exist. The registry login is removed right after the build.
  5. Forks: build, integ and stage only run in this repo. A fork run shows green with those jobs skipped.
  6. New scripts and tests in .github/scripts/: release-tags.sh (tags, build args, labels), staging-image.sh (the staging repo steps), practice mode in validate-release-version.sh. New bats tests with fake aws, docker and go. The go version -m output, the AWS CLI error format and the public 3.1.4 index used by the tests are recorded from real output.
  7. New pins: docker/setup-qemu-action v4.4.0, docker/setup-buildx-action v4.4.1, buildx v0.38.0, and the BuildKit and binfmt images by digest.

Accepted risks

  1. A real run is one untick away. Anyone who can start the workflow can untick "practice", which writes a permanent version tag in the staging repo. Mitigated by the default, the run name, and review of release.yml. Nothing is public at this stage.
  2. The Dockerfile's base images are pinned by tag, not digest, so two builds of one commit can differ. Changing that affects every build; follow-up.
  3. A real candidate that fails integ keeps its <version>-testing tag. A retry overwrites it, so at most one leftover per version.
  4. The staging account ID shows in run logs. Account IDs aren't secret, and masking them would make GitHub drop the job outputs.
  5. Integ has no time limit of its own (GitHub's 6-hour default), so a stuck test can hold the release queue. Cancelling the run frees it.
  6. Images carry build labels (run id, mode, commit). Not secret; useful for tracing.
  7. Anonymous Docker Hub pulls (BuildKit and binfmt images) can hit a rate limit; re-run the job.

Re-run buttons

  • Re-run failed jobs, after build succeeded (integ or stage failed): reuses build's digest; no rebuild.
  • Re-run failed jobs, after build failed: builds again; the new digest replaces the -testing tag.
  • Re-run the stage job alone after it succeeded: "already staged", success.
  • Re-run all jobs before stage ran: rebuilds and tests the new digest. Expected.
  • Re-run all jobs after stage ran: fails at "Check the tag is free" on purpose. Start a new run instead.
  • "Check the digest" failed: use "Re-run all jobs".
  • Practice runs older than 14 days: their images are gone; start a new practice run.

Before merge

  1. Rebase on ci: harden GitHub Actions workflows #685 (it declares the secrets integ.yml needs; this PR passes them by name) and on ci: add a release version check and pin integ tests to an image digest #689's final state.
  2. The staging repo's cleanup rules (untagged and rehearsal-* images expire; staged releases never do) and the staging environment's settings are in place.

Related Links


Testing

How was this tested?

  1. ShellCheck: clean.
  2. zizmor 1.30.1: release.yml is clean.
  3. Pins: every pinned SHA matches its tag.
  4. The bats tests ran locally with a small bats stand-in (bats-core isn't installed on my machine): all pass. This PR's CI is the first run on real bats-core.
  5. Not run yet: release.yml only runs from main, so it's tried after merge with practice runs (ticked box), then checked by hand: tags, platforms, binary versions, labels, re-running stage, and two runs queuing one after the other.

When testing locally, provide testing artifact(s):

  1. See above.

Reviewee Checklist

Update the checklist after submitting the PR

  • I have reviewed, tested and understand all changes
    If not, why:
  • I have filled out the Description and Testing sections above
    If not, why:
  • Build and Unit tests are passing
    If not, why: waiting on CI for this PR
  • Unit test coverage check is passing
    If not, why: no Go code changes
  • Integration tests pass locally
    If not, why: the release workflow runs only from main; tried with practice runs after merge
  • I have updated integration tests (if needed)
    If not, why:
  • I have ensured no sensitive information is leaking (i.e., no logging of sensitive fields, or otherwise)
    If not, why:
  • I have added explanatory comments for complex logic, new classes/methods and new tests
    If not, why:
  • I have updated README/documentation (if needed)
    If not, why: the release steps change once publishing is added
  • I have clearly called out breaking changes (if any)
    If not, why: none: the new jobs only run when the Release workflow is started by hand

Reviewer Checklist

All reviewers please ensure the following are true before reviewing:

  • Reviewee checklist has been accurately filled out
  • Code changes align with stated purpose in description
  • Test coverage adequately validates the changes

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Extend the Release workflow: after the version check, build the image
once for linux/amd64 and linux/arm64, push it to the private staging
repo, run the integration tests on that exact digest, then add the
release tag to that same digest. Nothing is rebuilt, copied or
published to ECR Public.

A "practice run" checkbox, ticked by default, uses the version already
on main and tags rehearsal-<run id> instead, so the whole flow can be
tried without a version bump. The version check skips its git tag and
ECR Public checks in a practice run.

The build checks what it pushed before anything else uses it: the
index digest matches its manifest bytes, it has exactly the two
platforms, both binaries carry the version, and the digest has no tag
of the other mode. Build labels (commit, run id and attempt, mode) give
every run its own digest. AWS credentials reach steps only through
step outputs, never the job environment.
@codecov

codecov Bot commented Oct 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.62%. Comparing base (47fd57f) to head (1122614).

Additional details and impacted files
@@                 Coverage Diff                  @@
##           ci/release-workflow     #690   +/-   ##
====================================================
  Coverage                61.62%   61.62%           
====================================================
  Files                       11       11           
  Lines                      740      740           
====================================================
  Hits                       456      456           
  Misses                     265      265           
  Partials                    19       19           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@SushaanthSrinivasan
SushaanthSrinivasan marked this pull request as ready for review October 16, 2026 18:48
@SushaanthSrinivasan
SushaanthSrinivasan requested a review from a team as a code owner October 16, 2026 18:48
integ:
name: Integration tests
needs: [build, check-digest]
uses: $/.github/workflows/integ.yml

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reusable workflow should start with ./ instead of $/.? double check

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its the new self repository syntax, github recommends $/ over ./ for same repo actions and reusable workflows
https://github.blog/changelog/2026-07-30-reference-same-repository-actions-with-self-repository-syntax/

This branch is waiting to be deployed

1 waiting deployment
code-review — 11226143 Waiting Oct 16, 2026 by SushaanthSrinivasan via review #108
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants