Skip to content

Self-host releases: only CoderHelm's release workflow can ship an update - #310

Merged
nambok merged 1 commit into
mainfrom
fix/release-supply-chain
Oct 26, 2026
Merged

nambok merged 1 commit into
mainfrom
fix/release-supply-chain

Conversation

@nambok

@nambok nambok commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR makes the self-host release and update chain refuse anything that only looks like a CoderHelm release. Before anything is installed, the installer, the update build, the Launch Stack installer and install.sh now require a signature made by exactly this repository's release.yml, on that release's own tag, in the Rekor log. The updater never moves an install backwards, except an admin rollback to a version that install already ran. Publishing a release now needs a signed tag on main and a reviewer's approval. Every action in every workflow is pinned by commit SHA, and CI enforces the pinning.

Changes

Verification (identity pinned, fails closed)

  • installer/lib/release.mjs, installer/lib/sigstore.mjs:
    • cosign runs with these exact values:
      • --certificate-identity https://github.com/CoderHelm/coderhelm-platform/.github/workflows/release.yml@refs/tags/<tag>
      • the GitHub Actions OIDC issuer
      • --certificate-github-workflow-repository and --certificate-github-workflow-ref
    • Before cosign runs, the installer reads the bundle's certificate itself. It checks the SAN, the issuer, the repository, the ref and the build signer, and it requires a Rekor entry.
    • The fallback is gh attestation verify with --signer-workflow, --source-ref, --cert-oidc-issuer and --deny-self-hosted-runners, run for every asset.
    • The release is refused when there is no bundle, when neither cosign nor gh is installed, or on any mismatch. No flag skips this. --require-signature is still accepted, as a no-op.
    • The signature is now checked before any tarball is unpacked. Before, it was checked after.
  • infra/lib/release-verify.ts is a new shared module, used by the update buildspec. The Launch Stack buildspec repeats the same lines, and a test keeps the two identical.
    • cosign is pinned by its SHA-256. Before, it was only checked against the checksums file published next to it.
    • $VERSION must match vX.Y.Z.
    • curl is HTTPS-only, including redirects.
    • The identity is exact, not a regexp.
    • The update build runs npm install --ignore-scripts.
  • selfhost/install.sh resolves latest to a tag, verifies the signature with cosign or gh, and stops if neither tool is installed.
  • selfhost/launch/bootstrap.mjs refuses a VERSION that isn't a release tag.

No downgrade or replay

  • Installer: rollback and update --rollback only go to a version this install ran before. state.versions is now recorded on every deploy.
  • Updater:
    • update_now only accepts a strictly newer version.
    • rollback only accepts a version in the install's ran_versions or history.
    • The release list is read only from api.github.com for this repository.
    • The status action has a new releases freshness field. It is marked stale when the list stops showing the newest version seen before, or when the list hasn't been read for 3 days.
  • The signature is bound to the tag, so an old release's signature can't stand in for a new tag.

Release workflow (release.yml)

  • check job: the run must be on a tag, the tag must be annotated and signed, GitHub must report it verified, its commit must be signed, and the commit must be an ancestor of main.
  • build job: contents: read, no secrets, no OIDC, and no rust-cache, so a cached build can't be poisoned. It uses cargo --locked and pins cargo-zigbuild==0.24.0.
  • publish job:
    • Runs with environment: release and does no checkout.
    • Re-checks the checksums, then signs with cosign v2.4.1, the same version the verifiers use.
    • Verifies its own signature with the exact identity before publishing.
    • Attests every file, install.sh included.
    • Never replaces a published release's files.
    • Uploads the same signed launch-stack.yaml to S3 with --if-none-match.
    • latest.json now carries templateSha256 and only ever moves forward.
  • ReleasePublisherRole now trusts only repo:CoderHelm/coderhelm-platform:environment:release, with StringEquals.
  • Workflow-level permissions: {}, with least privilege set per job.

CI

  • Every uses: in all 5 workflows is pinned by full SHA with a # vX.Y.Z comment. Majors are unchanged.
  • New Workflow security CI job:
    • .github/scripts/check_workflows.py fails on any unpinned action, on pull_request_target or workflow_run triggers, on github.event.*, head_ref or inputs.* used inside run:, and on jobs with no permissions.
    • It also runs actionlint, downloaded at a pinned version and checked by hash.
  • ci.yml now has permissions: contents: read. The deploy.yml build job also gets contents: read.
  • .github/dependabot.yml keeps the pinned actions current.

Settings and docs

  • scripts/release-protection.sh applies the repository settings. It can't be run from CI, so Nam runs it. It sets:
    • the tag rulesets
    • the main ruleset
    • the release environment
    • immutable releases
    • Actions settings: SHA pinning required, a read-only token, and approval for outside PRs
    • Dependabot
  • docs/self-host/updates-security.md covers who can publish, what installs check, the Launch Stack, how to check a release yourself (cosign verify-blob and gh attestation verify), and what happens when a check fails. updates.md and security.md are updated to match.

Before the next release

  1. Run scripts/release-protection.sh.
  2. Deploy the hosted worker stack, which carries the new publisher role trust.
  3. Push tags with git tag -s.

Verification

  • Installer: 90 pass. New test/release-security.test.mjs (21 tests) covers:
    • tampered SHA256SUMS
    • valid-looking signatures from another repository, another workflow, a branch, another tag and another issuer (cosign is never called)
    • a bundle with no Rekor entry
    • a missing or broken bundle
    • a file changed after signing (nothing is unpacked)
    • a redirect off GitHub, and http or other hosts
    • no cosign or gh
    • the gh fallback
    • cached releases re-verified
    • rollback limited to versions the install ran
    • a real GitHub Actions Sigstore bundle (cli/cli provenance), parsed and refused for the wrong identity
  • Launch Stack: 27 pass.
  • Python: 110 pass, including the new updater NoDowngrade tests (no backwards or sideways update_now; rollback only to ran versions; stale list reported; other repositories refused). The .github/scripts suite also passes (27 tests, including the workflow checker run against this repository's own workflows).
  • Infra: 60 pass. Tests cover the buildspec's exact identity and order, the pinned cosign, HTTPS-only curl, and the publisher trust on the environment.
  • Rust: cargo fmt --check, clippy -D warnings, and cargo test --workspace (2123 passed, 0 failed).
  • cfn-lint, actionlint (all workflows), shellcheck on install.sh and release-protection.sh: clean.
  • The release workflow itself can only run on a real tag. It has not been run.

Installs, updates and the Launch Stack now refuse anything that merely
looks like a CoderHelm release.

- Signature identity is exact: release.yml on refs/tags/<version>, GitHub's
  OIDC issuer, workflow repository and ref, and a Rekor entry. The installer
  also reads the certificate itself before cosign runs. No signature, no
  cosign or gh, or any mismatch: nothing is installed. No skip flag.
- No downgrade: updates only go forward; rollbacks only to versions the
  install ran before (installer and updater). The updater reads releases
  only from api.github.com for this repo and reports a stale list.
- Downloads only from this repo's GitHub release URLs over HTTPS (redirects
  included); cosign in CodeBuild pinned by SHA-256; VERSION must be a tag.
- release.yml: check (signed annotated tag on main), build (read-only, no
  secrets), publish (release environment with reviewers; signs, verifies,
  attests; never replaces a release or an S3 template; latest.json only
  moves forward). The AWS publisher role trusts only that environment.
- Every action in every workflow pinned by commit SHA; CI fails otherwise
  (check_workflows.py), plus actionlint and Dependabot for actions.
- scripts/release-protection.sh applies the tag rulesets, main protection,
  release environment, immutable releases and Actions settings.
- docs/self-host/updates-security.md: what is checked and how to verify a
  release yourself.
@nambok
nambok merged commit be78397 into main Oct 26, 2026
8 checks passed
@nambok
nambok deleted the fix/release-supply-chain branch October 26, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant