Updater: one pending update at a time, and the version route reads CloudFormation - #480

Summary

The updater could leave two prepared updates open at once. It reported the stored entry instead of what CloudFormation said. It could keep a model check "pending" forever. Now there is only ever one pending update, the version route reconciles with CloudFormation before it answers, and the model check is pending only while the release check really runs. The dashboard pin moves to the dashboard PR that renders this.

The research note (one primary status, honest progress, truth from the system doing the work, a time bound on stale states, saying what needs the admin) is in the dashboard PR.

Changes

• lambda/updater/handler.py: one pending update at a time
  • begin refuses to replace an update whose change set CloudFormation is still making or already running, and says why.
  • After the new change set exists, supersede_open marks every older prepared-but-unapplied update superseded (with superseded_by) and deletes its change set if it never ran.
  • A prepared update that hasn't run no longer counts as "an update is already running".
• lambda/updater/handler.py: reconcile, run by the version route before it answers
  • The newest open update becomes the pending one (pointers left by older updaters are fixed).
  • Older open updates are superseded, then native_poll runs.
  • Preparing, waiting for the OK, applying and the result now show at once, instead of up to 5 minutes later. That includes a change set executed in the AWS console or CLI, which is recorded as applied_by: someone in the AWS console.
• native_poll
  • It never writes over a newer state.
  • A change set that was just executed doesn't go back to waiting for the OK while CloudFormation still answers AVAILABLE.
  • For a console execution, failures are read from when the change set was made.
  • On UPDATE_COMPLETE it checks the stack's Version output before calling the update a success.
• native_status adds who started the update, when, and who applied it.
• history_view shows any row still open that isn't the pending one as superseded.
• Model check
  • model_check_view re-runs a pending check whose release has been verified since.
  • It marks a check stalled, with its reason, when the release check build ended without reporting back or the check is older than 30 minutes.
  • Pending carries its progress (step, start time, build status).
  • A failed release check stores a stalled record with its reason.
  • Stalled still holds the update.
  • start_verify restarts a build that ended without reporting, without waiting 30 minutes.
  • update_now stores the check that passed.
• services/gateway/src/routes/setup_checks.rs: a stalled check is a warning in Setup and a model_check_stalled blocker in the preflight.
• selfhost/versions.json: dashboard pin bumped with scripts/bump-dashboard.sh.

Verification

• python3 -m unittest discover -s lambda/tests: 330 passed.
  • New OnePendingUpdate tests cover:
    • a newer prepared update supersedes the older one
    • the observed state (an older pending pointer while a newer update is open) is reconciled to the newer one
    • history never shows two updates waiting
    • an update still being prepared, or already executing, is never replaced
    • preparing turns into waiting for the OK with no 5-minute wait
    • executed in the console reads as applying, then updated (Version output checked)
    • executed in the console and rolled back says why
    • a stack reporting a different release is not a success
    • a just-applied update stays applying
    • a poll never writes over a newer state
  • New ModelCheckTruth tests cover:
    • pending with its progress
    • a verified release is never still "checking"
    • a build that ended without a report is stalled and restarts
    • the 30-minute bound
    • a failed release check holds the update and says why
• cargo fmt --check --all passes.
• cargo clippy --workspace -- -D warnings passes.
• cargo test -p gateway setup_checks: 28 passed, including the new stalled assertions.