Skip to content

Admin teams: count runs and PRs reviewed once each, from where they are written - #474

Merged
nambok merged 1 commit into
mainfrom
admin-accurate-counts
Nov 4, 2026
Merged

nambok merged 1 commit into
mainfrom
admin-accurate-counts

Conversation

@nambok

@nambok nambok commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

The admin Teams page showed "46 runs · 0 reviews" for an organisation with many reviewed pull requests, and 0 reviews for every team. It read reviews from the analytics items, and that counter was never a count of reviews: it was added only when a review job's model usage reached the worker (in record_review_usage, next to the cost write), counted answers to questions and jobs that stopped before posting, and skipped carried approvals. Plugin runs were listed with the team's runs but never added to total_runs, and a redelivered run message counted its run twice.

Each count is now added exactly once, where its record is written, and the page reads those counts. A "review" on the page is a pull request the reviewer posted a review on, counted once in all and once per month: the rule the weekly email counts by ("PRs reviewed"). Paired with CoderHelm/coderhelm#315 (labels "in all" and "this month"), pinned here.

Changes

  • common::team_counts (new): add_once writes a once-only marker (COUNTED#<team> partition of the analytics table, never seen by period reads) and the ADD on the month and ALL_TIME items in one transaction; a marker already there cancels it, so replays and the backfill never add twice. counts_as_review is the one rule (posted, not an answer to a question), now also used by the weekly email and the analytics page.
  • Runs (total_runs): agent runs when create_run_record writes the record (once per run id); plugin runs on any write of their record through RunsTable (gateway and worker), plus completed / failed once they end.
  • Reviews (prs_reviewed): store_review_record, which every stored review goes through (full reviews, replies, carried approvals), counts the PR. The old ADD reviews and its calls on paths that never posted are gone; record_review_usage only stores the cost on the review record.
  • Admin (routes/admin.rs): Activity::from_items reads total_runs and prs_reviewed from ALL_TIME and this month's item: two GetItems per team, as before. API field names are unchanged (reviews_total / reviews_month now mean PRs reviewed).
  • scripts/team_counts_backfill.py (not run): adds the PRs reviewed (from RSTAT# / REVIEW# records) and plugin runs from before the deploy through the same markers, so whatever the services already counted is skipped. Dry run by default; --apply writes.
  • selfhost/versions.json: dashboard pinned to dbfbd17 (Admin teams: runs and PRs reviewed, in all and this month coderhelm#315).
  • Tests: admin::tests::the_counts_match_the_team_s_runs_and_reviews (a fixture team's runs and reviews, including a redelivered run, a plugin run written three times, a question, an unposted re-review and the same PR in two months, go through the counting code; the page's counts equal what the records say); team_counts::tests::every_run_kind_and_every_stored_review_is_counted (fails if a runs-table writer, the plugin run log or a stored review stops counting, or anything else adds to these counters; checked by removing each call); once-only and per-month tests for runs, plugin runs and PRs.

Verification

  • cargo fmt --all --check, cargo clippy -p common -p gateway -p worker --all-targets -D warnings: clean.
  • cargo test -p common -p gateway -p worker: 1,418 + 724 + 1,192 passed, 0 failed.
  • Backfill: its planning functions checked on a fixture (the same PR in two months, a question, an unposted re-review, a plugin run); --help runs. Not run against any table.
  • After the deploy: python3 scripts/team_counts_backfill.py --stage prod (dry run), then with --apply.

…re written

The admin Teams page read each team's runs and reviews from its analytics
items. Reviews were never really counted there: the `reviews` counter was
added only when a review job's model usage reached the worker, counted
answers to questions and jobs that stopped before posting, skipped carried
approvals, and shows 0 for every team. Plugin runs were listed with the
team's runs but never added to `total_runs`, and a redelivered run message
counted its run twice.

common::team_counts now adds each count exactly once: one transaction
writes a once-only marker (`COUNTED#<team>` partition of the analytics
table) and the ADD on the month and ALL_TIME items.

- Runs: agent runs when their record is created, plugin runs on any write
  of their record (and `completed` / `failed` once they end).
- PRs reviewed (`prs_reviewed`): every stored review (store_review_record)
  counts its pull request once per month and once in all, when it was
  posted and isn't an answer to a question: the rule the weekly email and
  the analytics page now share (counts_as_review).
- The admin page reads `total_runs` and `prs_reviewed` (Activity), and the
  dashboard labels say "in all" and "this month" (pinned here).
- scripts/team_counts_backfill.py adds the PRs reviewed and plugin runs from
  before the deploy through the same markers. Dry run by default; not run.
@nambok
nambok merged commit 0c63bb1 into main Nov 4, 2026
10 checks passed
@nambok
nambok deleted the admin-accurate-counts branch November 4, 2026 23:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant