feat(cache-warming): Keep the prompt cache of idle Conversations warm - #2485

Many prompt cache writes come from people who reply just after the 5 minute Anthropic cache TTL ends, so the next turn writes the full history again. This ports the cache warmer from BYK/loreai (packages/gateway/src/cache-warmer.ts). It is on for every app. JUNIOR_CACHE_WARMING_ENABLED=false is an operational kill switch, like JUNIOR_CONVERSATION_WORK_ENABLED: it defaults to true, stops storing requests, and drops warmups that are already queued.

How it works:

• When a turn ends (stopReason: "stop") on an anthropic/ model with a cached prefix of at least 50K tokens, Junior stores the exact provider context (gzip, max 1 MB) in the StateAdapter and sends two delayed, signed queue messages on a new junior_cache_warmups topic: the first warmup, and a horizon message 55 minutes later.
• Just before the TTL ends, a warmup replays the same system prompt, tools, messages, and reasoning level through completeText with maxTokens: 1. The cache read refreshes the TTL.
• Each next warmup is decided by Lore's model: a histogram of reply gaps (per Conversation, blended with global gaps) gives the chance of a reply in the next TTL window. That chance must exceed Lore's rising threshold k · read / (write − read), so warming stops by itself when it can no longer pay off. Warmups also stop at the 55 minute horizon.
• A new model call changes the generation, so old messages do nothing.

Cost guards:

• Only anthropic/ models. OpenAI and xAI models also use the Anthropic Messages API on the gateway, but cache differently.
• No warming for models with budget thinking, because the budget is added to the one-token output limit. Adaptive thinking (Opus 5.5) keeps max_tokens: 1.
• Every idle period of an eligible Conversation records one gap. The horizon message records "no reply" when nobody replied in 55 minutes. Without this, only replies would be counted and the model would overrate the reply chance (Conversations that never get a reply would look like quick replies).
• Breaker: a warmup that reads less than 90% of the expected prefix, or that the provider rejects, ends the idle period and counts as a miss. 3 misses in the last 20 warmups of a model stop warmups for that model until 6 hours after its last warmup. A success does not reset it.
• The snapshot is deleted when the next turn starts, when warmups stop, or after 55 minutes.
• No warmups until reply gaps are recorded. PI_CACHE_RETENTION=long turns warming off.

Not verified against the live AI Gateway: that max_tokens: 1 with Opus adaptive thinking is accepted and reads the same cache entry as the agent's request. If it does not, the breaker stops warmups after 3 misses per model. The cache_warming.warmup.completed / .failed logs show the outcome within the first hours.

End-to-end coverage: tests that run the agent must be integration evals with Roach recordings, and the eval fixture's in-process queue does not deliver this topic. The flow (same prefix, max_tokens: 1, stale-generation no-op, miss stops the period, horizon records no reply, breaker, OpenAI models skipped) was verified locally with a throwaway component test that is not committed.

nitro.ts: the queue trigger setup was repeated for each topic, so it now uses one small helper for all four routes.

via @BYK.

--

View Session [Sentry]