New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
My AI feature's cost or error rate can double overnight, and the first I hear about it is a message from support.
What you'll have at the end
A script that recomputes each feature's rolling cost and error rate on a short window, checks the newest window and the one right before it against that feature's own recent baseline, and posts an alert the moment both cross roughly triple the feature's normal swing, confirmed against a real injected spike and against an ordinary noisy window that correctly stays quiet.
You need
You already have a place that stores one row per call, naming the feature, the cost, and whether it succeeded or failed, and that you can query by a time range.
Not covered
Explaining why a flagged window spiked and tracing it to the customer, model, or code change behind it is not covered here.
Leans on
See your LLM's cost, latency, and errors on one dashboard
Start there first if you don't yet have a queryable per-call record of cost and status to build this alert's baseline from.
Nobody on the team knows which feature is burning your LLM budget
Once this alert fires, that recipe is how you trace which feature, model, or user actually drove it.
Buying an LLM observability platform usually beats building your own
Once alert upkeep like this piles up across many features, weighs whether a ready-made platform beats maintaining it yourself.
Checked 15 Aug 2026
Part of the LLMOps cookbook