LLMOps cookbook

13 recipes · 91 steps · about 83 minutes of reading

See your LLM's cost, latency, and errors on one dashboard
No idea something's wrong until a user complains or the bill arrives.
Beginner to intermediateRead 6 minDoing: an afternoon7 steps
Roll back a prompt change that broke your output
I shipped a small prompt tweak, and now some answers are subtly worse and nobody can say why.
Beginner to intermediateRead 6 minDoing: a couple of hours6 steps
Catch a spend spike before customers do
My AI feature's cost or error rate can double overnight, and the first I hear about it is a message from support.
IntermediateRead 5 minDoing: an afternoon6 steps
Nobody on the team knows which feature is burning your LLM budget
One big bill, and nobody can say which feature is driving it.
IntermediateRead 5 minDoing: an afternoon7 steps
Cache semantically similar prompts instead of paying twice
The same question comes in worded five different ways, and my cache only ever catches the exact repeats.
IntermediateRead 7 minDoing: a day7 steps
Pin your model's version so a silent update can't surprise you
Nothing changed in my code, so why do the model's answers look different this month?
IntermediateRead 5 minDoing: an afternoon6 steps
Catch your LLM's drift before a user has to report it
My eval suite passed on launch day, and nothing warned me before users started noticing the answers slipping weeks later.
IntermediateRead 8 minDoing: half a day to wire up, then it runs itself8 steps
Route your easy requests to a cheaper model
Every request goes to the same expensive model, whether it is a one-line summary or a hard multi-step question.
Intermediate to advancedRead 7 minDoing: a day8 steps
Your app breaks the moment your LLM provider does
Our only LLM provider went down for an afternoon, and every feature that depends on it went down with it.
Intermediate to advancedRead 7 minDoing: a day7 steps
Strip PII out of your LLM logs
A customer typed their card number straight into a chat box. It's sitting in plain text in our logs, and it's not the only one.
Intermediate to advancedRead 7 minDoing: an afternoon6 steps
Buying an LLM observability platform usually beats building your own
I keep adding tracing, cost tracking, and evals to my own logging code, and it is starting to look like a whole product.
Intermediate to advancedRead 5 minDoing: an afternoon6 steps
Canary a prompt change before it reaches every user
My last prompt tweak went to every single user before anyone had a chance to notice a regression.
Intermediate to advancedRead 9 minDoing: half a day to wire up, then about a week waiting on real traffic9 steps
Trace one wrong agent answer back to the tool call that caused it
A user reported one bad reply, and I have no way to tell which of my agent's several tool calls produced it.
Intermediate to advancedRead 6 minDoing: an hour8 steps

Cookbook badge

Finish all 13 recipes to earn this.

In these roadmaps

AI Engineer (LLM Applications)step 14 of 15

LLMOps cookbook

See your LLM's cost, latency, and errors on one dashboard

Roll back a prompt change that broke your output

Catch a spend spike before customers do

Nobody on the team knows which feature is burning your LLM budget

Cache semantically similar prompts instead of paying twice

Pin your model's version so a silent update can't surprise you

Catch your LLM's drift before a user has to report it

Route your easy requests to a cheaper model

Your app breaks the moment your LLM provider does

Strip PII out of your LLM logs

Buying an LLM observability platform usually beats building your own

Canary a prompt change before it reaches every user

Trace one wrong agent answer back to the tool call that caused it

In these roadmaps

LLMOps cookbook

See your LLM's cost, latency, and errors on one dashboard

Roll back a prompt change that broke your output

Catch a spend spike before customers do

Nobody on the team knows which feature is burning your LLM budget

Cache semantically similar prompts instead of paying twice

Pin your model's version so a silent update can't surprise you

Catch your LLM's drift before a user has to report it

Route your easy requests to a cheaper model

Your app breaks the moment your LLM provider does

Strip PII out of your LLM logs

Buying an LLM observability platform usually beats building your own

Canary a prompt change before it reaches every user

Trace one wrong agent answer back to the tool call that caused it

In these roadmaps