See your LLM's cost, latency, and errors on one dashboard

No idea something's wrong until a user complains or the bill arrives.

Beginner to intermediateRead 6 minDoing: an afternoon7 steps

What you'll have at the end

A page that shows cost, p50 and p95 latency, and error rate together, computed from your own stored call records, confirmed to update the moment a new failed call lands.

Before you start

You need

You have a script or service that already calls an LLM API on its own, without a framework that hides the request from you.

Not covered

Distributed tracing across multiple services, and the separate question of buying a ready-made observability platform instead of building your own, are not covered here.

Leans on

Capture the raw facts

List the four facts every call reports
Wrap every call through one shared function
Save each report as one queryable row

Turn the records into live numbers

Turn the saved rows into three numbers
Put the three numbers on one page

Make it notice trouble on its own

Set a threshold worth an alert
Send a failure through and watch it move

Where to go next

Catch a spend spike before customers do

Turns the same rolling numbers into an early warning instead of a page you have to remember to check.

Buying an LLM observability platform usually beats building your own

Weighs this page against a ready-made platform once the upkeep of running it starts to add up.

Checked 15 Aug 2026

Part of the LLMOps cookbook