LangSmith Cheat Sheet

Updated 2026-04-05

Next Topic: Large Language Models (LLMs) Cheat Sheet

LangSmith is a unified DevOps platform for developing, debugging, testing, deploying, and monitoring LLM applications and AI agents built by LangChain. It provides framework-agnostic observability with comprehensive tracing, evaluation datasets, online/offline evaluations, prompt management, human-in-the-loop workflows, and production monitoring to help teams move models from prototype to production. In 2025–2026, LangSmith expanded significantly with the Insights Agent (automated production trace analysis), Multi-turn Evals (conversation-level evaluation), OpenTelemetry (OTEL) integration, LangSmith Fleet — a no-code platform for building and managing AI agent fleets — and LangSmith Sandboxes (secure microVM-based code execution for agents, private preview March 2026). LangSmith's core differentiator remains end-to-end visibility into agent execution via traces and runs, enabling developers to understand what happens inside complex multi-step LLM workflows through detailed logging, cost tracking, latency metrics (P50/P99), and automated evaluations at scale.

What This Cheat Sheet Covers

This topic spans 20 focused tables and 131 indexed concepts. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Core ConceptsTable 2: Tracing and ObservabilityTable 3: Datasets and ExamplesTable 4: Offline EvaluationsTable 5: Online Evaluations and MonitoringTable 6: Human Annotation WorkflowsTable 7: Prompt EngineeringTable 8: Python SDKTable 9: JavaScript/TypeScript SDKTable 10: OpenTelemetry (OTEL) IntegrationTable 11: Framework IntegrationsTable 12: REST APITable 13: Production DeploymentTable 14: LangSmith Fleet (Agent Platform)Table 15: Self-Hosted LangSmithTable 16: Pricing and PlansTable 17: Comparison with AlternativesTable 18: Best PracticesTable 19: Common Use CasesTable 20: Advanced Features

Table 1: Core Concepts

Concept	Example	Description
Trace	Single request → LLM → retrieval → response Top-level execution unit	• End-to-end execution path capturing full request lifecycle • contains all runs/spans for a single request • analogous to spans in distributed tracing (OpenTelemetry) • includes input, output, metadata, timestamps, cost.
Run (Span)	Individual LLM call, retriever step, or tool invocation within trace	• Individual step within a trace • similar to spans in OpenTelemetry • tracks single operation (LLM, chain, tool, retriever) • includes token counts, latency, cost • nested structure for complex workflows.
Dataset	Collection of input-output pairs: `{"input": "What is AI?", "expected": "..."}` Versioned test cases	• Curated test cases for evaluation • contains example inputs and expected outputs • supports CSV, JSON, JSONL formats and file attachments (images, PDFs, audio, video) • versioned — new version on every change.
Experiment	Run application on dataset → compare v1 vs v2 prompts	• Evaluation run on a dataset producing scores and metrics (accuracy, latency, cost) • supports comparison view for A/B testing • baseline pinning for regression detection.

Table 1: Core Concepts

Concept	Example	Description
Trace	Single request → LLM → retrieval → response Top-level execution unit	• End-to-end execution path capturing full request lifecycle • contains all runs/spans for a single request • analogous to spans in distributed tracing (OpenTelemetry) • includes input, output, metadata, timestamps, cost.
Run (Span)	Individual LLM call, retriever step, or tool invocation within trace	• Individual step within a trace • similar to spans in OpenTelemetry • tracks single operation (LLM, chain, tool, retriever) • includes token counts, latency, cost • nested structure for complex workflows.
Dataset	Collection of input-output pairs: `{"input": "What is AI?", "expected": "..."}` Versioned test cases	• Curated test cases for evaluation • contains example inputs and expected outputs • supports CSV, JSON, JSONL formats and file attachments (images, PDFs, audio, video) • versioned — new version on every change.
Experiment	Run application on dataset → compare v1 vs v2 prompts	• Evaluation run on a dataset producing scores and metrics (accuracy, latency, cost) • supports comparison view for A/B testing • baseline pinning for regression detection.