LangSmith Cheat Sheet

Updated 2026-05-28

Next Topic: Large Language Models (LLMs) Cheat Sheet

🧠Study flashcards on this topic113 cards · spaced repetition→

LangSmith is a unified DevOps platform for developing, debugging, testing, deploying, and monitoring LLM applications and AI agents built by LangChain. It provides framework-agnostic observability with comprehensive tracing, evaluation datasets, online/offline evaluations, prompt management, human-in-the-loop workflows, and production monitoring to help teams move from prototype to production. In 2026, LangSmith expanded significantly with LangSmith Engine (autonomous failure clustering + PR proposals), SmithDB (purpose-built Rust/DataFusion database, up to 15x faster), Context Hub (versioned agent context management), LLM Gateway (runtime spend limits + PII redaction), and Sandboxes GA (hardware-virtualized microVMs for safe agent code execution). LangSmith's core differentiator remains end-to-end visibility into agent execution via traces, enabling developers to understand, evaluate, and continuously improve complex multi-step LLM and agent workflows.

What This Cheat Sheet Covers

This topic spans 22 focused tables and 163 indexed concepts, 113 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Core ConceptsTable 2: Tracing and ObservabilityTable 3: Datasets and ExamplesTable 4: Offline EvaluationsTable 5: Online Evaluations and MonitoringTable 6: LangSmith EngineTable 7: Human Annotation WorkflowsTable 8: Prompt EngineeringTable 9: Context HubTable 10: Python SDKTable 11: JavaScript/TypeScript SDKTable 12: OpenTelemetry (OTEL) IntegrationTable 13: Framework IntegrationsTable 14: REST APITable 15: Production DeploymentTable 16: LangSmith Fleet (Agent Platform)Table 17: Self-Hosted LangSmithTable 18: Pricing and PlansTable 19: Comparison with AlternativesTable 20: Best PracticesTable 21: Common Use CasesTable 22: Advanced Features

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Core Concepts

The vocabulary of LangSmith — understanding these building blocks is prerequisite for everything else in the platform. The trace → run → feedback hierarchy maps directly to how observability data flows into evaluations, annotation queues, and Engine-driven improvements.

Concept	Example	Description
Trace	Single request → LLM → retrieval → response Top-level execution unit	• End-to-end execution path capturing full request lifecycle • contains all runs/spans for a single request • analogous to spans in distributed tracing (OpenTelemetry) • includes input, output, metadata, timestamps, cost
Run (Span)	Individual LLM call, retriever step, or tool invocation within trace	• Individual step within a trace, similar to OpenTelemetry spans • tracks single operation (LLM, chain, tool, retriever) • includes token counts, latency, cost; nested for complex workflows.
Dataset	`{"input": "What is AI?", "expected": "..."}` Versioned test-case collection	• Curated test cases for evaluation • supports CSV, JSON, JSONL + file attachments (images, PDFs, audio, video) • versioned — new version on every change; pin experiments to a specific version.
Experiment	Run application on dataset → compare v1 vs v2 prompts	• Evaluation run on a dataset producing scores and metrics (accuracy, latency, cost) • supports comparison view for A/B testing and baseline pinning for regression detection
Evaluator	LLM-as-judge, code-based, or human reviewer Scores outputs on criteria	• Scoring function for evaluation • types: LLM-as-judge, code-based, human (annotation queues), composite (weighted multi-score) • applied to experiments or online runs; now reusable across projects.
Project	Tracing project for "customer-support-bot" Organizational container	• Workspace for traces grouping related traces by application/environment • contains datasets, experiments, automations, Engine analysis • configurable retention policies and project-level dashboards.
Thread	Sequence of traces representing conversation: `thread_id = "user123-session456"`	• Conversation container grouping related traces • set via `thread_id` metadata • enables multi-turn chat tracking, Messages View, and Multi-turn Evals.
Feedback	`client.create_feedback(run_id, key="quality", score=0.8)`	• Score or annotation attached to a run • sources: user ratings, evaluator scores, human annotations • queryable via API • used for continuous improvement and RLHF
SmithDB	Backs 100% of US Cloud ingestion + tracing UI P50 trace tree load: 92ms	• Purpose-built distributed database for agent observability (Interrupt 2026) • built in Rust on Apache DataFusion + Vortex; object-storage-backed LSM • up to 15x faster than previous stack; stateless ingestion/query/compaction services • supports tree-aware queries, full-text search, JSON filtering, thread reconstruction.
Context Hub	`langsmith hub push my-agent --type agent` Versioned AGENTS.md, skills, policies	• Central store for agent context files — AGENTS.md, skills, policies, examples • versioning with tags (`dev`, `staging`, `prod`), comments, CLI push/pull • can serve as a virtual filesystem backend for Deep Agents (ContextHubBackend).

Table 2: Tracing and Observability

Tracing is the foundation: every LangSmith workflow — evaluations, Engine analysis, annotation queues — starts with traces. Mastering filtering, sampling, metadata, and the SmithDB-powered search unlocks every downstream workflow.

Feature	Example	Description
Automatic tracing	`export LANGSMITH_TRACING=true` `export LANGSMITH_API_KEY=...`	• Zero-code tracing for LangChain/LangGraph • automatically captures all chains, agents, tools • works with Python/JavaScript SDKs.

The rest of this section is part of the full cheat sheet

Table 3: Datasets and Examples

Datasets bridge production traces and offline evaluation: the closer your test cases are to real user interactions, the more reliable your evals. File attachments and versioning make LangSmith datasets suitable for multimodal and long-lived applications.

Operation	Example	Description
Create dataset (UI)	Upload CSV/JSON/JSONL Add examples manually via UI	• Create datasets directly in LangSmith UI • upload files or manually add examples with inputs and expected outputs • supports file attachments (images, PDFs, audio, video).

The rest of this section is part of the full cheat sheet

Table 4: Offline Evaluations

Offline evals are the first safety net before deployment: run your application against a curated dataset, compare experiments, and use the 30+ built-in templates to skip the blank-slate evaluation setup that has blocked teams for weeks.

Feature	Example	Description
Run experiment (SDK)	`evaluate(target, data=dataset, evaluators=[accuracy])` Programmatic testing	• Code-based evaluation for CI/CD • `evaluate()` runs application on dataset with multiple evaluators • returns detailed experiment results

The rest of this section is part of the full cheat sheet

Table 1: Core Concepts

Concept	Example	Description
Trace	Single request → LLM → retrieval → response Top-level execution unit	• End-to-end execution path capturing full request lifecycle • contains all runs/spans for a single request • analogous to spans in distributed tracing (OpenTelemetry) • includes input, output, metadata, timestamps, cost
Run (Span)	Individual LLM call, retriever step, or tool invocation within trace	• Individual step within a trace, similar to OpenTelemetry spans • tracks single operation (LLM, chain, tool, retriever) • includes token counts, latency, cost; nested for complex workflows.
Dataset	`{"input": "What is AI?", "expected": "..."}` Versioned test-case collection	• Curated test cases for evaluation • supports CSV, JSON, JSONL + file attachments (images, PDFs, audio, video) • versioned — new version on every change; pin experiments to a specific version.
Experiment	Run application on dataset → compare v1 vs v2 prompts	• Evaluation run on a dataset producing scores and metrics (accuracy, latency, cost) • supports comparison view for A/B testing and baseline pinning for regression detection
Evaluator	LLM-as-judge, code-based, or human reviewer Scores outputs on criteria	• Scoring function for evaluation • types: LLM-as-judge, code-based, human (annotation queues), composite (weighted multi-score) • applied to experiments or online runs; now reusable across projects.
Project	Tracing project for "customer-support-bot" Organizational container	• Workspace for traces grouping related traces by application/environment • contains datasets, experiments, automations, Engine analysis • configurable retention policies and project-level dashboards.
Thread	Sequence of traces representing conversation: `thread_id = "user123-session456"`	• Conversation container grouping related traces • set via `thread_id` metadata • enables multi-turn chat tracking, Messages View, and Multi-turn Evals.
Feedback	`client.create_feedback(run_id, key="quality", score=0.8)`	• Score or annotation attached to a run • sources: user ratings, evaluator scores, human annotations • queryable via API • used for continuous improvement and RLHF
SmithDB	Backs 100% of US Cloud ingestion + tracing UI P50 trace tree load: 92ms	• Purpose-built distributed database for agent observability (Interrupt 2026) • built in Rust on Apache DataFusion + Vortex; object-storage-backed LSM • up to 15x faster than previous stack; stateless ingestion/query/compaction services • supports tree-aware queries, full-text search, JSON filtering, thread reconstruction.
Context Hub	`langsmith hub push my-agent --type agent` Versioned AGENTS.md, skills, policies	• Central store for agent context files — AGENTS.md, skills, policies, examples • versioning with tags (`dev`, `staging`, `prod`), comments, CLI push/pull • can serve as a virtual filesystem backend for Deep Agents (ContextHubBackend).

Table 2: Tracing and Observability

Feature	Example	Description
Automatic tracing	`export LANGSMITH_TRACING=true` `export LANGSMITH_API_KEY=...`	• Zero-code tracing for LangChain/LangGraph • automatically captures all chains, agents, tools • works with Python/JavaScript SDKs.

Table 3: Datasets and Examples

Operation	Example	Description
Create dataset (UI)	Upload CSV/JSON/JSONL Add examples manually via UI	• Create datasets directly in LangSmith UI • upload files or manually add examples with inputs and expected outputs • supports file attachments (images, PDFs, audio, video).

Table 4: Offline Evaluations

Feature	Example	Description
Run experiment (SDK)	`evaluate(target, data=dataset, evaluators=[accuracy])` Programmatic testing	• Code-based evaluation for CI/CD • `evaluate()` runs application on dataset with multiple evaluators • returns detailed experiment results