Vector Embeddings Cheat Sheet

Updated 2026-05-28

Next Topic: Vision-Language Models (VLMs) Cheat Sheet

🧠Study flashcards on this topic139 cards · spaced repetition→

Vector embeddings transform complex data—text, images, audio, video—into numerical representations in high-dimensional space where semantic similarity becomes geometric proximity. They power modern AI applications from semantic search to retrieval-augmented generation (RAG), enabling machines to understand meaning rather than just match keywords. The key insight: distance metrics in embedding space directly measure conceptual similarity, allowing algorithms to find related items without exact string matching. In 2025–2026, the embedding landscape has been reshaped by all-modality models like Gemini Embedding 2 and Voyage 4, mixture-of-experts architectures that cut active-parameter costs by 40%, and asymmetric retrieval patterns that embed documents and queries with different-sized models sharing a compatible vector space. Understanding dimensions, distance metrics, indexing strategies, quantization tradeoffs, and the full retrieval pipeline is essential for building efficient, production-scale AI systems.

What This Cheat Sheet Covers

This topic spans 14 focused tables and 142 indexed concepts, 139 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Core Embedding ConceptsTable 2: Embedding Models and ArchitecturesTable 3: Distance and Similarity MetricsTable 4: Vector Databases and StoresTable 5: Approximate Nearest Neighbor (ANN) AlgorithmsTable 6: Embedding Training and Fine-Tuning TechniquesTable 7: Retrieval Strategies and ApplicationsTable 8: Embedding Optimization TechniquesTable 9: Pooling StrategiesTable 10: Chunking Strategies for RAGTable 11: Sparse Retrieval MethodsTable 12: Indexing and Storage ConsiderationsTable 13: Embedding Evaluation MetricsTable 14: Advanced Embedding Techniques

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Core Embedding Concepts

Embeddings represent data as high-dimensional vectors where geometric distance encodes meaning; the concepts here define the vocabulary shared across every retrieval and generative AI system you will encounter. Getting these fundamentals right — especially understanding dense vs sparse and contextualized vs static — prevents the most common architectural mistakes.

Concept	Example	Description
Dense embedding	`[0.23, -0.71, 0.44, ...]` (768 floats)	• A continuous, low-dimensional vector where most values are non-zero • captures semantic meaning but loses exact-term recall
Sparse embedding	`{42: 1.3, 1891: 0.7}` (vocab-sized)	• A high-dimensional, mostly-zero vector • each non-zero dimension corresponds to a specific token • enables exact-term retrieval
Sentence embedding	`model.encode("hello world")`	• A single fixed-length vector representing an entire sentence or passage • the standard unit for semantic search and RAG retrieval
Contextualized embedding	`"bank"` in "river bank" ≠ "bank" in "savings bank"	• Vectors computed from full sentence context, so the same word gets a different vector depending on surrounding text (e.g • BERT, GPT).
Static embedding	`word2vec["king"] - word2vec["man"] + word2vec["woman"]`	• A fixed per-word vector regardless of context • fast and memory-efficient but cannot represent polysemy
Semantic similarity	`cos_sim(embed("cat"), embed("feline"))` → 0.91	• Measures conceptual relatedness between two vectors • cosine similarity is the most common metric
Embedding dimension	768153630724096	• The length of the vector • higher dimensions can capture more nuance but cost more memory and compute • Modern models range from 256 to 7168.
Vector space	All sentences from a corpus projected into ℝ^1536	• The mathematical space all embeddings live in • geometric operations (distance, projection) directly correspond to semantic operations
Multimodal embedding	`embed(image) ≈ embed("a photo of a dog")`	• A single shared vector space for multiple data types (text, image, audio, video) • enables cross-modal retrieval without modality-specific indexes
Token-level embedding	Each token in a query → its own 128-dim vector	• Preserves per-token representations rather than collapsing to one vector • used by ColBERT for late interaction matching
Modality gap	Qwen3-VL-2B gap = 0.25; Gemini Embedding 2 gap = 0.73	• The average cosine distance between text and image vectors in a shared multimodal embedding space • lower = tighter cross-modal alignment and more accurate cross-modal retrieval

Table 2: Embedding Models and Architectures

Embedding models span three decades of research from predictive word vectors to multi-billion-parameter instruction-tuned transformers; the right model matches task, context length, modality, and deployment constraints. The 2025-2026 generation is defined by all-modality support, instruction-aware encoding, and mixture-of-experts architectures that serve multiple model sizes from a shared embedding space.

Model	Example	Description
Sentence-BERT (SBERT)	`SentenceTransformer("all-mpnet-base-v2").encode(texts)`	• Siamese BERT with contrastive objectives • ~1000x faster than cross-encoder for sentence vectors • most widely deployed open-source sentence encoder

The rest of this section is part of the full cheat sheet

Table 3: Distance and Similarity Metrics

Three metrics dominate production - cosine similarity, dot product, and L2 - and choosing the wrong one silently degrades recall; the remaining metrics cover quantized binary representations and multi-vector scoring. Always normalize embeddings to unit length if using dot product as a cosine proxy to gain maximum SIMD throughput.

Metric	Example	Description
Cosine similarity	`dot(a,b) / (norm(a) * norm(b))` -> [-1, 1]	• Measures the angle between vectors, ignoring magnitude • the most common text similarity metric • equivalent to dot product when vectors are L2-normalized

The rest of this section is part of the full cheat sheet

Table 4: Vector Databases and Stores

The vector database landscape ranges from serverless cloud-native services to SQLite extensions; the right choice depends on scale, latency requirements, multi-tenancy needs, and whether you want vectors co-located with relational data. Hybrid search (vector + keyword) is now a standard expectation and is supported natively by every leading option.

Database	Example	Description
Pinecone	`index.upsert(vectors=[("id1", [0.1,...], {"meta":"val"})])`	• Fully managed serverless • easiest operational path to production • automatic scaling, hybrid search • standard starter choice for new projects

The rest of this section is part of the full cheat sheet

Table 1: Core Embedding Concepts

Concept	Example	Description
Dense embedding	`[0.23, -0.71, 0.44, ...]` (768 floats)	• A continuous, low-dimensional vector where most values are non-zero • captures semantic meaning but loses exact-term recall
Sparse embedding	`{42: 1.3, 1891: 0.7}` (vocab-sized)	• A high-dimensional, mostly-zero vector • each non-zero dimension corresponds to a specific token • enables exact-term retrieval
Sentence embedding	`model.encode("hello world")`	• A single fixed-length vector representing an entire sentence or passage • the standard unit for semantic search and RAG retrieval
Contextualized embedding	`"bank"` in "river bank" ≠ "bank" in "savings bank"	• Vectors computed from full sentence context, so the same word gets a different vector depending on surrounding text (e.g • BERT, GPT).
Static embedding	`word2vec["king"] - word2vec["man"] + word2vec["woman"]`	• A fixed per-word vector regardless of context • fast and memory-efficient but cannot represent polysemy
Semantic similarity	`cos_sim(embed("cat"), embed("feline"))` → 0.91	• Measures conceptual relatedness between two vectors • cosine similarity is the most common metric
Embedding dimension	768153630724096	• The length of the vector • higher dimensions can capture more nuance but cost more memory and compute • Modern models range from 256 to 7168.
Vector space	All sentences from a corpus projected into ℝ^1536	• The mathematical space all embeddings live in • geometric operations (distance, projection) directly correspond to semantic operations
Multimodal embedding	`embed(image) ≈ embed("a photo of a dog")`	• A single shared vector space for multiple data types (text, image, audio, video) • enables cross-modal retrieval without modality-specific indexes
Token-level embedding	Each token in a query → its own 128-dim vector	• Preserves per-token representations rather than collapsing to one vector • used by ColBERT for late interaction matching
Modality gap	Qwen3-VL-2B gap = 0.25; Gemini Embedding 2 gap = 0.73	• The average cosine distance between text and image vectors in a shared multimodal embedding space • lower = tighter cross-modal alignment and more accurate cross-modal retrieval

Table 2: Embedding Models and Architectures

Model	Example	Description
Sentence-BERT (SBERT)	`SentenceTransformer("all-mpnet-base-v2").encode(texts)`	• Siamese BERT with contrastive objectives • ~1000x faster than cross-encoder for sentence vectors • most widely deployed open-source sentence encoder

Table 3: Distance and Similarity Metrics

Metric	Example	Description
Cosine similarity	`dot(a,b) / (norm(a) * norm(b))` -> [-1, 1]	• Measures the angle between vectors, ignoring magnitude • the most common text similarity metric • equivalent to dot product when vectors are L2-normalized

Table 4: Vector Databases and Stores

Database	Example	Description
Pinecone	`index.upsert(vectors=[("id1", [0.1,...], {"meta":"val"})])`	• Fully managed serverless • easiest operational path to production • automatic scaling, hybrid search • standard starter choice for new projects