Vision-Language Models (VLMs) Cheat Sheet

Updated 2026-05-25

Next Topic: vLLM (LLM Inference Engine) Cheat Sheet

🧠Study flashcards on this topic127 cards · spaced repetition→

Vision-Language Models (VLMs) are multimodal AI systems that seamlessly integrate visual perception and natural language understanding, enabling machines to reason about images, videos, and text simultaneously. These models power applications from visual question answering to zero-shot image classification, fundamentally changing how AI interprets the visual world. At their core, VLMs learn a shared embedding space where semantically similar images and text descriptions cluster together—a capability that emerged from contrastive learning techniques pioneered by CLIP and refined by subsequent architectures. A critical insight: the quality of vision-language alignment depends not just on model architecture, but on how effectively the model bridges the modality gap between continuous visual features and discrete linguistic tokens through specialized fusion mechanisms, and how well training data covers the diversity of visual scenes and instructions encountered at inference time.

What This Cheat Sheet Covers

This topic spans 16 focused tables and 130 indexed concepts, 127 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Foundation VLM ArchitecturesTable 2: Vision EncodersTable 3: Text EncodersTable 4: Multimodal Fusion StrategiesTable 5: Pre-training ObjectivesTable 6: Core VLM TasksTable 7: Alignment & Training StrategiesTable 8: Attention MechanismsTable 9: Key Architectural ComponentsTable 10: Loss Functions & OptimizationTable 11: Evaluation MetricsTable 12: Benchmark DatasetsTable 13: Advanced TechniquesTable 14: Common Implementation DetailsTable 15: Video Understanding in VLMsTable 16: Hallucination & Safety in VLMs

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Foundation VLM Architectures

The VLM landscape has evolved through three architectural eras: early contrastive models (CLIP), efficient connector-based models (BLIP-2, LLaVA), and the current generation treating vision as a native LLM modality with dynamic resolution and strong reasoning. Understanding each generation's design tradeoffs — frozen encoders vs. joint training, contrastive vs. generative objectives, lightweight connectors vs. heavy cross-attention — is essential before working with any modern VLM.

Model	Example	Description
CLIP	`image_encoder = ViT-L/14` `text_encoder = Transformer`	• Dual-encoder architecture trained on 400M image-text pairs using InfoNCE contrastive loss • enables powerful zero-shot transfer to downstream tasks without fine-tuning.
LLaVA	`vision_encoder = CLIP ViT` `llm = Vicuna-13B` `projection = linear`	• Large Language and Vision Assistant using GPT-4-generated instruction data • simple linear projection layer maps visual features to LLM token space for multimodal conversations
LLaVA-1.5	`vision_encoder = CLIP-ViT-L-336px` `connector = MLP_2-layer` `llm = Vicuna-13B`	• Replaces linear projection with 2-layer MLP connector and adds academic-task VQA data • achieves SOTA across 11 benchmarks using only 1.2M public training samples in ~1 day on 8 A100s
BLIP-2	`Q-Former + frozen vision encoder` `+ frozen LLM`	• Connects frozen image encoder and LLM via lightweight Querying Transformer (Q-Former) with 32 learnable queries • achieves SOTA with 54× fewer trainable parameters than Flamingo
Flamingo	`Perceiver Resampler + gated cross-attention` `few_shot_examples = 4`	• Few-shot learning specialist using Perceiver Resampler to compress visual tokens • integrates images into frozen LLM via gated cross-attention layers inserted between Transformer blocks
InstructBLIP	`BLIP-2 + instruction_tuning` `datasets = 26_transformed`	• Instruction-tuned version of BLIP-2 trained on 26 vision-language datasets • follows natural language instructions for diverse tasks without task-specific heads.
Qwen2-VL	`dynamic_resolution = any` `M-RoPE = temporal+spatial` `params = 2B/7B/72B`	• Introduces Naive Dynamic Resolution — images processed at any resolution into a variable number of visual tokens — plus Multimodal Rotary Position Embedding (M-RoPE) for spatial and temporal position fusion • 72B version rivals GPT-4o on multimodal benchmarks
InternVL2	`vision = InternViT-6B` `llm = InternLM2/Qwen2.5` `connector = MLP`	• First open-source MLLM to exceed 70% on MMMU • uses a large 6B vision encoder (InternViT), progressive scaling strategy, and chain-of-thought fine-tuning • sizes range 1B–78B
BLIP	`model = MED(unimodal + multimodal)` `data = CapFilt bootstrapping`	• Unified encoder-decoder framework (MED) with image-text contrastive, matching, and captioning losses • includes CapFilt bootstrapping to filter noisy web captions
CoCa	`contrastive_loss + captioning_loss` `single_encoder_decoder = unified`	• Contrastive Captioner unifying image-text contrastive learning with generative captioning in a single Transformer • decoder attends to encoder outputs via cross-attention
MiniGPT-4	`Q-Former + projection + Vicuna` `alignment_stage = 5M_pairs`	• Aligns frozen BLIP-2 Q-Former to frozen Vicuna LLM using a single projection layer • two-stage training with conversational fine-tuning achieves GPT-4-like capabilities with minimal compute
Qwen-VL	`vision_encoder = ViT-G` `llm = Qwen-7B` `resolution = dynamic`	• Bilingual VLM supporting dynamic resolution and fine-grained localization • trained on 1.4B image-text pairs • superseded by Qwen2-VL but still widely referenced

Table 2: Vision Encoders

The vision encoder determines what a VLM can perceive — patch size, resolution support, and pre-training methodology all constrain downstream capability. CLIP-family encoders are pre-aligned with text and dominate most connector-based architectures; large-scale encoders like InternViT and EVA-CLIP unlock finer-grained understanding at the cost of compute.

Encoder	Example	Description
Vision Transformer (ViT)	`patch_size = 16` `image_res = 224` `tokens = 196 + 1_cls`	• Splits images into fixed-size patches treated as sequence tokens • applies self-attention across all patches • prepends learnable [CLS] token for image-level representation

The rest of this section is part of the full cheat sheet

Table 3: Text Encoders

Text encoders in VLMs range from bidirectional BERT-style models for understanding tasks to causal GPT-style decoders for generation. The dominant modern pattern is a large autoregressive LLM serving as the entire language backbone, with the text encoder role absorbed into the LLM itself.

Encoder	Example	Description
CLIP Text Encoder	`architecture = Transformer` `vocab_size = 49408` `max_length = 77`	• Masked self-attention Transformer trained contrastively • learns text representations aligned to visual space • prepends [SOS] and appends [EOS] tokens • 77-token limit reflects CLIP's training distribution

The rest of this section is part of the full cheat sheet

Table 4: Multimodal Fusion Strategies

Fusion strategy determines how and when visual information enters the language stream — it is one of the most impactful architectural decisions in VLM design. The trend has shifted from complex cross-attention (Flamingo) to simpler projection-based early fusion (LLaVA), with resampling methods (Q-Former, Perceiver Resampler) occupying the middle ground for token-budget-sensitive tasks like video.

Strategy	Example	Description
Early Fusion	`concat([vision_tokens, text_tokens])` `→ unified_transformer`	• Merges vision and text tokens at input level before processing • enables deep cross-modal interaction • dominant in LLaVA-family models — visual tokens are prepended to text tokens and processed uniformly by the LLM

The rest of this section is part of the full cheat sheet

Table 1: Foundation VLM Architectures

Model	Example	Description
CLIP	`image_encoder = ViT-L/14` `text_encoder = Transformer`	• Dual-encoder architecture trained on 400M image-text pairs using InfoNCE contrastive loss • enables powerful zero-shot transfer to downstream tasks without fine-tuning.
LLaVA	`vision_encoder = CLIP ViT` `llm = Vicuna-13B` `projection = linear`	• Large Language and Vision Assistant using GPT-4-generated instruction data • simple linear projection layer maps visual features to LLM token space for multimodal conversations
LLaVA-1.5	`vision_encoder = CLIP-ViT-L-336px` `connector = MLP_2-layer` `llm = Vicuna-13B`	• Replaces linear projection with 2-layer MLP connector and adds academic-task VQA data • achieves SOTA across 11 benchmarks using only 1.2M public training samples in ~1 day on 8 A100s
BLIP-2	`Q-Former + frozen vision encoder` `+ frozen LLM`	• Connects frozen image encoder and LLM via lightweight Querying Transformer (Q-Former) with 32 learnable queries • achieves SOTA with 54× fewer trainable parameters than Flamingo
Flamingo	`Perceiver Resampler + gated cross-attention` `few_shot_examples = 4`	• Few-shot learning specialist using Perceiver Resampler to compress visual tokens • integrates images into frozen LLM via gated cross-attention layers inserted between Transformer blocks
InstructBLIP	`BLIP-2 + instruction_tuning` `datasets = 26_transformed`	• Instruction-tuned version of BLIP-2 trained on 26 vision-language datasets • follows natural language instructions for diverse tasks without task-specific heads.
Qwen2-VL	`dynamic_resolution = any` `M-RoPE = temporal+spatial` `params = 2B/7B/72B`	• Introduces Naive Dynamic Resolution — images processed at any resolution into a variable number of visual tokens — plus Multimodal Rotary Position Embedding (M-RoPE) for spatial and temporal position fusion • 72B version rivals GPT-4o on multimodal benchmarks
InternVL2	`vision = InternViT-6B` `llm = InternLM2/Qwen2.5` `connector = MLP`	• First open-source MLLM to exceed 70% on MMMU • uses a large 6B vision encoder (InternViT), progressive scaling strategy, and chain-of-thought fine-tuning • sizes range 1B–78B
BLIP	`model = MED(unimodal + multimodal)` `data = CapFilt bootstrapping`	• Unified encoder-decoder framework (MED) with image-text contrastive, matching, and captioning losses • includes CapFilt bootstrapping to filter noisy web captions
CoCa	`contrastive_loss + captioning_loss` `single_encoder_decoder = unified`	• Contrastive Captioner unifying image-text contrastive learning with generative captioning in a single Transformer • decoder attends to encoder outputs via cross-attention
MiniGPT-4	`Q-Former + projection + Vicuna` `alignment_stage = 5M_pairs`	• Aligns frozen BLIP-2 Q-Former to frozen Vicuna LLM using a single projection layer • two-stage training with conversational fine-tuning achieves GPT-4-like capabilities with minimal compute
Qwen-VL	`vision_encoder = ViT-G` `llm = Qwen-7B` `resolution = dynamic`	• Bilingual VLM supporting dynamic resolution and fine-grained localization • trained on 1.4B image-text pairs • superseded by Qwen2-VL but still widely referenced

Table 2: Vision Encoders

Encoder	Example	Description
Vision Transformer (ViT)	`patch_size = 16` `image_res = 224` `tokens = 196 + 1_cls`	• Splits images into fixed-size patches treated as sequence tokens • applies self-attention across all patches • prepends learnable [CLS] token for image-level representation

Table 3: Text Encoders

Encoder	Example	Description
CLIP Text Encoder	`architecture = Transformer` `vocab_size = 49408` `max_length = 77`	• Masked self-attention Transformer trained contrastively • learns text representations aligned to visual space • prepends [SOS] and appends [EOS] tokens • 77-token limit reflects CLIP's training distribution

Table 4: Multimodal Fusion Strategies

Strategy	Example	Description
Early Fusion	`concat([vision_tokens, text_tokens])` `→ unified_transformer`	• Merges vision and text tokens at input level before processing • enables deep cross-modal interaction • dominant in LLaVA-family models — visual tokens are prepended to text tokens and processed uniformly by the LLM