Small Language Models (SLMs) Cheat Sheet

Updated 2026-05-18

Next Topic: spaCy Industrial NLP Library Cheat Sheet

🧠Study flashcards on this topic83 cards · spaced repetition→

Small Language Models (SLMs) are compact AI models with 1B-13B parameters designed for efficient deployment on edge devices and resource-constrained environments. Unlike their larger counterparts that require cloud infrastructure, SLMs enable on-device inference with faster response times, lower latency, and enhanced privacy — making them ideal for mobile, IoT, and offline applications. The critical insight: SLMs trade broad general knowledge for domain-specific expertise and efficiency, achieving 70-90% of LLM performance while using a fraction of resources through techniques like quantization, distillation, and pruning.

What This Cheat Sheet Covers

This topic spans 12 focused tables and 89 indexed concepts, 83 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Core SLM CharacteristicsTable 2: Major SLM Model FamiliesTable 3: Model Compression TechniquesTable 4: Quantization MethodsTable 5: Knowledge Distillation ApproachesTable 6: Parameter-Efficient Fine-Tuning (PEFT)Table 7: On-Device Deployment FrameworksTable 8: Hardware and Memory RequirementsTable 9: Inference Optimization TechniquesTable 10: Evaluation and BenchmarkingTable 11: Domain Specialization StrategiesTable 12: SLM vs LLM Decision Framework

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Core SLM Characteristics

Understanding what defines a small language model and how size correlates with deployment constraints helps determine when SLMs are the right choice over large models.

Characteristic	Example	Description
Parameter Count	`1B-13B parameters`	• Typically ranges from 100 million to 13 billion parameters; models above 13B are generally classified as LLMs • smaller parameter counts enable faster inference and lower memory footprint
Model Size	`FP16: ~2GB (1B) to 26GB (13B)` `INT4: ~0.5GB (1B) to 6.5GB (13B)`	• Size in GB depends on precision; FP16 requires ~2 bytes per parameter, INT4 ~0.5 bytes • critical for determining whether a model fits in device memory
Training Data Volume	`500B-9T tokens`	• SLMs like Phi-4 (14B) trained on 9 trillion tokens • smaller models compensate for size through high-quality, curated datasets and longer training
Inference Latency	`<100ms per token on edge devices`	• SLMs achieve sub-100ms latency on mobile CPUs/GPUs • 2-5x faster than streaming from cloud LLMs due to elimination of network overhead
Context Window	`128K tokens (Llama 3.2 1B/3B)`	• Modern SLMs support long context windows (up to 128K tokens) rivaling LLMs • enables processing entire documents without chunking
Memory Footprint	`1.8GB VRAM (Llama 3.2 1B)` `3.4GB VRAM (Llama 3.2 3B)`	• GPU memory requirement at runtime; 1B models fit on 2GB VRAM, 7B on 8GB with 4-bit quantization • determines device compatibility
Domain Specialization	`Medical: BioGPT-Small` `Legal: LegalBERT-Small`	• SLMs excel when fine-tuned on domain-specific data; often outperform general LLMs on specialized tasks • training on narrow data improves accuracy
Energy Consumption	`0.5-2W inference power`	• Edge deployment reduces energy to watts instead of kilowatts • battery-powered devices benefit from lower computational requirements

Table 2: Major SLM Model Families

The landscape of production-ready small language models spans models from major tech companies, each optimized for different deployment scenarios and hardware constraints.

Model	Example	Description
Microsoft Phi-3	`Phi-3-mini: 3.8B parameters` `Phi-3-small: 7B parameters`	• State-of-the-art performance per parameter; 3.8B mini matches models 10x larger • trained on high-quality synthetic data using larger model outputs

The rest of this section is part of the full cheat sheet

Table 3: Model Compression Techniques

Compression reduces model size and computational requirements while preserving accuracy, enabling deployment on devices with limited resources.

Technique	Example	Description
Structured Pruning	`Remove entire neurons/layers` `SlimLLM framework`	• Removes groups of parameters (neurons, attention heads, layers) maintaining dense matrices • easier hardware acceleration than unstructured pruning

The rest of this section is part of the full cheat sheet

Table 4: Quantization Methods

Quantization reduces numerical precision of model weights and activations, dramatically lowering memory usage and enabling faster inference on specialized hardware.

Method	Example	Description
INT4 Quantization	`4-bit integer weights` `~75% size reduction`	• Weights stored as 4-bit integers; reduces model size by 75% from FP16 • minimal accuracy loss (<3%) for most SLMs

The rest of this section is part of the full cheat sheet

Table 1: Core SLM Characteristics

Understanding what defines a small language model and how size correlates with deployment constraints helps determine when SLMs are the right choice over large models.

Characteristic	Example	Description
Parameter Count	`1B-13B parameters`	• Typically ranges from 100 million to 13 billion parameters; models above 13B are generally classified as LLMs • smaller parameter counts enable faster inference and lower memory footprint
Model Size	`FP16: ~2GB (1B) to 26GB (13B)` `INT4: ~0.5GB (1B) to 6.5GB (13B)`	• Size in GB depends on precision; FP16 requires ~2 bytes per parameter, INT4 ~0.5 bytes • critical for determining whether a model fits in device memory
Training Data Volume	`500B-9T tokens`	• SLMs like Phi-4 (14B) trained on 9 trillion tokens • smaller models compensate for size through high-quality, curated datasets and longer training
Inference Latency	`<100ms per token on edge devices`	• SLMs achieve sub-100ms latency on mobile CPUs/GPUs • 2-5x faster than streaming from cloud LLMs due to elimination of network overhead
Context Window	`128K tokens (Llama 3.2 1B/3B)`	• Modern SLMs support long context windows (up to 128K tokens) rivaling LLMs • enables processing entire documents without chunking
Memory Footprint	`1.8GB VRAM (Llama 3.2 1B)` `3.4GB VRAM (Llama 3.2 3B)`	• GPU memory requirement at runtime; 1B models fit on 2GB VRAM, 7B on 8GB with 4-bit quantization • determines device compatibility
Domain Specialization	`Medical: BioGPT-Small` `Legal: LegalBERT-Small`	• SLMs excel when fine-tuned on domain-specific data; often outperform general LLMs on specialized tasks • training on narrow data improves accuracy
Energy Consumption	`0.5-2W inference power`	• Edge deployment reduces energy to watts instead of kilowatts • battery-powered devices benefit from lower computational requirements

Table 2: Major SLM Model Families

The landscape of production-ready small language models spans models from major tech companies, each optimized for different deployment scenarios and hardware constraints.

Model	Example	Description
Microsoft Phi-3	`Phi-3-mini: 3.8B parameters` `Phi-3-small: 7B parameters`	• State-of-the-art performance per parameter; 3.8B mini matches models 10x larger • trained on high-quality synthetic data using larger model outputs

Table 3: Model Compression Techniques

Compression reduces model size and computational requirements while preserving accuracy, enabling deployment on devices with limited resources.

Technique	Example	Description
Structured Pruning	`Remove entire neurons/layers` `SlimLLM framework`	• Removes groups of parameters (neurons, attention heads, layers) maintaining dense matrices • easier hardware acceleration than unstructured pruning

Table 4: Quantization Methods

Quantization reduces numerical precision of model weights and activations, dramatically lowering memory usage and enabling faster inference on specialized hardware.

Method	Example	Description
INT4 Quantization	`4-bit integer weights` `~75% size reduction`	• Weights stored as 4-bit integers; reduces model size by 75% from FP16 • minimal accuracy loss (<3%) for most SLMs