Running AI models efficiently requires understanding hardware architecture, memory systems, and software optimization techniques that bridge the gap between training and deployment. Modern AI hardware has evolved from general-purpose GPUs to specialized accelerators with dedicated tensor cores, high-bandwidth memory, and custom instruction sets optimized for matrix operations. Whether deploying in the cloud or at the edge, choosing the right combination of hardware capabilities, quantization formats, and inference frameworks determines latency, throughput, cost, and energy efficiency—often with orders of magnitude differences between optimal and naive configurations.
What This Cheat Sheet Covers
This topic spans 14 focused tables and 93 indexed concepts, 88 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.
A jump-to index of every table row in this cheat sheet.
An interactive map of every table and concept in this topic.
Table 1: GPU Architecture Components for AI
GPU architecture for machine learning centers on parallel processing units optimized for matrix operations. CUDA cores handle general parallel tasks, while tensor cores accelerate AI-specific workloads with specialized matrix multiply-accumulate operations. Memory hierarchy—from on-chip registers to HBM—determines how quickly data flows to compute units, with bandwidth often becoming the bottleneck in inference-heavy workloads.
| Component | Example | Description | |
|---|---|---|---|
10,752 cores in RTX 4090 | • General-purpose parallel processing units that execute floating-point and integer operations • handle tasks like data preprocessing and non-matrix computations | ||
512 Tensor Cores in H100 | • Specialized hardware units that accelerate matrix multiply-accumulate (MMA) operations for AI workloads • deliver up to 3× higher throughput than CUDA cores for deep learning inference | ||
132 SMs in H100 with 4 warp schedulers each | • Cluster of cores plus shared memory that executes warps (groups of 32 threads) • each SM contains CUDA cores, tensor cores, registers, and L1 cache for context-switch-free multitasking | ||
4 warp schedulers per SM | • Hardware scheduler that selects which group of 32 threads to execute each clock cycle • hides memory latency by switching between warps with zero overhead using massive register files | ||
128-byte aligned access across 32 threads | • Technique to combine multiple memory requests from a warp into fewer transactions • properly aligned contiguous access patterns prevent 20–40% bandwidth loss from uncoalesced reads | ||
HBM3e delivers 4.8 TB/s on H200HBM3 provides 3.35 TB/s on H100 | • Stacked memory chips directly integrated with the GPU die via interposer • provides 10–15× higher bandwidth than GDDR while consuming less power per bit transferred | ||
5.3 TB/s on AMD MI300X with 192GB HBM3 | • Rate at which data moves between memory and GPU cores • often the primary bottleneck in LLM inference where model weights exceed cache capacity and must be streamed repeatedly | ||
NVLink 5 provides 1.8 TB/s per GPUNVLink 6 (Rubin) scales to 3.6 TB/s | • High-speed GPU-to-GPU interconnect that enables direct peer memory access • bypasses CPU and PCIe for multi-GPU systems with 14× higher bandwidth than PCIe Gen6 |