NVIDIA TensorRT for Inference Optimization Cheat Sheet

NVIDIA TensorRT is a high-performance deep learning inference SDK that compiles trained neural networks into optimized inference engines for NVIDIA GPUs. It sits at the intersection of model deployment and hardware-level execution, transforming framework-trained models — typically exported as ONNX — into GPU-specific execution plans that exploit layer fusion, precision calibration, and kernel auto-tuning. The core value proposition is throughput and latency reduction: a model that runs at 100 ms in PyTorch may execute in under 10 ms after TensorRT compilation. The critical mental model is that TensorRT builds once, runs many times — the expensive tactic selection and kernel benchmarking happen at engine build time, so the resulting serialized engine is a highly specialized binary artifact tied to a specific GPU architecture, TensorRT version, and input shape range.