NVIDIA TensorRT is a high-performance deep learning inference SDK that compiles trained neural networks into optimized inference engines for NVIDIA GPUs. It sits at the intersection of model deployment and hardware-level execution, transforming framework-trained models — typically exported as ONNX — into GPU-specific execution plans that exploit layer fusion, precision calibration, and kernel auto-tuning. The core value proposition is throughput and latency reduction: a model that runs at 100 ms in PyTorch may execute in under 10 ms after TensorRT compilation. The critical mental model is that TensorRT builds once, runs many times — the expensive tactic selection and kernel benchmarking happen at engine build time, so the resulting serialized engine is a highly specialized binary artifact tied to a specific GPU architecture, TensorRT version, and input shape range.
What This Cheat Sheet Covers
This topic spans 17 focused tables and 131 indexed concepts, 101 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.
A jump-to index of every table row in this cheat sheet.
An interactive map of every table and concept in this topic.
Table 1: Core Build-Phase API Classes
The three objects you construct every time you build a TensorRT engine are the Builder, the Network, and the BuilderConfig. Understanding each class's responsibility — and the order in which they are used — is the minimum prerequisite for any TensorRT workflow.
| Class | Example | Description | |
|---|---|---|---|
builder = trt.Builder(logger)engine_bytes = builder.build_serialized_network(network, config) | • Top-level factory that produces the serialized engine • also creates the Network and Config objects via create_network() and create_builder_config(). | ||
network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED)) | • Graph container where layers and tensors are defined • must be created with STRONGLY_TYPED or EXPLICIT_BATCH flag in TensorRT 10+. | ||
config = builder.create_builder_config()config.set_flag(trt.BuilderFlag.FP16) | Controls all build-time knobs: precision flags, timing cache, workspace memory, auxiliary streams, optimization level, and hardware compatibility. | ||
engine = runtime.deserialize_cuda_engine(engine_bytes) | • Deserialized, executable engine object • tied to one GPU architecture • exposes binding names and tensor info | ||
runtime = trt.Runtime(logger)engine = runtime.deserialize_cuda_engine(plan) | • Deserializes serialized engine bytes into a runnable ICudaEngine• must set engine_host_code_allowed = True for version-compatible engines | ||
class MyLogger(trt.ILogger): def log(self, sev, msg): ... | • Abstract logging interface passed to Builder and Runtime • severity levels: VERBOSE, INFO, WARNING, ERROR, INTERNAL_ERROR |