Hugging Face Transformers Cheat Sheet

Updated 2026-04-28

Next Topic: In-context Learning Cheat Sheet

🧠Study flashcards on this topic116 cards · spaced repetition→

Hugging Face Transformers is a Python library that provides unified access to thousands of pretrained transformer models across natural language processing, computer vision, audio, and multimodal tasks. With the release of Transformers v5, the library went PyTorch-only, made quantization a first-class feature, and introduced a modular AttentionInterface plus a built-in transformers serve command for OpenAI-compatible inference. The Auto classes intelligently detect model architectures, while the Pipeline API and TRL post-training library give practitioners everything from instant inference to full RLHF alignment workflows.

What This Cheat Sheet Covers

This topic spans 19 focused tables and 188 indexed concepts, 116 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Core Loading and Auto ClassesTable 2: Pipeline API for InferenceTable 3: TokenizationTable 4: Model Training with TrainerTable 5: Fine-Tuning TechniquesTable 6: TRL Post-TrainingTable 7: Chat TemplatesTable 8: Model ArchitecturesTable 9: Datasets IntegrationTable 10: Text Generation ParametersTable 11: Model Saving and SharingTable 12: Optimization and QuantizationTable 13: Accelerate for Distributed TrainingTable 14: Callbacks and LoggingTable 15: Model Evaluation and MetricsTable 16: Advanced Tokenization FeaturesTable 17: Model Hub and Repository ManagementTable 18: Deployment and InferenceTable 19: Common Tasks and Use Cases

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Core Loading and Auto Classes

The Auto classes are the front door to the whole library — instead of importing a specific model class, you hand from_pretrained a checkpoint name and the right tokenizer, config, and architecture are resolved for you. Which AutoModelFor* you pick declares the task head you want bolted on, from causal generation to token classification, and the same pattern extends to multimodal processors and GGUF-quantized weights.

Class	Example	Description
AutoTokenizer	`from transformers import AutoTokenizer` `tokenizer = AutoTokenizer.from_pretrained("gpt2")`	• Automatically selects and loads the correct tokenizer for a given model • handles text-to-token conversion with model-specific vocabulary.
AutoModel	`from transformers import AutoModel` `model = AutoModel.from_pretrained("bert-base-uncased")`	• Loads the base model architecture from a checkpoint • returns raw hidden states without task-specific heads.
AutoModelForCausalLM	`from transformers import AutoModelForCausalLM` `model = AutoModelForCausalLM.from_pretrained("gpt2")`	• Loads model for causal (left-to-right) language modeling • predicts next token given previous context, used for text generation.
AutoModelForSequenceClassification	`model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=3)`	• Loads model with a classification head for labeling entire sequences • commonly used for sentiment analysis or topic classification.
AutoProcessor	`from transformers import AutoProcessor` `processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32")`	• Loads the unified processor for multimodal models • handles combined image + text inputs in a single interface, first-class in v5.
AutoConfig	`from transformers import AutoConfig` `config = AutoConfig.from_pretrained("t5-small")`	• Loads model configuration settings (hidden size, layers, attention heads) • enables architecture inspection without loading weights.
AutoModelForTokenClassification	`model = AutoModelForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english")`	• Loads model with per-token classification heads • used for named entity recognition (NER) or part-of-speech tagging.
AutoModelForQuestionAnswering	`model = AutoModelForQuestionAnswering.from_pretrained("distilbert-base-cased-distilled-squad")`	• Loads model for extractive question answering • predicts start and end positions of answer spans in context.
AutoModelForMaskedLM	`model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")`	• Loads model for masked language modeling (predicting masked tokens) • used for bidirectional pretraining tasks like BERT.
from_pretrained	`model = AutoModel.from_pretrained("model-id", torch_dtype=torch.float16, token="hf_...")`	• Core method to load model weights from Hub or local path • use `token=` (replaces deprecated `use_auth_token`) for private repos.
GGUF loading	`model = AutoModelForCausalLM.from_pretrained("TheBloke/TinyLlama-GGUF", gguf_file="tinyllama.Q6_K.gguf")`	• Loads GGUF-format quantized models directly into PyTorch for fine-tuning or inference • dequantizes to fp32 on load.

Table 2: Pipeline API for Inference

When you just want an answer and not a training loop, pipeline() is the shortcut — it wires up tokenization, the model, and postprocessing behind a single task name so one line of code runs sentiment analysis, transcription, or vision-language inference. Each row is a different task string you can pass, plus the handful of arguments (device, batch size) that matter most in practice.

Pipeline	Example	Description
pipeline	`from transformers import pipeline` `classifier = pipeline("sentiment-analysis")`	• High-level API that abstracts preprocessing, inference, and postprocessing • simplest way to run a model with one line of code.

The rest of this section is part of the full cheat sheet

Table 3: Tokenization

Models never see raw text — the tokenizer turns strings into the input IDs and attention masks the model actually consumes, and turns IDs back into text afterward. These methods cover the day-to-day mechanics: padding and truncation so batches line up, picking the return tensor format, handling special tokens, and reaching for the Rust-backed fast tokenizers when speed matters.

Method	Example	Description
tokenizer call	`tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")` `inputs = tokenizer("Hello world", return_tensors="pt")`	• Converts text to input IDs, attention masks, and token type IDs • returns dictionary ready for model input.

The rest of this section is part of the full cheat sheet

Table 4: Model Training with Trainer

The Trainer class hides the boilerplate of a training loop — device placement, gradient accumulation, checkpointing, logging — so you mostly configure rather than code. Almost everything here is a TrainingArguments field you set once (learning rate, batch size, evaluation and save strategy), with a few methods like train, evaluate, and predict to drive the run.

Component	Example	Description
Trainer	`from transformers import Trainer` `trainer = Trainer(model=model, args=training_args, train_dataset=train_ds)`	• High-level training loop that handles device placement, gradient accumulation, logging, and checkpointing • powered by Accelerate for distributed training.

The rest of this section is part of the full cheat sheet

Table 1: Core Loading and Auto Classes

Class	Example	Description
AutoTokenizer	`from transformers import AutoTokenizer` `tokenizer = AutoTokenizer.from_pretrained("gpt2")`	• Automatically selects and loads the correct tokenizer for a given model • handles text-to-token conversion with model-specific vocabulary.
AutoModel	`from transformers import AutoModel` `model = AutoModel.from_pretrained("bert-base-uncased")`	• Loads the base model architecture from a checkpoint • returns raw hidden states without task-specific heads.
AutoModelForCausalLM	`from transformers import AutoModelForCausalLM` `model = AutoModelForCausalLM.from_pretrained("gpt2")`	• Loads model for causal (left-to-right) language modeling • predicts next token given previous context, used for text generation.
AutoModelForSequenceClassification	`model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=3)`	• Loads model with a classification head for labeling entire sequences • commonly used for sentiment analysis or topic classification.
AutoProcessor	`from transformers import AutoProcessor` `processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32")`	• Loads the unified processor for multimodal models • handles combined image + text inputs in a single interface, first-class in v5.
AutoConfig	`from transformers import AutoConfig` `config = AutoConfig.from_pretrained("t5-small")`	• Loads model configuration settings (hidden size, layers, attention heads) • enables architecture inspection without loading weights.
AutoModelForTokenClassification	`model = AutoModelForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english")`	• Loads model with per-token classification heads • used for named entity recognition (NER) or part-of-speech tagging.
AutoModelForQuestionAnswering	`model = AutoModelForQuestionAnswering.from_pretrained("distilbert-base-cased-distilled-squad")`	• Loads model for extractive question answering • predicts start and end positions of answer spans in context.
AutoModelForMaskedLM	`model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")`	• Loads model for masked language modeling (predicting masked tokens) • used for bidirectional pretraining tasks like BERT.
from_pretrained	`model = AutoModel.from_pretrained("model-id", torch_dtype=torch.float16, token="hf_...")`	• Core method to load model weights from Hub or local path • use `token=` (replaces deprecated `use_auth_token`) for private repos.
GGUF loading	`model = AutoModelForCausalLM.from_pretrained("TheBloke/TinyLlama-GGUF", gguf_file="tinyllama.Q6_K.gguf")`	• Loads GGUF-format quantized models directly into PyTorch for fine-tuning or inference • dequantizes to fp32 on load.

Table 2: Pipeline API for Inference

Pipeline	Example	Description
pipeline	`from transformers import pipeline` `classifier = pipeline("sentiment-analysis")`	• High-level API that abstracts preprocessing, inference, and postprocessing • simplest way to run a model with one line of code.

Table 3: Tokenization

Method	Example	Description
tokenizer call	`tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")` `inputs = tokenizer("Hello world", return_tensors="pt")`	• Converts text to input IDs, attention masks, and token type IDs • returns dictionary ready for model input.

Table 4: Model Training with Trainer

Component	Example	Description
Trainer	`from transformers import Trainer` `trainer = Trainer(model=model, args=training_args, train_dataset=train_ds)`	• High-level training loop that handles device placement, gradient accumulation, logging, and checkpointing • powered by Accelerate for distributed training.