Text-to-Speech (TTS) Synthesis Cheat Sheet

Updated 2026-05-25

Next Topic: Token Management Cheat Sheet

🧠Study flashcards on this topic134 cards · spaced repetition→

Text-to-Speech (TTS) synthesis is a branch of speech processing and deep learning that converts written text into natural-sounding human speech. The field has shifted from rule-based and concatenative systems to end-to-end generative models capable of producing highly realistic, expressive, emotionally rich speech — with the best systems now reaching human parity on standard benchmarks. Modern TTS typically involves two stages: converting text into intermediate acoustic representations (mel-spectrograms or discrete tokens), then generating raw audio through a vocoder or codec decoder. A critical insight for practitioners: the choice between autoregressive (higher naturalness, slower), non-autoregressive (faster, parallel), and flow/diffusion-based (probabilistic, controllable) paradigms fundamentally shapes the quality-speed trade-off, while the emergence of codec language models (VALL-E, Orpheus, Chatterbox) has opened a new paradigm treating TTS as conditional language modeling over discrete audio tokens.

What This Cheat Sheet Covers

This topic spans 17 focused tables and 152 indexed concepts, 134 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Foundational TTS ArchitecturesTable 2: Neural Vocoder ArchitecturesTable 3: Generative Model ParadigmsTable 4: Codec Language Models & Token-Based TTSTable 5: Text Preprocessing & Input RepresentationsTable 6: Acoustic Feature RepresentationsTable 7: Prosody & Expressive ControlTable 8: Multi-Speaker & Voice CloningTable 9: Duration & Alignment ModelingTable 10: Training Techniques & Loss FunctionsTable 11: Inference & Real-Time OptimizationTable 12: Quality Evaluation MetricsTable 13: Model Compression & DeploymentTable 14: Multilingual & Cross-Lingual TTSTable 15: Advanced Voice Cloning & AdaptationTable 16: Training Data & Dataset RequirementsTable 17: Responsible AI & Safety in TTS

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Foundational TTS Architectures

Modern TTS architectures span from seq2seq acoustic models to end-to-end flow-matching and LLM-based systems. Knowing which architecture underpins a model tells you what trade-offs to expect in terms of speed, quality, controllability, and voice cloning capability.

Model	Example	Description
VITS	`text → conditional VAE + flows → raw waveform`	• End-to-end model combining VAE, normalizing flows, and adversarial training to generate waveforms without a separate vocoder • achieves real-time synthesis and is widely used as a production backbone
Tacotron 2	`text → encoder → attention → decoder → mel-spectrogram`	• Foundational autoregressive seq2seq model generating mel-spectrograms frame-by-frame using location-sensitive attention • high quality but slow inference
FastSpeech 2	`text → pitch/energy/duration predictors → mel-spectrogram`	• Non-autoregressive model predicting pitch, energy, and duration directly from text • simpler pipeline and faster inference than its predecessor
StyleTTS 2	`text → style diffusion → SLM discriminator → waveform`	• First TTS to surpass human recordings on LJSpeech • models speaking style as a latent random variable via diffusion and uses WavLM as a discriminator for end-to-end training
F5-TTS	`text (padded with fillers) → DiT flow matching → mel-spectrogram`	• Fully non-autoregressive flow-matching model using a Diffusion Transformer • no duration model or phoneme aligner needed • RTF 0.15, trained on 100K multilingual hours with Sway Sampling inference
FastSpeech	`text → FFT blocks → length regulator → mel-spectrogram`	• First non-autoregressive TTS using explicit duration prediction and parallel generation • ~270× faster than Tacotron 2, superseded by FastSpeech 2.
Matcha-TTS	`text → OT-CFM decoder → mel-spectrogram`	• Encoder-decoder using optimal-transport conditional flow matching (OT-CFM) • probabilistic, non-autoregressive, no external alignments, smallest memory footprint among comparable models
MaskGCT	`text → semantic tokens → masked acoustic tokens`	• Fully non-autoregressive two-stage model: predicts SSL semantic tokens from text, then reconstructs acoustic tokens • no alignment or phone-level duration prediction • trained on 100K hours
YourTTS	`text + speaker embedding → zero-shot synthesis`	• VITS-based multi-speaker multilingual model supporting zero-shot voice cloning • fine-tunes to a new speaker with under 1 minute of speech
Glow-TTS	`text → flow-based model → mel-spectrogram`	Non-autoregressive flow-based model using monotonic alignment search (MAS) to learn alignment without an external aligner.
Transformer TTS	`text → multi-head attention → decoder → mel-spectrogram`	Replaces RNN layers with self-attention for better long-range dependency modeling and faster training on parallel hardware.
Grad-TTS	`text → encoder → diffusion decoder → mel-spectrogram`	• Uses score-based diffusion to gradually denoise mel-spectrograms from Gaussian noise • quality improves with more diffusion steps

Table 2: Neural Vocoder Architectures

The vocoder is the component that converts intermediate acoustic representations (mel-spectrograms or codec tokens) into raw audio waveforms. Vocoder choice determines the ceiling on audio fidelity and is often the primary bottleneck for inference speed.

Vocoder	Example	Description
HiFi-GAN	`mel-spec → multi-period + multi-scale discriminators → waveform`	• State-of-the-art GAN vocoder with hierarchical discriminators • generates 22.05 kHz audio at 167× real-time on a V100 • best balance of quality and speed and the most widely deployed vocoder today

The rest of this section is part of the full cheat sheet

Table 3: Generative Model Paradigms

The generative paradigm defines how a TTS model produces outputs — whether sequentially, in parallel, or by denoising from noise. Understanding these paradigms is essential for choosing the right architecture and anticipating its behavior at training and inference time.

Approach	Example	Description
Non-autoregressive	FastSpeech, MaskGCT generate all frames in parallel	• All output elements generated simultaneously • 100–300× faster inference • requires explicit duration modeling or masked prediction to determine output length

The rest of this section is part of the full cheat sheet

Table 4: Codec Language Models & Token-Based TTS

Codec language models treat TTS as a conditional language modeling task over discrete audio tokens rather than continuous mel-spectrogram regression. This paradigm scales with data and unlocks powerful in-context voice cloning — a speaker's voice can be cloned from just 3–10 seconds of audio without any fine-tuning.

Model	Example	Description
VALL-E	`3-second prompt → codec LM → EnCodec tokens → audio`	• Pioneering codec language model treating TTS as conditional LM over EnCodec tokens • trained on 60K hours • zero-shot voice cloning from 3-second prompts preserving speaker emotion and environment

The rest of this section is part of the full cheat sheet

Table 1: Foundational TTS Architectures

Model	Example	Description
VITS	`text → conditional VAE + flows → raw waveform`	• End-to-end model combining VAE, normalizing flows, and adversarial training to generate waveforms without a separate vocoder • achieves real-time synthesis and is widely used as a production backbone
Tacotron 2	`text → encoder → attention → decoder → mel-spectrogram`	• Foundational autoregressive seq2seq model generating mel-spectrograms frame-by-frame using location-sensitive attention • high quality but slow inference
FastSpeech 2	`text → pitch/energy/duration predictors → mel-spectrogram`	• Non-autoregressive model predicting pitch, energy, and duration directly from text • simpler pipeline and faster inference than its predecessor
StyleTTS 2	`text → style diffusion → SLM discriminator → waveform`	• First TTS to surpass human recordings on LJSpeech • models speaking style as a latent random variable via diffusion and uses WavLM as a discriminator for end-to-end training
F5-TTS	`text (padded with fillers) → DiT flow matching → mel-spectrogram`	• Fully non-autoregressive flow-matching model using a Diffusion Transformer • no duration model or phoneme aligner needed • RTF 0.15, trained on 100K multilingual hours with Sway Sampling inference
FastSpeech	`text → FFT blocks → length regulator → mel-spectrogram`	• First non-autoregressive TTS using explicit duration prediction and parallel generation • ~270× faster than Tacotron 2, superseded by FastSpeech 2.
Matcha-TTS	`text → OT-CFM decoder → mel-spectrogram`	• Encoder-decoder using optimal-transport conditional flow matching (OT-CFM) • probabilistic, non-autoregressive, no external alignments, smallest memory footprint among comparable models
MaskGCT	`text → semantic tokens → masked acoustic tokens`	• Fully non-autoregressive two-stage model: predicts SSL semantic tokens from text, then reconstructs acoustic tokens • no alignment or phone-level duration prediction • trained on 100K hours
YourTTS	`text + speaker embedding → zero-shot synthesis`	• VITS-based multi-speaker multilingual model supporting zero-shot voice cloning • fine-tunes to a new speaker with under 1 minute of speech
Glow-TTS	`text → flow-based model → mel-spectrogram`	Non-autoregressive flow-based model using monotonic alignment search (MAS) to learn alignment without an external aligner.
Transformer TTS	`text → multi-head attention → decoder → mel-spectrogram`	Replaces RNN layers with self-attention for better long-range dependency modeling and faster training on parallel hardware.
Grad-TTS	`text → encoder → diffusion decoder → mel-spectrogram`	• Uses score-based diffusion to gradually denoise mel-spectrograms from Gaussian noise • quality improves with more diffusion steps

Table 2: Neural Vocoder Architectures

Vocoder	Example	Description
HiFi-GAN	`mel-spec → multi-period + multi-scale discriminators → waveform`	• State-of-the-art GAN vocoder with hierarchical discriminators • generates 22.05 kHz audio at 167× real-time on a V100 • best balance of quality and speed and the most widely deployed vocoder today

Table 3: Generative Model Paradigms

Approach	Example	Description
Non-autoregressive	FastSpeech, MaskGCT generate all frames in parallel	• All output elements generated simultaneously • 100–300× faster inference • requires explicit duration modeling or masked prediction to determine output length

Table 4: Codec Language Models & Token-Based TTS

Model	Example	Description
VALL-E	`3-second prompt → codec LM → EnCodec tokens → audio`	• Pioneering codec language model treating TTS as conditional LM over EnCodec tokens • trained on 60K hours • zero-shot voice cloning from 3-second prompts preserving speaker emotion and environment