Feature engineering is the process of transforming raw data into meaningful features that improve machine learning model performance. It sits at the intersection of domain knowledge and data science, converting observations into numeric representations that algorithms can interpret. While automated approaches exist, manual feature engineering remains critical—selecting the right transformations, encoding strategies, and scaling methods often determines whether a model achieves mediocre or exceptional results. A key principle is always fitting transformations on training data only, then applying them to test data to prevent data leakage. Understanding these techniques empowers you to extract maximum signal from your data.
What This Cheat Sheet Covers
This topic spans 19 focused tables and 123 indexed concepts, 121 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.
A jump-to index of every table row in this cheat sheet.
An interactive map of every table and concept in this topic.
Table 1: Categorical Encoding Methods
Most algorithms only understand numbers, so any text category—colors, job titles, product types—has to be converted before training. The right choice hinges on whether the categories have a natural order, how many distinct values exist, and whether you're willing to let the target variable inform the encoding. These methods range from the everyday workhorses like one-hot and label encoding to specialized tools for high cardinality and dirty, misspelled data.
| Method | Example | Description | |
|---|---|---|---|
['red','blue'] →[[1,0],[0,1]] | • Creates binary column for each category • most common encoding for nominal variables with low cardinality | ||
['low','med','high'] →[0, 1, 2] | • Maps categories to integers • suitable for ordinal variables where order matters or tree-based models | ||
cat → mean(target|cat) | • Replaces category with target mean for that group • powerful for high cardinality but risks overfitting without cross-fitting | ||
['small','medium','large'] →[1, 2, 3] | • Assigns ordered integers based on inherent ranking • preserves ordinal relationships | ||
cat → count(cat)/total | • Encodes by occurrence frequency • useful when frequency correlates with target | ||
[0,1,2,3] →[[0,0],[0,1],[1,0],[1,1]] | • Converts integers to binary digits as columns • reduces dimensionality vs one-hot for high cardinality | ||
cat → tree.predict(cat) | • Fits a decision tree per feature to predict target, replaces categories with tree leaf predictions • produces monotonic relationship with target | ||
Neural network learns cat → [0.2, 0.8, 0.1] | • Neural network layer maps categories to dense vectors • captures similarities in learned representations | ||
cat → ln(P(good)/P(bad)) | • Weight of Evidence from credit scoring • measures predictive power via log odds; binary classification only | ||
hash(cat) % n_features | • Applies hash function to map categories to fixed buckets • memory-efficient but irreversible | ||
Infrequent cats → "Rare" | • Groups categories below a frequency threshold into a single "Rare" label • reduces cardinality and handles unseen categories | ||
"bachlors" ≈ "bachelors" →0.88 | • Replaces categories with string similarity scores between category names • handles dirty data with misspellings and abbreviations |