What Is a Random Forest?

Introduced by Leo Breiman in 1999, Random Forests became the go-to method for tabular data throughout the 2000s.

Random Forests vs. Logistic Regression

Logistic regression is often called the "simple baseline" — but it's surprisingly fragile in practice.

FastAI Setup & Titanic Preprocessing

One import brings in NumPy, pandas, and matplotlib automatically:

from fastai.imports import *

# Download Titanic train + test
path = untar_data(URLs.TITANIC)
df = pd.read_csv(path/'train.csv')

Essential preprocessing steps

• Fill missing numerics with median/mode — trees need no NaN values
• Log-transform Fare → log_fare = np.log1p(Fare) for a nicer distribution in plots
• Convert Sex and Embarked to category dtype
• Separate features into categorical and continuous lists

Pandas category dtype & cat_codes

Converting a string column to category is a critical step:

df['Sex'] = df['Sex'].astype('category')

# Inspect what pandas stores internally:
df['Sex'].cat.categories   # → Index(['female', 'male'])
df['Sex'].cat.codes        # → 0 = female, 1 = male

What Is a Binary Split?

A binary split partitions all rows into exactly two groups based on a single rule:

The Titanic Sex split — a very strong rule

With just one yes/no question, we can predict survival with reasonable accuracy. This is the entire basis of decision trees — we just keep asking more questions.

Scoring a Split — The Variance Method

How do we know if a split is good? We measure how "pure" each resulting group is using its standard deviation:

The OneR Model

OneR = "One Rule" — find the single best binary split across all features and stop there.

Why start so simple?

• A 1990s meta-analysis found OneR often matched or beat more complex models
• It creates a baseline to beat — if your complex model doesn't clearly improve on this, something's wrong
• It's transparent and explainable — you can show a business stakeholder exactly what the model does

From One Split to a Full Tree

A decision tree is nothing more than recursively applied binary splits. After the first split (Sex), we split each group again:

After two levels of splitting we have 4 leaf nodes — a proper decision tree.

Building a Tree with scikit-learn

from sklearn.tree import DecisionTreeClassifier, plot_tree
import matplotlib.pyplot as plt

tree = DecisionTreeClassifier(
    max_leaf_nodes=4,
    random_state=42
).fit(X_train, y_train)

fig, ax = plt.subplots(figsize=(12, 5))
plot_tree(tree, feature_names=X_train.columns,
          class_names=['Died','Survived'], filled=True, ax=ax)
plt.show()

Key parameter: max_leaf_nodes

• Controls tree depth / capacity — the main knob for bias-variance tradeoff
• Start small (4–8 leaves) to understand the model, then increase
• Alternative: min_samples_leaf=50 — each leaf must have ≥ 50 rows

Interpreting the 4-Leaf Tree

Gini Impurity — scikit-learn's Split Criterion

scikit-learn uses Gini impurity to score splits (instead of our variance method). For a binary target:

where p is the proportion of class-1 in the node.

How Well Does a Single Tree Perform?

Why can one rule sometimes beat a tree?

• With only ~890 training rows, deeper splits create tiny, noisy leaf nodes
• Small leaves → unstable predictions → higher variance
• This is the core problem that Random Forests solve

The Problem with a Single Tree

• Deeper splits → smaller leaf nodes → fewer rows per leaf
• Tiny leaves give unstable, high-variance predictions
• Accuracy plateaus; adding more depth eventually overfits

Leo Breiman's insight — Bagging

What if we built many trees and averaged them? Errors that are random will cancel out; only the signal remains.

Building a Random Forest — Step by Step

A Random Forest is bagged decision trees with an extra twist: random feature subsets at each split.

Building a Random Forest in scikit-learn

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(
    n_estimators=100,       # number of trees
    min_samples_leaf=5,     # minimum rows per leaf
    max_features=0.5,       # random subset of features per split
    oob_score=True,         # built-in validation (explained next)
    n_jobs=-1,              # use all CPU cores
    random_state=42
).fit(X_train, y_train)

print(f"OOB accuracy: {rf.oob_score_:.3f}")
print(f"Validation MAE: {mean_absolute_error(y_val, rf.predict(X_val)):.3f}")

Results on Titanic

How Many Trees Do You Need?

More trees always help — error decreases monotonically as you add trees. It never gets worse.

Out-of-Bag (OOB) Error — Free Validation

Because each tree only sees a bootstrapped subset of data, approximately 25–37% of rows are left out for any given tree. These are the "out-of-bag" rows.

Feature Importance

For every split in every tree, track how much the Gini impurity improved × how many rows were in that node. Sum across all trees → feature importance score.

import matplotlib.pyplot as plt

importances = pd.Series(rf.feature_importances_, index=X_train.columns)
importances.sort_values().plot.barh(figsize=(8, 5), color='#C0272D')
plt.title('Feature Importances')
plt.tight_layout()
plt.show()

Titanic result

Sex ≫ Pclass ≈ Fare ≫ Age ≫ Embarked

Partial Dependence Plots (PDPs)

Feature importance tells you what matters. PDPs tell you how a feature affects the prediction, all else held equal.

How to compute a PDP for one feature

1. Take the validation set
2. Set the target column to a fixed value (e.g., Age = 5) for all rows
3. Run the forest and average all predictions → one point on the curve
4. Repeat for every candidate value across the feature's range → full curve

Explaining a Single Prediction

Feature importance explains the model overall. But sometimes you need to explain why this specific passenger was predicted to die.

Tree-path contribution method

1. For one row, trace its path through every tree in the forest
2. At each split, record: which feature was used, and how much it changed the prediction
3. Sum contributions across all trees → per-feature Δ prediction for that row

Bagging vs. Boosting

Two fundamentally different ensemble strategies — know when to use each.

Breiman's "Two Cultures" — A Final Thought

Leo Breiman (2001) argued there are two cultures in statistical modelling:

The Kaggle Iteration Mindset

Jeremy Howard's approach to competition — applicable to every real project.

Submission workflow

test_pred = rf.predict(X_test)
sub = pd.DataFrame({
    'PassengerId': test_df.PassengerId,
    'Survived': test_pred
})
sub.to_csv('submission_rf.csv', index=False)
# → Upload to Kaggle. Repeat. Improve.

What We Covered Today