Training error is a rigged exam — the model graded itself on questions it studied. Only held-out error measures learning.
Underfit = too little capacity (high bias): wrong on everything, including training data. Overfit = too much (high variance): perfect on training, lost on test.
The cures for overfitting are boring and reliable: more data, regularisation (Ridge/Lasso), early stopping, simpler models, dropout.
Learning curves diagnose which disease you have: both errors high and close → bias; train low and test high → variance.
In the Test Kitchen: crank the degree to 9, then hit NEW PLATES — the wild wiggles change completely. A model that changes that much was never describing the dish.
Don't just read the recipe — taste it. Drag, click and break things below.
Filled dots are training plates the chef tastes; hollow ones are test plates held back. Raise the polynomial degree: training error always falls, but past the sweet spot the test error turns and climbs — the recipe fits the noise, not the dish. That gap is overfitting.
verdict: under-seasoned (high bias)
FIG L.8: BIAS–VARIANCE — TRAIN ERROR ONLY EVER FALLS; TEST ERROR IS U-SHAPED. TRUST THE HOLLOW PLATES
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
for degree in (1, 3, 9):
model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
model.fit(X_train, y_train)
print(degree,
mse(y_train, model.predict(X_train)), # always shrinks
mse(y_test, model.predict(X_test))) # U-shaped — trust this one