Regularisation: Why Can a Less Exact Fit Be More Reliable?

Regularisation means training a model not only to explain the available data, but also to pay a cost for an excessively complex solution. It deliberately gives up a perfect fit to the training examples in exchange for more stable judgements on new data.

Suppose a curve is made to pass through every measured point, including measurement error. Its training error may be tiny, yet its sharp bends may merely memorise noise. With regularisation, the model tends towards a smoother curve. It may fit the old observations less exactly while better preserving the stable relationship that produced them.

Regularisation is not normalisation, which mainly changes the scales of variables. Nor does it simply mean making every model smaller: a constraint may penalise large parameters, too many features or excessive structural depth. Too little regularisation still permits overfitting; too much produces underfitting. The question is not whether simpler is always better, but which complexity the evidence supports. A reliable model does not remember everything; it refuses to treat accidental detail as a rule.

https://scikit-learn.org/stable/auto_examples/model_selection/plot_train_error_vs_test_error.html