I knew the slogan "ridge shrinks, lasso selects," but I could not say why. So I took six candidate penalties, drew each one, and asked two questions about its shape. This note is the picture that finally made it click for me.
Regularized least squares adds a penalty R to the usual error, and λ sets how hard the penalty bites:
To see the shape of the problem, I shrank it to one coefficient. With a standardized predictor, the error term is just a parabola centered at the least squares estimate, which I call b:
Now every question about R becomes a question about its graph. I only needed two.
A penalty is a fine for leaving home, where home is β = 0. A sensible fine is smallest at 0 and grows as |β| grows, in both directions. If the fine ever turns negative, it becomes a reward, and the optimizer runs toward the reward instead of home.
This is the part I had been missing. Zoom in near β = 0 and compare how much each penalty charges for a tiny coefficient.
In slope terms: a corner means the slope stays at ±1 right up to 0, so there is always a constant push toward 0. A smooth bottom means the slope fades to 0 as β → 0, so the push disappears before β gets there.
Pick a penalty, then move b (what the data wants) and λ (how strict the penalty is). The dot is the β that actually minimizes J. Start with b = 0.3 and λ = 1, then flip between |β| and β⁴.
| R(β) | Pulls toward 0? | Corner at 0? | Why |
|---|---|---|---|
| β³ | No | – | Negative for β < 0, so the minimum runs off to −∞. |
| sin β | No | – | Periodic with negative dips; it pulls β toward −π/2. |
| β⁴ | Yes | No | Grows both ways but is flatter than ridge near 0. Shrinks only. |
| |β| | Yes | Yes | Lasso. Constant slope ±1 all the way to 0. |
| ln(1+|β|) | Yes | Yes | Acts like |β| near 0, but grows slowly, so large coefficients are shrunk less. |
| ‖β‖₀ | Yes | Yes, the strongest | Costs 1 for any nonzero β. Pure selection (best subset), but discontinuous and hard to optimize. |
What I'll remember: before trusting any new regularizer, draw it once and look at β = 0.