Sunmin.Kim
regularization · ISLR 6.2 · 2026-10-25

Why Lasso Hits Zero

I knew the slogan "ridge shrinks, lasso selects," but I could not say why. So I took six candidate penalties, drew each one, and asked two questions about its shape. This note is the picture that finally made it click for me.

The setup

Regularized least squares adds a penalty R to the usual error, and λ sets how hard the penalty bites:

J(β) = Σ (yi − ŷi)² + λ · R(β)

To see the shape of the problem, I shrank it to one coefficient. With a standardized predictor, the error term is just a parabola centered at the least squares estimate, which I call b:

J(β) = (β − b)² + λ · R(β)

Now every question about R becomes a question about its graph. I only needed two.

Question 1: does it pull β toward zero?

A penalty is a fine for leaving home, where home is β = 0. A sensible fine is smallest at 0 and grows as |β| grows, in both directions. If the fine ever turns negative, it becomes a reward, and the optimizer runs toward the reward instead of home.

|β| rises on both sides, so it pulls β home. β³ drops forever on the left, so β escapes to −∞. sin β waves up and down and drags β toward −π/2, not 0.

Question 2: is there a corner at zero?

This is the part I had been missing. Zoom in near β = 0 and compare how much each penalty charges for a tiny coefficient.

At β = 0.1, |β| charges 0.1, β² charges 0.01, and β⁴ charges 0.0001. A V-shaped penalty still pushes hard on a tiny β, so the cheapest move is to set it to exactly 0. A round penalty barely notices a tiny β, so it only shrinks.

In slope terms: a corner means the slope stays at ±1 right up to 0, so there is always a constant push toward 0. A smooth bottom means the slope fades to 0 as β → 0, so the push disappears before β gets there.

Try it: the one-coefficient lab

Pick a penalty, then move b (what the data wants) and λ (how strict the penalty is). The dot is the β that actually minimizes J. Start with b = 0.3 and λ = 1, then flip between |β| and β⁴.

J(β) error (β − b)² penalty λR(β)
With |β| the threshold is λ/2: any b smaller than that in size gets set to exactly 0 (soft thresholding). With β⁴ the minimizer only creeps toward 0.

The scorecard

R(β)Pulls toward 0?Corner at 0?Why
β³No–Negative for β < 0, so the minimum runs off to −∞.
sin βNo–Periodic with negative dips; it pulls β toward −π/2.
β⁴YesNoGrows both ways but is flatter than ridge near 0. Shrinks only.
|β|YesYesLasso. Constant slope ±1 all the way to 0.
ln(1+|β|)YesYesActs like |β| near 0, but grows slowly, so large coefficients are shrunk less.
‖β‖₀YesYes, the strongestCosts 1 for any nonzero β. Pure selection (best subset), but discontinuous and hard to optimize.

One-line takeaway

A penalty shrinks if it grows away from 0. It selects only if it has a corner at 0. Lasso hits zero for the same reason a marble stops dead at the bottom of a V but keeps rolling around the bottom of a bowl.

What I'll remember: before trusting any new regularizer, draw it once and look at β = 0.