While working through ISLR exercise 4.8.3 for my ML class, I realized LDA quietly makes one assumption: every class shares the same variance. This note is me convincing myself, with pictures, why dropping that single assumption bends the decision boundary.
One feature X, K classes. Class k generates data from a normal distribution N(μk, σ²k). The LDA derivation in ISLR 4.4 assumes σ²₁ = … = σ²K. Drop that assumption and show the Bayes classifier is quadratic, not linear.
Say class A and class B have bell curves of the same width. The classification rule is simple: at any x, the taller curve wins. Equal widths means the two curves cross exactly once, and that crossing is the boundary. One boundary point means everything left of it is A and everything right of it is B. That is a linear classifier — LDA.
Now make A narrow and B wide. Drag the slider below and watch the boundary.
The key scene is the far left. As x moves left, away from A's mean, the narrow A curve crashes to zero quickly, but the wide B curve decays slowly. So at some point B becomes taller again. Result: two boundary points, and the regions split as B | A | B. No single point (no line) can draw that shape. An equation with up to two roots is a quadratic.
The Bayes classifier picks the class with the largest πk·fk(x). Taking logs does not change which one is largest, so we can expand each class's "score" δk(x) = log(πk fk(x)). Plugging in the normal density:
Everything else is a constant. The whole argument lives in one place: the x² coefficient, −1/(2σ²k).
Figure 3 shows exactly this. Plot the score difference δA(x) − δB(x): with equal variances it is a straight line crossing zero once; with different variances it is a parabola crossing zero twice.
What I'll remember: when someone says "QDA is more flexible than LDA," the entire difference is a single subscript on σ.