Linear Discriminant Analysis

My notes on LDA from the textbook "An Introduction to Statistical Learning"

Linear Discriminant Analysis (LDA)

\[Pr(Y=k|X=x)=\frac{\pi_k \frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{1}{2}\left( \frac{x-\mu_k}{\sigma} \right)^2}}{\sum_{l=1}^{K}\pi_l\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{1}{2}\left( \frac{x-\mu_l}{\sigma} \right)^2}}\]

• Does a very good job at determining the probability than an observation $\left(Y,X\right)$ belong to a certain class given a value of $X$.
• The log odds derived from LDA is a linear function of predictor $X$.
• This linearity is a consequence of the Gaussian assumption for the class densities, as well as the assumption of a common covariance matrix.
• As seen, LSA and logistics regression have the same form of log odds. Both of whicha re linear functions of $x$. The only difference is the way the parameters are estimated. This same connection between LDA and logistic regression also holds for multidimensional data wit multiple predictors.
• In general, logistic regression is safer than LDA because LDA has strong assumptions about the normal distribution of $X$ and strong assumptions about the covariance matrix.
• LDA is useful:
  • when $n$ is small
  • or the classes are well separated
  • and Gaussian assumptions are reasonable