6.3. Logistic Regression

Logistic regression, despite its name, it a classification technique. We have seen that the Bayes classifier assigns the class \(\hat y = \classify(\v c)\) to an object characterized with feature vector \(\v x\) based on:

\[\classify(\v x) = \arg\max_y \P(Y=y\given \v X=\v x)\]

For the Bayesian classifier the a posteriori probability \(\P(Y=y\given \v X = \v x)\) is then expressed in the class conditional probabilities of the data and the a priori probabilities of the classes. Effectively the joint distribution of the feature vector and class is estimated. A Bayesian classifier is therefore a generative classifier (given the joint distribution you could generate examples).

The logistic regression classifier is an example of a discriminative classifier and it starts with the same a posteriori probability as given above. But unlike the Bayes classifier it does not calculate this a posteriori probability from (an estimate of) the joint distribution but it estimates the a posteriori probability directly from the training set using a ‘simple’ parameterized model. As such the logistic regression classifier concentrates more on the decision boundaries of the different classes and not on an estimator for the probability.

• 6.3.1. Logistic Regression Model
• 6.3.2. Maximum Likelihood Estimator
• 6.3.3. Gradient Descent
• 6.3.4. Vectorized Logistic Regression Algorithm
• 6.3.5. Extended Features in Logistic Regression
• 6.3.6. Multi Class Logistic Regression
  • 6.3.6.1. One vs All Multi Class
  • 6.3.6.2. One vs One Multi Class
  • 6.3.6.3. Softmax Multi Class