Softmax and Multinomial Logistic Regression

What to do when your problem has more than two classes

Logistic regression is one of the first models anyone learns when they get into machine learning, and for good reason: it’s simple, interpretable, and surprisingly powerful. With two classes (0 and 1), everything fits together beautifully: a sigmoid function squashes any number into a probability between 0 and 1, and a decision boundary separates the «yes» from the «no.»

But what happens when the world isn’t black and white? Flagging an email as spam or not-spam is two classes. But labeling an image as cat, dog, bird, or horse is four. Recognizing a handwritten digit is ten. Detecting the language of a piece of text could be fifty.

This is where the sigmoid falls short, and where Softmax steps in.

---

The core idea: splitting 100% of your confidence

Before we touch the formula, hold onto this picture. Imagine your model is a judge who has to split 100 points of confidence across all the possible classes. It can’t hand out 90 to «cat» and 90 to «dog»: the total always has to add up to 100. If it raises its confidence in one class, it has to lower it somewhere else.

That’s exactly what Softmax does: it takes the «raw» scores your model produces (numbers that can be negative, huge, or tiny) and turns them into a probability split that adds up to exactly 1.

---

What is Softmax?

Softmax is a function that takes a vector of numbers (one per class) and turns them into probabilities that sum to 1.

If your model produces a vector of scores (often called logits):

z=[z1​,z2​,z3​]

Softmax transforms it class by class with:

softmax(zi​)=∑j​ezj​ezi​​

The formula is intuitive if you read it in two pieces:

• The numerator $e^{z_i}$ exponentiates the class score. The exponential does two useful things: it turns any number (even a negative one) into a positive value, and it amplifies the differences (a slightly higher score gets a big boost).
• The denominator $\sum_j e^{z_j}$ adds up those exponentials across every class. Dividing by that total is what forces everything to sum to 1.

A worked example

Say your model has to choose between cat, dog, and bird, and it produces these scores:

$z=[2.0,1.0,0.1]$

On their own, these numbers don’t mean much. Let’s run them through Softmax:

• $e^{2.0} \approx 7.39$
• $e^{1.0} \approx 2.72$
• $e^{0.1} \approx 1.11$
• Total $\approx 11.22$

Divide each one by the total:

Now they mean something: the model is 66% sure it’s a cat, leaves 24% for dog, and 10% for bird. And notice they add up to 1.

Key properties

• Every probability lands between 0 and 1.
• The probabilities sum to exactly 1.
• The highest score becomes the highest probability (the ranking is preserved).
• Softmax is essentially the multiclass version of the sigmoid. In fact, if you apply Softmax to just two classes, you get the plain old sigmoid back.

---

What is Multinomial Logistic Regression?

Once you understand Softmax, this model is easy to sum up:

Multinomial logistic regression = logistic regression + Softmax + multiclass cross-entropy.

Here’s the full pipeline:

In plain words: the model produces a vector $z$ with one value per class, Softmax turns that vector into probabilities, and the loss checks how good the probability was that it assigned to the correct class. That error is then used to nudge the coefficients, and the whole thing repeats.

---

The loss function: multiclass cross-entropy

To train the model, we need a way to measure how wrong it is. That measure is cross-entropy:

CE=−∑i=1N​log(pi,correct class​)

Don’t let the formula scare you, the intuition is dead simple. You just look at the probability the model gave to the class that was actually correct, and apply a negative logarithm to it:

• If the model gave a high probability to the correct class → low loss (well done).
• If it gave a low probability to the correct class → high loss (bad).
• If it was very confident and wrong → enormous loss (the logarithm punishes confident mistakes harshly).

That last point is the key one: cross-entropy doesn’t just penalize getting it right or wrong, it penalizes being sure and still being wrong. That’s what pushes the model to calibrate its confidence properly. It’s, once again, the multiclass version of the log-loss you already knew from the binary case.

---

How is it trained?

Here’s some of the best news: pretty much the same way as binary logistic regression. There’s no new procedure to learn. It’s the same loop as always:

1. Gradient descent.
2. The error gets multiplied by the input features.
3. The coefficients get updated.
4. L1 or L2 regularization is added if you need it to avoid overfitting.

The only meaningful difference is that the error is no longer a single number, it’s now a vector (one error per class). In practice, the model learns a set of weights per class instead of a single one.

---

How do you interpret the decision boundary?

In the binary case you had a single line separating the 0s from the 1s. With more classes, the concept generalizes: instead of one boundary, the space gets carved into regions, one per class. At every point, Softmax decides which region «wins.»

Compared to other classifiers, the shape of those regions changes quite a bit:

Softmax draws linear boundaries between classes, which is why it gives such «clean» regions. That’s its great strength (simplicity and interpretability) and also its limit, as we’re about to see.

---

When should you use Softmax?

It’s a solid choice when:

• You have more than two mutually exclusive classes (each example belongs to exactly one).
• You want well-calibrated probabilities, not just a label.
• You need a model that’s linear, fast, and interpretable.
• You’re going to plug this into a neural network: Softmax is the standard output layer of almost every modern classifier.

When should you NOT use Softmax?

It’s worth reaching for a different tool when:

• The classes are not mutually exclusive (a photo can contain a dog and a ball at the same time). Softmax breaks here because it forces you to split a single 100%; the right approach is multi-label classification with independent sigmoids.
• The problem is highly non-linear: Softmax’s straight boundaries aren’t enough, and models like trees, random forests, or kernel SVMs will do better.

---

In a nutshell

• Softmax turns raw scores into probabilities that sum to 1.
• Multinomial logistic regression is logistic regression extended to several classes: a linear model + Softmax + cross-entropy.
• The loss is multiclass cross-entropy, which especially punishes being very confident and wrong.
• It’s trained with gradient descent, just like the binary version; the only difference is that the error is now a vector.
• It’s the foundation of the output layer in almost every modern neural network, which makes understanding it well an essential step on the road to deep learning.