Loss functions: MSE and cross-entropy

In short: The loss function is a number that measures how wrong a model’s prediction is. Training means making this number smaller.

In more detail: Without a measurable goal nothing can be optimised. The choice of loss function determines what the model understands as “good”.

In detail

MSE (mean squared error): (1/n) Σ (y − ŷ)². For regression. Large errors count disproportionately.
MAE: mean absolute error, less sensitive to outliers.
Cross-entropy: −Σ y·log(p). For classification. The more confident the model is about the wrong class, the larger the loss. For language models: negative logarithm of the probability of the correct next token.
Perplexity: e^(loss), intuitively the “effective number of equally likely alternatives” when guessing the next token.
Hinge loss, Huber loss, contrastive losses (for embeddings) are others.
Regularisation: extra penalty terms against excessively large weights.
Whole picture: loss → gradient → adjust weights, see Gradient descent and learning from data.
Caution: a low loss does not automatically mean a useful model. A wrong goal leads to misbehaviour (“reward hacking”).