Outliers and missing values: Tidying data before training

In short: Before models train, missing values must be handled and outliers examined; otherwise they distort means and results.

In more detail: Real data is incomplete and contains errors and extremes. How you handle them often decides the results more than the choice of model. The goal is not to make data “pretty” but to make justified decisions and document them.

In Depth

Missing values

First ask: why is the value missing? (random, systematic, not collected?). Options:

• Delete rows or columns if few are affected.
• Replace (imputation) by mean, median (more robust), most frequent value, last valid value (time series) or by model (KNN imputer).
• Create an indicator (“was missing”), because missingness can itself be informative.

df["age"] = df["age"].fillna(df["age"].median())
df.isna().sum()

Impute only after splitting into training and test with values from training to avoid data leakage.

Detecting outliers

• Box-plot rule (IQR): values beyond Q1 − 1.5·IQR or Q3 + 1.5·IQR.
• Z-score: more than 3 standard deviations from the mean (only sensible for roughly normally distributed data).
• Visualisation: histogram, box plot, scatter plot.
• Models: Isolation Forest, DBSCAN, Local Outlier Factor.

What to do with outliers?

First check whether it is a measurement or typing error (age 250, weight −5) or a genuine rare case (millionaire in income data, fraud). Correct or remove errors, keep genuine extremes if they matter (for example fraud detection). Alternatives: winsorizing (cap at limits), log transform, robust methods (median, Huber loss, tree models).

Documentation and dangers

Record every cleaning (script instead of manual work in the sheet) so it is reproducible. Aggressive removal can make minorities or rare groups invisible and amplify biases. Test how strongly results change due to your decisions (sensitivity analysis).

See also: Data cleaning with pandas, pandas, Bias, fairness and data quality, matplotlib