Data cleaning in practice with pandas

In short: A large part of every data job is tidying up. With pandas you can clean missing values, duplicates, wrong types and outliers systematically before a model is trained.

Look first

import pandas as pd
df = pd.read_csv("data.csv")
df.info()          # columns, types, missing values
df.describe()      # statistics of the numbers
df.head()          # first rows

Typical steps

1. Missing values: df.isna().sum() counts them. Remove (df.dropna()), fill with a default (df["age"].fillna(df["age"].median())) or mark as its own category. The choice depends on the cause.
2. Duplicate rows: df.drop_duplicates().
3. Types: df["date"] = pd.to_datetime(df["date"]), df["price"] = pd.to_numeric(df["price"], errors="coerce").
4. Unify text: df["city"].str.strip().str.lower(); merge different spellings.
5. Outliers: limits via quartiles or domain knowledge, then check instead of deleting blindly (anomaly detection).
6. Categories to numbers: pd.get_dummies(df["colour"]).

Important rules

• Never overwrite the original; produce a cleaned copy reproducibly from the raw data (script instead of manual work).
• Compute the cleaning only with values from the training part (for example the mean), otherwise test data leaks in.
• Document what was changed.

See also: Pandas, Data, features and labels