Data quality at scale: dedup, filtering, mixing

In short: With very large training data, quality matters as much as size. Three steps carry the most: remove duplicates, filter out the bad, and choose the mix of sources deliberately.

Deduplication

Web text contains a lot of repetition: copied articles, templates, licence texts. Models that see this often memorise it verbatim and score worse in tests. You remove

• exact copies using a hash of the text,
• near-duplicates with MinHash: similar word groups give similar fingerprints, and texts above a similarity threshold count as duplicates.

Filtering

• Rules: too short, mostly symbols, extreme repetition, wrong language.
• Quality model: a small model scores whether a text looks like well-written sources.
• Safety and law: personal data and copyright (Training data, copyright and privacy in AI).
• Test leaks: benchmark tasks must not be in the training data, otherwise results are inflated.

Mixing

What share comes from where: web, books, code, science, conversations? More code often improves reasoning too. Mixes are tried on small models and the best is carried over to the large one (Scaling laws).

Keep bias in view

Every filter rule also decides what the model never sees. Filters that are too strict can drop dialects or minorities (Bias, fairness and data quality).