Data annotation and labelling: How training data arises

In short: For supervised learning people must provide data with the right answer (labels); quality, consistency and working conditions in labelling shape the later model.

In more detail: A model meant to recognise cats needs images with the information “cat” or “not cat”. This labelling (annotating) is often the most expensive and most important part of an AI project. “Garbage in, garbage out”: wrong labels produce a faulty model.

In Depth

Kinds of annotation

• Classification: one label per example (spam/not spam).
• Object detection: bounding boxes around objects.
• Segmentation: pixel-accurate masks.
• Keypoints, skeletons, transcription of audio, entities in texts (names, places), ratings of answers (for language models, such as preference comparisons). Tools: Label Studio, CVAT, Labelbox, Prodigy.

Ensuring quality

Write clear guidelines with examples and edge cases. Have several people label the same data and measure agreement (Cohen’s kappa). Use samples and reviews, gold standards (known correct examples) and check anomalies. Unclear cases belong in discussion, not guessing. Models help too: active learning selects examples where the model is unsure; pre-annotation by a model that humans correct.

Bias and ethics

Labellers bring their own views and prejudices; the choice of data and categories shapes what the model “sees”. A lot of labelling work is done worldwide by click workers at low wages, sometimes with distressing content. Fair pay, support and transparency about the origin of data are part of responsible AI.

Privacy and practice

Data with personal reference needs a legal basis and protection (pseudonymisation, access restriction). Start with small pilot datasets to test the guidelines before labelling large quantities. Version datasets and labels, and document origin, licence and known weaknesses (datasheets for datasets).

See also: Learning types, Bias, fairness and data quality, Training data, copyright and privacy, Data augmentation