How AI understands text: NLP tasks

In short: Natural language processing (NLP) covers everything machines do with language: translating, summarising, answering questions, detecting sentiment, finding names, generating text. Large language models solve most of these tasks with a single model.

In more detail: In the past there was a separate model for each task. Today a prompt suffices.

In detail

• Classification: spam, sentiment, topic.
• Sequence labelling: proper names (NER), parts of speech.
• Translation: formerly rule-based, then statistical, neural since 2016, today with language models.
• Summarisation: extractive (choose sentences) or abstractive (reformulate).
• Question answering: with retrieval-augmented generation on your own documents.
• Information extraction: get amounts and dates from invoices, output JSON.
• Dialogue: chat assistants.
• Measuring: BLEU and ROUGE (text overlap, outdated), benchmarks and human evaluation, see Benchmarks for language models.
• Hard cases: irony, ambiguity, dialect, technical language, under-represented languages.
• Classic methods: word counting (TF-IDF), n-grams, rules. They are fast, cheap and often sufficient for simple tasks.

See also: How a language model generates text: step by step, Tokenisation: BPE and vocabulary