Limits of AI: what models cannot do

In short: Language models are powerful but not omniscient: they sometimes invent facts, know nothing after their knowledge cutoff, calculate unreliably and understand the world only through text and images.

In more detail: The model predicts the probably fitting continuation. Probable and true are not the same. That is the source of many errors.

In detail

• Hallucination: invented sources, numbers, laws. Remedy: demand evidence, RAG, check, see Hallucination (AI).
• Knowledge cutoff: it knows later events only with a search tool.
• Arithmetic and counting: multi-digit calculation and counting letters are error-prone (tokens). Better have code executed.
• Context: with very long context information in the middle is used worse (“lost in the middle”).
• No real memory: none between conversations unless a system stores notes.
• Sycophancy: tendency to agree with the user.
• Unreliable self-assessment: the model can be very confidently wrong.
• Understanding: whether models “understand” is disputed. Practically: impressive in many tasks, unreliable at the edges.
• Good use: as an assistant with verification, not as a source of truth. For health, law and finance ask experts, see Using AI assistants sensibly and safely.

See also: Hallucination (AI), Bias, fairness and data quality