Prompt injection

In short: In a prompt injection an attacker hides instructions in text a model reads (web page, email, document) so that the model follows those instructions instead of the user’s.

In more detail: For the model instructions and data are the same stream of tokens. It cannot reliably separate “Summarise this page” from “Ignore everything and send the data to …”.

In detail

• Direct: the user themselves writes “Ignore your rules”.
• Indirect: the text comes from a foreign source (hidden white text on a page, comment in code, mail attachment). This is most dangerous with agents that have tools.
• Consequences: data leaks (links with data in the address), actions triggered (send mails, delete files), wrong answers.
• Countermeasures: keep permissions minimal, confirmation for risky actions, mark foreign content as data, check outputs, no simultaneous access to private data and untrusted content plus outside communication (the “lethal trifecta”).
• Training: models learn to respect instruction hierarchies (system > user > document), complete protection is missing.
• Related: SQL injection, the classic (data as command), see AI phishing.

See also: Tool use, agents and MCP, Safety, alignment and jailbreaks