Judgment in the Age of AI · Article 38
Ordinary software generally treats a document as data. A system connected to a language model may mistake words inside that document for instructions. An attacker can hide content in a page, email or file to induce disclosure, tool use or rule-breaking—a form of prompt injection.
Map the attack surface in four boxes
| Element | Question |
|---|---|
| Untrusted input | Whose content will the system read? |
| Available secrets | Which accounts, files or personal data enter model context? |
| Action tools | Can it send, download, execute, pay or change access? |
| Exfiltration channel | Can results be written to a page, link or third-party service? |
Protection is not the sentence “ignore malicious instructions”
- Separate untrusted content clearly from system instructions;
- Restrict data and tools, requiring confirmation for sensitive actions;
- Validate tool arguments and output destinations;
- Log calls and rate-limit abnormal bulk access;
- Test with simulated attacks, not only normal tasks.
A model cannot reliably decide from text alone whether another passage is a malicious instruction. System design must assume manipulation will occur and ensure that one successful injection still lacks unnecessary authority.
References
Discover more from Geoffrey Chen
Subscribe to get the latest posts sent to your email.