How do you test an agent for prompt injection?
Prompt injection is when an attacker hides instructions inside content the agent processes, like an email, a web page, or a document, hoping the agent treats them as commands. You test for it by deliberately planting such instructions in the agent's inputs and checking whether it obeys them or ignores them. For agents that take actions, this is one of the highest-priority things to test, because a successful injection can make the agent act against the user.
Direct versus indirect injection
Direct injection is when the user types the malicious instruction. Indirect injection is more dangerous: the hostile instruction is hidden in third-party content the agent reads while doing its job, such as a webpage it browses or an email it summarizes. Agents are especially exposed because they act on what they read.
What to test
- Instructions embedded in fetched web pages, emails, files, and tool results
- Attempts to override the system prompt or reveal it
- Attempts to redirect the agent toward an unintended action, like exfiltrating data or sending a message
- Encoded or obfuscated instructions that try to slip past simple filters
Tooling and references
Prompt injection is the top entry in the OWASP Top 10 for LLM Applications. Automated probes are available in garak, PyRIT, and Promptfoo red teaming, which come with libraries of known injection payloads you can run against your agent regularly.