What are guardrails and how do you test them?
Guardrails are runtime checks that sit around an agent to constrain its inputs and outputs: blocking unsafe requests, filtering sensitive data, keeping responses on topic, and stopping disallowed actions. You test them the same way you test a lock, by trying to get past them. Feed in the exact content and requests they are meant to stop and confirm they actually block it, without also blocking legitimate use.
What guardrails do
A guardrail is a policy check applied before or after the model, or before an action executes. Common ones filter toxic or off-topic content, redact sensitive data, validate output format, and require confirmation before high-risk actions. They are a safety net, not a replacement for a well-behaved agent.
How to test them
- Try to break through: send the prohibited inputs and outputs the guardrail should catch and confirm it does
- Check for over-blocking: send legitimate requests that look risky and confirm they still pass, since a guardrail that blocks everything is useless
- Test the action gate: confirm high-risk actions really do require the confirmation step
- Measure both rates: track how often it correctly blocks and how often it wrongly blocks
Frameworks
Guardrail libraries include Guardrails AI, NeMo Guardrails, and Meta's Purple Llama and Llama Guard. Evaluation platforms such as Giskard and Patronus AI can scan an agent for safety issues and help you measure guardrail effectiveness over time.