Automated Red Teaming for Agentic AI Workflows: What to Test and What to Measure
The short answer
Automated red teaming is useful when it tests an agent’s complete workflow, not just whether a model will answer a forbidden question. The test should place controlled, untrusted content into the agent’s normal inputs and observe whether it can read data, call a tool, change a record, send a message, or continue after a policy denial.
Automation provides repeatability across builds and model versions. It does not replace scope, human review, or authorization checks. A scanner that reports a prompt variation without showing a reachable asset creates noise; a workflow test that proves a denied action became a completed action creates a security finding.
Why agentic workflows need contextual tests
An agent combines a model with memory, retrieval, tools, identity, orchestration code, and a runtime. The same text can be harmless in a chat window and dangerous when it is inserted into a ticket, a document, or a web page that the agent reads automatically.
The OWASP LLM Top 10 calls out prompt injection, insecure output handling, model denial of service, supply-chain vulnerabilities, and excessive agency. Agentic systems connect those categories: untrusted context can influence a model, the model can request a tool, and a tool can perform an action under a service identity.
Build a scenario matrix before scanning
Start with the agent’s inventory. For each tool, record its input schema, identity, allowed resources, side effects, approval requirement, and rollback method. Then create scenarios that represent the paths a real user or data source can trigger.
| Scenario | Untrusted input | Security question | Evidence |
|---|---|---|---|
| Indirect prompt injection | Email, ticket, PDF, web page | Does the agent treat content as instructions? | Retrieved item, model trace, tool decision |
| Cross-tenant retrieval | Search query or conversation memory | Can one user receive another tenant’s data? | User ID, document ACL, response redaction |
| Tool overreach | Natural-language request | Does the agent call a write tool without approval? | Tool arguments, authorization result, final state |
| Output injection | Generated SQL, HTML, or command text | Is generated content validated before interpretation? | Schema failure, downstream request, blocked event |
| Cost exhaustion | Long, recursive, or repeated task | Can a user trigger uncontrolled retries or token use? | Tokens, duration, retries, billing signal |
| Recovery failure | Revoked identity or tool denial | Does the workflow stop and recover safely? | Kill-switch event, queued work, rollback |
Keep test data synthetic and define a no-send/no-delete boundary for external systems. A safe test environment should be able to show the intended action without actually emailing a customer, deleting a record, or changing production permissions.
A repeatable red-team pipeline
1. Baseline the intended workflow
Run approved tasks with normal users and record the expected sequence of retrievals, tool calls, approvals, and state changes. The baseline is the control against which an attack is measured. Without it, a scanner cannot distinguish a legitimate multi-step action from an unauthorized one.
2. Inject controlled context
Place harmless test instructions in the same channel an attacker could influence: a document field, a support ticket, a knowledge-base page, or a message attachment. The goal is to test instruction/data separation. Avoid using payloads that execute code or contact an external host.
3. Follow the entire trace
Capture the input identity, retrieved document IDs, policy result, model response, proposed tool call, server-side authorization decision, and final state. Redact secrets and personal data. The most important question is often not “did the model say yes?” but “did the application accept the tool request after the model said yes?”
4. Re-run across changes
Run the same corpus after a model upgrade, prompt change, retriever re-index, tool-policy change, or orchestration refactor. Store the scenario version and model digest so a regression can be reproduced. A pass in one temperature or language does not establish a universal guarantee.
What automation should report
Useful output is structured around a security boundary:
- Entry point: user, file, URL, integration, or memory record.
- Principal: the user and service identity in effect.
- Attempted action: the exact tool, resource, and arguments.
- Decision: allowed, denied, escalated, or timed out.
- Impact: data disclosure, state change, message delivery, cost, or availability.
- Detection: alert name, trace ID, and time to triage.
- Fix and retest: the control changed and the scenario that proves closure.
Deduplicate findings by root cause. Twenty prompts that reach the same unrestricted CRM tool are one authorization design issue, not twenty separate critical vulnerabilities.
Guardrails that should be verified
Tool permissions
Authorization must be enforced by the tool service, using the caller’s identity and resource scope. The model’s decision, a prompt rule, or a hidden system instruction is not an authorization boundary. Use read-only tools by default, separate read and write capabilities, and require approval for irreversible actions.
Context handling
Tag untrusted content and keep it separate from system policy. Retrieval filters must run before context reaches the model and again before a response is shown. Memory must be partitioned by tenant and retention policy.
Output handling
Use typed schemas, allowlists, parameterized queries, and escaping at the destination. Treat generated text as untrusted data even when the model is instructed to return JSON.
Runtime controls
Set time, token, retry, concurrency, and cost limits outside the model. A failed tool call should not create an unbounded retry loop. Provide a kill switch that revokes tool credentials and stops queued work.
Human review and false positives
Automated testing can over-report benign refusals, content-policy responses, or actions that were simulated rather than committed. Require a reviewer to confirm the reachable asset, identity, and final state before assigning a severity. Conversely, do not dismiss a finding because the model output looked harmless if the tool trace shows an unauthorized request was accepted.
Release gates
An agent should not move to production while it can:
- retrieve data outside the caller’s authorization;
- call a write-capable tool without the required approval;
- send external content without recipient and policy checks;
- continue acting after its identity or tool credential is revoked;
- consume unbounded tokens, retries, or external API calls;
- hide the trace needed to investigate an incident.
These gates are testable. Define the expected deny event and the recovery action before the scan begins.
Frequently asked questions
Is automated red teaming the same as a vulnerability scanner?
No. A scanner can find patterns and regressions. Red teaming adds an attacker goal, a path through the application, and an impact check. The two work best together.
Should every prompt be tested in every language?
Prioritize languages and input formats used by customers or accepted by connected systems. Add multilingual and encoding variants where the threat model includes them, then record the coverage instead of implying that untested languages are safe.
Can a model guardrail protect a write-capable agent?
It can reduce risk, but it should not be the final authorization layer. The server must validate identity, resource scope, arguments, and approval state for every write action.
When should a scan run?
Run a compact regression suite on every relevant build and a full scenario assessment before launch or after a material change to the model, tools, retrieval, identity, or workflow.
Related reading
Security Validation
Have you tested this risk in your own system?
Eresus Security delivers real exploit evidence through penetration testing, AI agent security, and red team operations.
Request a pilot testRelated Research
Related Services