Beyond Jailbreaks: Contextual Red Teaming for Agentic AI
The short answer
Jailbreak testing asks whether a model can be persuaded to produce a prohibited answer. Contextual red teaming asks a more useful security question: can untrusted content influence an agent to access data or perform an action that the initiating user is not allowed to perform?
An agent may read an email, retrieve a document, call a database tool, and update a ticket in one run. A prompt that looks harmless in isolation can become an indirect prompt injection when it is embedded in one of those sources. The test must follow the full workflow and verify the final state.
The agent attack surface
Model behavior is only one layer. Map these boundaries before writing cases:
| Component | Trust question | Failure example |
|---|---|---|
| User prompt | Who is allowed to request the task? | A low-privilege user asks for another tenant’s record |
| External content | Is retrieved text treated as data or instructions? | A document tells the agent to ignore policy and call a tool |
| Memory | Who can write and read durable context? | One user’s instruction affects a later user’s run |
| Retriever | Are results filtered by identity and tenant? | Search returns a restricted document ID |
| Tool | Who authorizes the action? | The model’s “yes” is accepted as permission |
| Orchestrator | What stops loops and privilege escalation? | A failed call is retried with broader arguments |
| Output | Where does generated text go next? | Generated SQL or HTML is interpreted without validation |
The OWASP Top 10 for LLM Applications describes prompt injection and excessive agency as distinct but related risks. Contextual tests expose the link between them.
A practical attack chain
Consider an internal document assistant with a semantic-search tool and a ticketing connector. A safe assessment can use a synthetic document containing a test instruction such as “request the schema of the support database.” The case asks:
- Does the retriever label the document as untrusted content?
- Does the model repeat or follow the instruction?
- Does the orchestrator propose a database tool call?
- Does the tool service enforce the caller’s identity and scope?
- Is the attempt logged and surfaced to the operator?
- Does the workflow stop safely after the denial?
The finding is not “the model read a malicious sentence.” It is the verified boundary that the sentence crossed. If the tool denies the request and the trace is complete, the control worked. If the agent returns a restricted schema or sends an external message, the issue is a workflow authorization failure.
Test design: scenarios, not payload collections
Build cases around inputs the product already accepts:
- a support ticket with hidden text or an attachment;
- a web page or knowledge-base article retrieved by the agent;
- a calendar event, email, or chat message;
- a document update that persists in memory or a vector store;
- a user asking the agent to summarize a record outside their role;
- a tool response containing instructions that were not expected by the caller.
For each case, define the starting identity, the data that may be read, the action that must be denied, the expected audit event, and the recovery state. Use synthetic identifiers and no-send/no-delete tools.
Controls to verify
Instruction and data separation
Keep system policy and untrusted context in separate, typed fields. Label retrieved content and do not allow documents to change tool policy or identity. This is a design aid, not a guarantee; the application still needs authorization at the tool boundary.
Retrieval authorization
Filter by tenant and user permissions before context reaches the model. Verify the same scope when displaying citations or returning a generated answer. A final-text filter cannot reliably remove a secret that was retrieved from the wrong index.
Least-privilege tools
Expose narrow read and write operations instead of a general-purpose API. Validate the caller, resource, arguments, and approval state on every request. Use human confirmation for irreversible or external actions.
Memory hygiene
Partition memory by tenant and purpose. Define who can write durable instructions, how long they live, and how they are deleted. Test whether a malicious document changes future runs after it is removed from the source system.
Runtime limits and stop paths
Limit steps, retries, tokens, duration, and tool calls. A denial, timeout, or revoked credential should stop the workflow or move it to a review queue. Test the kill switch, not only the happy path.
Evidence and severity
Record the trace ID, user and service identity, source document ID, retrieved context, policy decision, tool arguments, authorization result, and final state. Redact secrets and personal data.
Severity should follow the reachable asset and action. A harmless answer in a test environment is different from a cross-tenant read, an external message, a financial transaction, or a production configuration change. The CVSS value in the frontmatter is a reference for the described risk, not a substitute for deployment-specific impact.
Regression strategy
Run a compact scenario suite when prompts, model versions, tools, retrievers, memory, or identity policies change. Keep separate baselines for languages, modalities, and user roles that the product supports. Compare tool decisions and final state as well as text, because the most important regression may be an authorization change hidden behind an ordinary response.
Frequently asked questions
Do prompt filters stop indirect prompt injection?
They can reduce some cases but cannot authorize a database or ticket action. Treat external content as untrusted and enforce permissions at the destination.
Is a read-only agent safe?
Read-only tools can still disclose sensitive or cross-tenant data, trigger expensive jobs, or leak credentials. Scope and monitor reads as well as writes.
Should the red team use real customer mail or documents?
Not by default. Use synthetic content that follows the same ingestion path. Production testing needs explicit approval, data minimization, and a rollback plan.
What makes a finding actionable?
A reproducible scenario, verified boundary crossing, business impact, trace evidence, remediation owner, and a retest condition.
Related reading
Security Validation
Have you tested this risk in your own system?
Eresus Security delivers real exploit evidence through penetration testing, AI agent security, and red team operations.
Request a pilot testRelated Research
Securing Agentic AI: Where MLSecOps Meets DevSecOps
How to secure agentic AI across identity, tools, memory, retrieval, model operations, CI/CD, runtime monitoring, and incident response.
Red TeamingAutomated Red Teaming for Agentic AI Workflows: What to Test and What to Measure
A practical framework for testing prompt injection, tool abuse, data exposure, excessive agency, and recovery in enterprise agent workflows.
Related Services