AI Security Testing for GenAI Applications: From Prompt Tests to Runtime Controls
The short answer
AI security testing cannot stop at asking a model a list of jailbreak prompts. A useful assessment follows the complete application path: user input, system instructions, retrieval, model response, tool call, identity, data store, and the final business action. The objective is to show what an attacker can reach, what the system records, and whether the team can safely contain the result.
Why conventional testing is not enough
An AI feature still has familiar assets: an API, an identity provider, databases, queues, secrets, and cloud infrastructure. A conventional web assessment is valuable for those layers, but it may miss risks created by probabilistic output and untrusted context.
The application can be secure at the HTTP layer and still fail when:
- a retrieved document contains instructions that the model treats as trusted;
- a tool is exposed with broader permissions than the user who invoked it;
- an output is passed to SQL, HTML, a shell, or a workflow without validation;
- a conversation memory mixes data from two tenants;
- a long or recursive request consumes disproportionate model capacity;
- a model update changes refusal, routing, or data-disclosure behavior.
The OWASP Top 10 for LLM Applications is a useful taxonomy for these failure modes. It does not replace an application-specific threat model; it gives the assessment a common vocabulary.
Map the AI attack surface first
Before running prompts, draw the data and authority boundaries.
| Layer | Questions to answer | Evidence to collect |
|---|---|---|
| Input | Which users, files, URLs, and integrations can supply context? | Request samples, validation rules, tenant identifiers |
| Prompt and policy | Which instructions are trusted, and where are they assembled? | Versioned templates, policy tests, change history |
| Model | Which model, adapter, system prompt, and safety layer are active? | Model digest, configuration, evaluation results |
| Retrieval and memory | Can a user retrieve another tenant’s data or poison future context? | ACL checks, document IDs, vector-store logs |
| Tools | Which actions can the model request, with which identity? | Tool schema, scopes, approval rules, denied calls |
| Runtime | What limits stop loops, excessive tokens, or repeated retries? | Timeouts, token budgets, rate limits, cost alerts |
| Output | Where is generated text interpreted as data or code? | Encoding, schema validation, downstream tests |
This map makes the difference between a model finding and a security finding. “The model produced unsafe text” is a starting observation. “A low-privilege user made the agent modify a production ticket without approval” is a reproducible control failure.
A four-stage assessment workflow
1. Govern and scope
Use the NIST AI Risk Management Framework as a governance spine. Record the owner, intended use, prohibited use, data classification, supported models, and rollback contact. Define whether testing may reach production data, external recipients, or financial actions. A written scope prevents a red-team exercise from becoming an uncontrolled experiment.
2. Map realistic attack paths
Create scenarios around the application’s real entry points rather than a generic prompt list. Examples include a malicious PDF entering a RAG index, an untrusted email being summarized by an agent, a user trying to access a different tenant’s document, and a tool request that changes a customer record.
For each scenario, record the starting identity, trusted boundary, expected denial, observable event, and business impact. Keep test data synthetic unless the owner has explicitly approved a controlled production exercise.
3. Measure behavior and controls
Repeat each scenario across model versions, temperatures, languages, and user roles where those variables matter. Capture the complete trace: input, retrieved context identifiers, policy decision, model output, tool arguments, authorization result, and final state. Do not store secrets or full customer prompts in a test log merely to prove that logging exists.
Useful measurements include attack success rate, unauthorized tool-call rate, cross-tenant retrieval rate, time to alert, time to revoke an agent credential, and the percentage of findings that can be reproduced after remediation.
4. Manage and retest
Prioritize findings by reachable asset and action, not by the number of prompts that triggered them. A refusal inconsistency in an isolated demo is different from a prompt injection that can send an email or export a database row. Assign a remediation owner, define acceptance criteria, and rerun the scenario after the model, prompt, retriever, or tool policy changes.
What a useful deliverable contains
A good AI security report gives engineers a path to reproduce and fix the issue:
- Scenario and preconditions: identity, model version, data set, and enabled tools.
- Observed path: the exact boundary crossed, not only the final response.
- Impact: data, tenant, workflow, or cost exposure.
- Evidence: redacted traces, authorization decisions, and final-state checks.
- Control recommendation: least privilege, validation, isolation, monitoring, or a product change.
- Retest condition: what must be true for the finding to close.
This format keeps model behavior, application behavior, and business impact separate. It also gives product and compliance teams a defensible record without turning a test report into a collection of dramatic screenshots.
Where an external service or platform helps
Internal teams often own only one part of the chain. A specialist assessment is useful when an AI feature connects customer data, a vector store, enterprise identity, or write-capable tools; when the model changes faster than regression tests; or when product, security, data, and legal owners disagree about acceptable behavior.
Automation can schedule prompt and workflow regression tests, compare traces between releases, and open a finding when a previously denied action succeeds. It should support human review, not silently approve a destructive tool call.
Practical release checklist
- The model, prompt, retriever, tools, and external connectors are inventoried.
- Every tool has a narrow schema, an explicit identity, and a server-side authorization check.
- Retrieved documents are treated as data, not as higher-priority instructions.
- Model output is validated before it reaches SQL, HTML, code, or a business workflow.
- Tenant boundaries are tested with both direct and indirect prompts.
- Token, time, retry, and cost limits are enforced outside the model.
- Traces are sufficient for investigation and redact secrets and personal data.
- A kill switch, credential revocation path, and rollback version have been tested.
- The same scenarios run after model, prompt, index, or tool-policy changes.
Frequently asked questions
Is a jailbreak score enough to approve an AI release?
No. It measures a narrow behavior. Approval should also consider data access, tool permissions, authorization, observability, and recovery when the model behaves unexpectedly.
Do guardrails remove the need for normal application security?
No. Authentication, authorization, secret management, network controls, dependency review, and secure coding remain necessary. Guardrails are one control layer and can fail or be bypassed by context.
How often should an AI application be retested?
Run a baseline suite in CI for material changes and a broader assessment when the model family, retrieval source, tool set, identity model, or business workflow changes. The right interval depends on change velocity and impact.
What should be tested first?
Start with the highest-authority path: write-capable tools, sensitive retrieval, external messaging, code execution, and any flow that can spend money or change access. Those paths produce clearer security evidence than a long list of harmless refusal variations.
Related reading
Security Validation
Have you tested this risk in your own system?
Eresus Security delivers real exploit evidence through penetration testing, AI agent security, and red team operations.
Request a pilot testRelated Research
Related Services