The Evolution of AI Security: Why Secure by Design Matters

The short answer

AI security moved from protecting a model endpoint to protecting a changing system: data, artifacts, prompts, retrieval, tools, identities, and runtime infrastructure. Secure by design means these boundaries are defined before deployment, measured during operation, and revisited when the model or workflow changes.

The NIST AI Risk Management Framework uses four functions—Govern, Map, Measure, and Manage—that fit this lifecycle. The framework does not prescribe one product. It gives security, engineering, and business teams a shared way to decide what must be known and what evidence is required.

How the attack surface changed

Early machine-learning security focused on training data, model integrity, and the serving API. Generative and agentic systems add several new paths:

This is why a model benchmark or a one-time jailbreak count cannot represent the security of an AI product.

Core AI-specific threats

Data and retrieval poisoning

An attacker may insert misleading content into training, fine-tuning, a knowledge base, or a vector index. The content does not need to change weights to influence a production response. Protect source ownership, approval, versioning, document ACLs, and re-indexing workflows. Quarantine suspicious items rather than deleting evidence during an incident.

Prompt injection

Prompt injection attempts to make the model ignore or reinterpret higher-priority instructions. It can be direct or indirect, arriving through a document, email, web page, image, or tool response. Treat all external content as untrusted and enforce the actual authorization at the tool or data service.

Insecure output handling

Generated text becomes dangerous when another component interprets it as SQL, HTML, code, a shell argument, or a workflow command. Use typed schemas, escaping, parameterization, allowlists, and destination-side authorization. A model saying “this is safe” is not validation.

Artifact and deserialization risk

Model files and notebooks can contain executable behavior through unsafe serialization, custom operators, or dependencies. Verify hashes and signatures, scan artifacts, and load untrusted files in an isolated environment with no production credentials. Keep a record of the exact artifact that was promoted.

Excessive agency and resource abuse

An agent can be technically correct and still have too much authority. Narrow tool scopes, human approval, quotas, loop breakers, and a kill switch are security controls. Track token, duration, retry, and external API budgets to detect denial-of-wallet behavior.

Defense in depth by lifecycle

Govern

Name the owner, intended use, prohibited use, data classes, supported users, and incident contact. Decide which actions require approval and which data cannot be sent to a model or provider.

Map

Inventory models, prompts, datasets, vector stores, tools, identities, connectors, and runtime locations. Draw the trust boundaries and record which user identity reaches each downstream service.

Measure

Run a versioned evaluation set that includes normal tasks, prompt injection, retrieval authorization, output handling, tool abuse, and resource limits relevant to the product. Record trace evidence and model digests, not only pass rates.

Manage

Prioritize by reachable asset and business impact. Remediate with least privilege, isolation, validation, monitoring, and recovery. Retest after model, prompt, retriever, tool, or identity changes.

A maturity model that is useful in practice

The aim is not to claim that a system is “AI secure.” It is to reduce unknowns and make unsafe behavior containable.

Release and operations checklist

• Pin and verify model, adapter, tokenizer, container, and dependency artifacts.
• Separate system instructions from untrusted retrieved content.
• Apply tenant and user authorization before retrieval and at data destinations.
• Expose the smallest possible tool set and validate every argument server-side.
• Enforce token, time, retry, concurrency, and cost limits outside the model.
• Redact sensitive data in traces while retaining enough evidence to investigate.
• Alert on unusual tool calls, cross-tenant reads, artifact changes, and cost spikes.
• Test credential revocation, tool disablement, index quarantine, and rollback.
• Repeat the security corpus after material changes.

Frequently asked questions

Is secure by design the same as AI safety?

They overlap but are not identical. AI safety often addresses harmful or unreliable behavior; AI security focuses on protecting systems, data, identities, and operations from intentional or accidental compromise. Both should inform the threat model.

Do guardrails replace access control?

No. Guardrails can guide model behavior. Access control must be enforced by the service that owns the data or action.

What should a small team do first?

Inventory the model and connectors, remove unnecessary tool authority, add hard resource limits, and test the highest-impact data and write paths with synthetic cases.

When does the model need to be retested?

After changes to the model, prompt, retrieval data, tool schema, identity policy, serving runtime, or business workflow. These changes can alter the attack surface even when the API does not change.

Related reading

• Tools and Technologies for Secure-by-Design AI Systems
• Beyond Jailbreaks: Contextual Red Teaming for Agentic AI
• AI Security Testing for GenAI Applications