AI Security: Trust Boundaries, Injection, and Permissions

Security starts with the system, not the chatbot

An AI application may combine a model with documents, search, databases, APIs, code execution, stored state, and user accounts. Each connection creates a boundary where information or authority crosses from one component to another.

That is why AI security is not just “make the model refuse bad requests.” NIST's Generative AI Profile treats information security as a system risk and explicitly discusses prompt injection, data privacy, third-party components, monitoring, and consequences to interconnected systems. [nist-ai-600-1]

From Model to System is the architecture lesson. This page asks what can go wrong at those boundaries.

This lesson is application security: untrusted data, model output, tools, and permissions.

Risk and alignment are covered in AI Safety. Fairness and governance are covered in Ethics, Bias & Fairness. Agent architectures and tool use are covered in AI Agents & Tool Use. This lesson is what those tools are allowed to do, and how untrusted input can reach them.

Prompt injection: data can look like an instruction

A normal application often knows the difference between code and data. A language model receives sequences of tokens containing instructions, quoted material, retrieved documents, webpages, emails, and tool results in the same broad input stream.

Direct prompt injection happens when a user supplies adversarial instructions intended to override or redirect the application's intended behavior.

Indirect prompt injection happens when those instructions arrive through content the system retrieved or opened — for example, a webpage, document, message, or other external source. NIST AI 600-1 explicitly identifies direct and indirect prompt injection as information-security concerns. [nist-ai-600-1]

The key lesson is not a magic phrase for defeating injection. It is that untrusted content remains untrusted even when a model can read it.

A document is not an administrator

Imagine an assistant that reads incoming invoices.

One PDF contains ordinary invoice text plus a hidden or visible instruction telling the assistant to ignore its workflow and send account data somewhere else.

The document is data. It should not acquire authority merely because its text resembles an instruction.

Defenses therefore belong around the model too: constrain tools, separate privileges, validate actions, restrict destinations, require approval for consequential operations, and monitor what happened. OWASP's GenAI guidance treats prompt injection as one risk among several precisely because a secure system cannot rely on the model alone to distinguish trusted intent from adversarial content. [owasp-llm-top10]

Least privilege limits the blast radius

If an assistant only needs to read one folder, giving it write access to the entire drive creates unnecessary authority.

If it only needs to draft an email, permission to send mail without review changes the consequence of a mistake.

If it only needs product prices, it should not automatically inherit access to payroll records because both happen to live in the same database.

Least privilege means giving a component only the access needed for its job. With AI systems, that principle applies to tools, service accounts, data stores, retrieval sources, and actions.

A model can still make a bad suggestion. Least privilege helps prevent that suggestion from becoming an unrestricted action.

Sensitive information can leak through the system

Sensitive data can enter an AI system through prompts, uploaded files, retrieval, logs, tool responses, training pipelines, or stored conversation state. It can leave through generated output, tool calls, logs, or downstream applications.

OWASP lists sensitive-information disclosure as a major LLM-application risk. [owasp-llm-top10] NIST likewise treats privacy and information security as connected risks requiring governance, measurement, and controls. [nist-ai-600-1] [nist-ai-rmf-2023]

Do not assume that because a model “needs context,” it needs every available record.

Model output is untrusted output

Generated text can contain malformed data, unsafe commands, invented URLs, markup, or strings that another system interprets as instructions.

If an application takes model output and passes it directly into a shell, SQL query, browser action, HTML renderer, or business API, the downstream component may give that output powers it never should have had.

OWASP calls this improper output handling. [owasp-llm-top10]

The safe mental model is simple: model output is data until a trusted component validates it for a specific use.

Excessive agency is a permissions problem

An agent may be able to choose tools and sequence actions. That does not mean it should possess every tool or every permission.

OWASP describes excessive agency as risk created by excessive functionality, permissions, or autonomy. [owasp-llm-top10]

A useful design question is: What is the smallest set of actions this system needs to accomplish the task?

Then ask which of those actions need a person to approve them.

Retrieval adds another trust boundary

Retrieval can make an answer more grounded by supplying relevant external information. It also means external content enters the model's context.

A retrieved page can be wrong, malicious, stale, or written for a different audience. A vector database can expose material the current user should not see if authorization is applied incorrectly.

Retrieval quality and retrieval authorization are different problems. Finding a semantically similar document does not prove the user is allowed to receive it.

The supply chain still matters

AI applications depend on models, packages, datasets, plugins, APIs, containers, model files, and hosted services. A weakness or compromise in one dependency can affect the composed system.

OWASP includes supply-chain risk in its LLM-application guidance. [owasp-llm-top10] NIST AI 600-1 likewise calls for attention to third-party components and provenance. [nist-ai-600-1]

“AI-powered” does not suspend software supply-chain security.

Logging is evidence, not decoration

When a system can retrieve data or take actions, useful logs help answer what input arrived, what tool was requested, what permission allowed it, what action occurred, and whether a person approved it.

Logs themselves can contain sensitive information, so logging also needs access control and retention rules.

The goal is not to record everything forever. It is to preserve enough trustworthy evidence to investigate important behavior without creating a new uncontrolled data store.

A demo is not a security evaluation

A working chat window shows that the pieces can be wired together. It does not show that untrusted content cannot gain extra authority, that tools are scoped, or that outputs are validated before they reach other systems.

Security evaluation asks those questions on purpose. It is not the same as a successful demo.

What to take with you

• Treat external text and retrieved content as untrusted data, even when a model reads it.
• Prompt injection can be direct or arrive indirectly through documents, webpages, messages, and other sources.
• Give tools and service accounts the least privilege needed for the task.
• Model output is untrusted output until validated for its downstream use.
• Retrieval must enforce authorization as well as relevance.
• Sensitive data can enter and leave through more places than the prompt box.
• AI systems inherit ordinary software and supply-chain security problems.
• Consequential actions deserve narrow permissions, monitoring, and often explicit approval.
• A demo is not a security evaluation.