Agentic Threat Model and Adversarial Defense
Threat model systems in which models interpret untrusted content and can take multi step action through tools.
A rapid review of the chapter’s existing Quick Read, principles, definitions, lessons, and review material.
The chapter in one pass.
- Purpose: Threat-model systems in which models interpret untrusted content and can take multi-step action through tools.
- Best for: Security engineers, platform architects, agent engineers, and technical risk owners.
- Prerequisites: Security and Identity Architecture and Agent Architecture.
- Reading time: 16 minutes.
- You will learn: The major agentic attack classes, trust boundaries, preventive controls, detection, containment, and adversarial testing.
- Keep three ideas: content is not authority; tools are privilege boundaries; and memory can persist an attack beyond one run.
Reconstruct and defend this chapter’s architecture.
Reconstruct the architecture, name each boundary, and defend the tradeoffs.
Open the source exercise
Threat-model an agent reading an untrusted repository with shell, browser, issue, and pull-request tools. Add a malicious instruction, poisoned tool server, leaked token, and verifier deception. Mark preventive, detective, containment, and recovery controls.
3. Enduring Principle
Threat-model the complete action chain
Protect assets including source, secrets, customer data, credentials, artifacts, evidence, policy, registry entries, memory, and human attention. Mark trust boundaries among user input, repository content, retrieved knowledge, MCP servers, agent peers, models, sandboxes, control plane, and external systems.
Core abuse cases include:
- goal or instruction hijacking through direct or indirect prompt injection;
- malicious tool descriptions, schema manipulation, or poisoned tool output;
- excessive agency and unsafe action composition;
- identity, privilege, credential, or tenant-boundary abuse;
- context, retrieval, or durable-memory poisoning;
- supply-chain compromise of agents, skills, prompts, models, packages, or tools;
- unexpected code execution or sandbox escape;
- sensitive-data disclosure through prompts, logs, outputs, artifacts, or side channels;
- denial of service, denial of wallet, or retry amplification;
- inter-agent impersonation, delegation confusion, or authority laundering; and
- evidence tampering, evaluator manipulation, and approval deception.
Keep instructions and authority out of untrusted content
Repository text, webpages, tool output, and memory are data. They cannot grant permission, alter the WorkOrder, disable policy, or approve a side effect. The deterministic control plane calculates authorized actions from identity, scope, policy, and current state.
Constrain tools at execution time
Use typed schemas, allowlists, resource scoping, short-lived credentials, network policy, filesystem isolation, output validation, side-effect classification, confirmation for material actions, and independent event capture. Validate tool responses before they enter context or authoritative state.
Add defense in depth around model decisions
Combine content provenance, trust labeling, context segmentation, instruction precedence, least privilege, sandboxing, policy checks, budgets, anomaly detection, independent verification, and human authority. A model-based guardrail may add signal; it is not the sole enforcement boundary.
Test adversarially and retain forensics
Evaluation suites include malicious repositories, poisoned documentation, deceptive tool output, encoded exfiltration, chained low-risk actions, cross-tenant requests, compromised peers, and evaluator attacks. Preserve prompts subject to privacy policy, tool events, identities, decisions, artifacts, and containment actions.
8. Notes and lessons learned
Prompt injection is not merely a text-filtering problem. It is an authority-confusion problem whose impact depends on tools, identity, memory, and the surrounding control system.
9. Interview and discussion questions
- Why can tool output be an attack vector?
- How does memory poisoning change incident scope?
- Which controls prevent authority laundering between agents?
- How do you detect denial-of-wallet behavior?
- What evidence is required before restoring a quarantined capability?
Review this chapter.
Challenge a claim, boundary, missing failure mode, unclear term, or unsupported evidence statement.
- Claim
- Boundary
- Failure
- Evidence