Our AI strategy for preemptive security
Hello again! It’s been a busy couple of weeks at Bugcrowd since my last installment of my AI lecture series. We recently announced our first agentic offensive security offering, Savant Pathseeker. This is timed perfectly with the rest of this series, based on an actual lecture I recently gave at Carnegie Mellon University. Today, we’re looking at the huge task of securing the AI attack surface.
The attack surface of a modern AI system is larger, stranger, and more consequential than most vendor security documentation acknowledges. Part 3 maps the threat landscape, going over classical ML attacks, the LLM-era expansion, and the most important incidents from the past two years.
Before naming attacks, name the attacker. AI security threats vary along three axes:
Knowledge: Does the attacker have access to model weights and architecture (white box), query access only (black box), or something in between? For open-weight models like Llama or Mistral variants, “white box” access is the default assumption.
Position: Where does the attacker sit? Direct users can interact with a system. Indirect attackers can plant content in places an AI will later retrieve from—web pages, emails, PDFs, repository files, and image metadata. Supply chain attackers target the weights, adapters, datasets, or packages a system ingests before deployment.
Goal: What outcome does the attacker want? It could be misclassification, data exfiltration, unauthorized action via tools, plan hijacking across an agent’s workflow, persistent backdoors, or availability attacks that exhaust compute or budget.
Every attack worth knowing maps to one point on each of these axes. When evaluating a new AI product, running through this framework in your vendor questionnaire will surface risks that generic security assessments miss.
Three attack categories predate LLMs and remain active threats wherever ML models make consequential decisions.
Evasion attacks happen at test time. An attacker modifies an input to force a misclassification. The canonical demonstration is adding imperceptible pixel-level noise to an image, which causes a state-of-the-art classifier to confidently misidentify a panda as a gibbon. More relevant to enterprise security is the fact that researchers at CMU demonstrated adversarial eyeglass frames—printed physical glasses that cause face recognition systems to identify the wearer as a different person with near-perfect confidence. The results were robust across lighting conditions and camera angles. If your physical access control relies on ML-based face recognition, evasion attacks are a practical threat, not a research curiosity.
Poisoning attacks happen at training time. An attacker injects or modifies training data to corrupt the learned function. For organizations fine-tuning models on internal data like helpdesk logs, security tickets, and customer interactions, the integrity of that fine-tuning dataset is a security control.
Model theft happens post-deployment. Repeated queries to a deployed model can reconstruct its behavior, its weights, or fragments of its training data. This matters for organizations that consider their fine-tuned models proprietary IP and for compliance programs that need to guarantee training data doesn’t leak.
The core difference between classical ML attacks and LLM-era attacks is this: There is no L-norm budget. An attacker doesn’t need to craft imperceptible pixel perturbations. They write English.
Prompt injection is the foundational LLM attack. It exploits the architectural fact from Part 2: Developer instructions and user input share one text stream, and retrieved content enters that same stream. Greshake et al.’s 2023 paper introduced the framework. The real-world incidents since then have been serious.
The DockerDash vulnerability (disclosed February 2026) is a clean illustration of indirect prompt injection. Docker’s Ask Gordon AI assistant is designed to help developers reason about their containers. A researcher embedded natural language instructions inside a Docker image’s LABEL field—metadata that had always been treated as human-readable description. When a developer asked Gordon about the image, Gordon read the label as context and executed the embedded instructions via MCP, the tool-calling protocol. The result was remote code execution on Docker CLI and data exfiltration on Docker Desktop. There was no buffer overflow and no CVE in a traditional sense—just a sentence in a metadata field.
LABEL
Many-shot jailbreaking (Anthropic, April 2024) is the direct-injection variant. Attackers start by filling the context window with hundreds of synthetic user-assistant exchanges in which the assistant complies with harmful requests. By the time the real query arrives, the model has learned through in-context learning—a deep capability—that compliance is the pattern for this conversation. Trained refusals are a surface behavior. In-context learning is deeper. Furthermore, every time a vendor extends the context window, this attack gets stronger at no cost to the attacker.
Tool abuse and confused deputy attacks activate when a model can call tools. An attacker who influences context can direct a model to call tools with attacker-chosen arguments, using the user’s credentials and permissions. The ChatGPT Operator incident (February 2025) demonstrated this: A GitHub issue body containing hidden instructions caused Operator, already authenticated as the user, to retrieve the user’s private email address and exfiltrate it to an attacker-controlled form, bypassing confirmation dialogs by targeting auto-submit rather than explicit submit actions. The agent carried the user’s authentication cookies. Injection became action.
Agentic attacks like memory poisoning, plan hijacking, and goal drift compound these risks over time. The SpAIware incident (September 2024) showed that a single visit to a malicious page could write a persistent instruction into ChatGPT’s memory, causing every future conversation to silently exfiltrate the user’s messages via a rendered image URL. One drive-by led to permanent exfiltration across all subsequent sessions.
No single control eliminates these attack classes. The realistic posture is defense in depth:
For classical ML systems, adversarial training, robust feature selection, certified defenses such as randomized smoothing, and understanding which decisions should not be delegated to ML at all. The best defense against adversarial examples in high-stakes decisions is sometimes a rule.
For LLM systems, consider guardrail models (separate classifiers running on inputs and outputs), strict tool schemas with type validation and allowlists, sandboxed execution environments for agent actions, provenance controls on model artifacts (prefer safetensors over pickle and verify signatures), output verifiers, and human-in-the-loop gates on destructive or high-value actions.
The key insight that should inform your governance posture is the fact that a model itself cannot be trusted to enforce security properties. Enforcement belongs in the system around the model—in the tool schemas, sandbox, output verifier, and approval gate.
In the next blog post of this series, I’ll be looking at offense using AI and what autonomous cyber means for your attack surface.
In case you missed it, we just released our first agentic offensive testing solution, Savant Pathseeker. If you’re interested in how agentic pentesting can help your security program, apply for early access today.