Our AI strategy for preemptive security
Welcome back to my AI lecture series, based on an actual lecture I recently gave at Carnegie Mellon University. In the first installment of this series, I broke down the different mechanisms behind AI to help CISOs define AI as a holistic goal. For today’s post, we’ll be looking at the anatomy of a modern AI system.
Most security governance frameworks treat AI as a black box. This mentality is a liability. Understanding the internal architecture of LLM-powered systems is the prerequisite for meaningful risk assessment, vendor evaluation, and incident response.
Here’s the structural model every CISO should adopt and enforce.
Modern AI models don’t emerge from a single training process. They are built in stages, each of which produces an artifact that can be inspected, attacked, or defended.
Pretraining ingests web-scale data to produce a base model. The artifact is a set of weights encoding statistical patterns across billions of documents. The exposure is corpus poisoning—an attacker who can influence what a crawler ingests can shape a model’s behavior before a single instruction is given.
Supervised Fine-Tuning (SFT) takes human-written demonstrations and trains a model to follow instructions. The artifact is an SFT checkpoint. The exposure? Poisoned demonstrations can introduce backdoor triggers that activate on specific inputs.
Reinforcement Learning from Human Feedback (RLHF)/Direct Preference Optimization (DPO) train a model to prefer outputs humans rate as better. This is the alignment stage. The exposure is counterintuitive and important—the pipeline designed to make a model safer is also the one that can teach it a backdoor. Rando and Tramèr demonstrated at ICLR 2024 that poisoning roughly 5% of preference pairs installs a universal trigger that disables a model’s refusals for any prompt. Alignment training is a poisoning vector.
Adapters/LoRA allow organizations to fine-tune models on domain-specific data without retraining from scratch. The exposure is adapter weights that are small, portable files. JFrog disclosed in February 2024 that approximately 100 models on Hugging Face executed arbitrary code on load via pickle payloads. Calling from_pretrained() is effectively executing untrusted code.
At inference time—when a model is actually running—everything the model sees is concatenated into a single text stream called the context window. This is the most important architectural fact for security purposes.
The context window contains the system prompt (instructions from the application developer), the user prompt (input from the end user), retrieved documents from RAG pipelines, results from prior tool calls, and conversation history. All of this is in-band text. Nothing is structurally marked as more or less trustworthy. A model’s instruction-following behavior is learned during training—it is not enforced by the architecture.
This has a direct security implication: There is no technical mechanism in the format itself that prevents content in a retrieved document from being interpreted as instructions. The trust boundary is behavioral, not structural.
Modern LLM deployments don’t just generate text—they take actions. A model emits structured calls, a host system executes them, and results come back into the context. This loop is how models browse the web, query databases, send emails, and write code to disk.
When you add tools, the threat model expands significantly. A model’s outputs are now arguments to real API calls. Errors and adversarial inputs don’t just produce wrong answers—they produce wrong actions.
Agents take this further. A loop around an LLM with memory and tools can pursue a goal across many steps. Agents have short-term memory (the context window), long-term memory (vector stores, databases, and files), and a registry of available tools. They plan, act, observe results, and iterate. This architecture enables autonomous multistep task completion. It also enables plan hijacking, memory poisoning, and runaway tool chains.
Every layer of this stack is a procurement and risk question:
These are answerable questions. The organizations that start asking them now will be better positioned when regulators come knocking. In the next part of this AI lecture blog series, I’ll cover the basics of securing the AI attack surface.