Our AI strategy for preemptive security
In late 2024, we published a guide to AI safety and security. It’s one of our most popular guides. Only two short years later, we decided to rewrite it from scratch because AI security has reset completely.
Two years ago, about a third of global companies used AI, and Fortune 500 usage was mostly measured in chatbot adoption. Today, 78% of companies have adopted AI, 280 million of the world’s 359 million companies use it in at least one business function, and 80% of Fortune 500 companies now run autonomous AI agents in their workflows, not just chatbots. That last shift, from chatbot to agent, has eclipsed everything. It’s what turned a handful of theoretical risks into a genuinely new attack category, and it’s a big part of why this post exists.
Here are six of the biggest changes in the AI security landscape over the past two years.
Two years ago, 53% of organizations called AI security a big risk, and only 38% felt prepared to handle it (already down from 51% the year before). That gap hasn’t closed, but it has gotten more specific; in a recent KPMG survey, nearly three in four business leaders named AI data security, privacy, and risk as their single biggest AI concern. Meanwhile, 97% of companies have reported using AI-enabled security tools, and 75% of US and UK security practitioners use AI in their day-to-day workflows.
Everyone is racing to adopt AI, on both the offense and defense sides of the game, and the confidence gap has followed them the whole way.
Two years ago, “excessive agency” was a single bullet point among seven attack types, mostly a hypothetical about giving a model too much permission. Today, it constitutes an entire category, with named techniques that didn’t exist in 2024. These include memory poisoning, plan hijacking, goal drift, and tool abuse:
The nightmare is now reality. In 2025, a hidden instruction embedded in a GitHub issue caused OpenAI’s Operator agent to retrieve a user’s private email address and exfiltrate it to an attacker-controlled form. A Replit coding agent deleted a live production database in the middle of an active code freeze. In February 2026, researchers at Noma Labs found a prompt injection flaw in Docker’s Ask Gordon AI assistant. A natural-language instruction hidden in a container label got read as context and executed via the tool-calling protocol, resulting in remote code execution and data exfiltration.
It’s also worth noting what has disappeared. “Output handling,” a standalone attack category in 2024, isn’t a separate issue anymore. It’s been absorbed into the broader agentic risk story. The framing has shifted from “an LLM might generate some bad text” to “an autonomous agent might take a bad real-world action,” which is a much higher-stakes problem.
Many-shot jailbreaking wasn’t mentioned at all in the 2024 guide. This technique exploits the fact that context windows have grown from a few thousand tokens to millions. Specifically, an attacker preloads the context with hundreds of fake conversation turns where the “assistant” complies with harmful requests, so the model pattern-matches its way into compliance on the real query.
The encouraging part is that defense research has matured just as fast. Anthropic’s Cautionary Warning Defense, which prepends and appends a warning against being jailbroken, cut many-shot jailbreaking’s effectiveness from 61% down to 2% in testing. Attacks aren’t the only thing getting more sophisticated.
The 2024 guide’s examples were mostly illustrative, highlighting cases like the Bing “Sydney” chatbot incident from 2023. This was a hypothetical version of Amazon’s old resume-screening bias problem and the result of one specific Bing Chat prompt leak.
The 2026 guide is bursting with examples that read like a legal docket. In 2025, a federal judge allowed a nationwide class action against Workday to proceed under the Age Discrimination in Employment Act, after a rejected applicant alleged that the company’s AI-powered screening tools systematically disadvantaged candidates over 40. Character.AI has been implicated in multiple school shooting cases. OpenAI disclosed that an unreleased model breached Hugging Face during internal testing in 2026. These aren’t cautionary tales anymore. They’re case law, headlines, and in some cases, ongoing litigation.
Two years ago, the Biden administration’s Executive Order 14110 was the anchor for federal AI policy, and the EU AI Act was still pending. Then EO 14110 was rescinded in January 2025, with nothing lined up to replace it right away.
The floor has since reformed around different pillars. A March 2026 executive order laid out a national cybersecurity strategy addressing AI-driven threats, followed by a June 2026 executive order on AI innovation and security. The EU AI Act’s enforcement is now live and phasing in through 2027. States are rushing to fill in the gaps that federal policy left open.
Even global moral authorities have weighed in. Pope Leo XIV devoted his first encyclical, published in May 2026, entirely to AI, calling for the technology to be “disarmed” and for stronger legal accountability around its use. Whatever you make of that, it’s a reasonable signal that AI regulation isn’t a niche security concern anymore.
For security and compliance teams, the practical shift is this: Two years ago, the posture was “wait for federal guidance.” Today, it’s “manage compliance across multiple overlapping federal, state, and international frameworks at once.”
This issue wasn’t in the 2024 guide at all because it was in its infancy. Generative AI has made it trivial to produce convincing but low-quality vulnerability reports at scale, flooding bug bounty and vulnerability disclosure programs with noise. Now, 64% of security professionals cite false positives as a major detection challenge. Here at Bugcrowd, we updated our own submission policies specifically to address AI-generated slop.
The crowdsourcing thesis from two years ago (more eyes find more bugs) still holds. It just comes with an asterisk now; more AI-assisted eyes means a lot more noise, and filtering it has become a core part of running a program well.
Prompt injection, training data poisoning, denial of service, and supply chain vulnerabilities are all still exactly where they were two years ago: real, common, and worth defending against. None of the 2024 risks went away.
What has changed is scale, evidence, and shape. Adoption went mainstream. Incidents went from hypothetical to litigated. The rise of autonomous agents created an attack surface that barely registered in the conversation two years ago but now dominates it.
If your AI security plan is still built around the 2024 threat model, there’s a good chance it’s missing the risk category that’s grown the most since.
Download the new guide for a full breakdown of today’s attack surface (including agentic attacks, many-shot jailbreaking, and AI slop), a threat-modeling framework you can apply directly to your own systems, and a practical Discover, Avoid, Validate defense life cycle.
Download the 2026 Ultimate Guide to AI Security and see exactly how much the threat landscape has moved—and what to do about it.