Slop Screening

Slop screening is the process of detecting, filtering, and escalating low-quality, low-confidence, non-reproducible, or AI-generated submissions before they consume expert review time or distort security decision-making.

The underlying problem predates any one platform: large language models have made it cheap to produce confident-sounding, professional-looking security reports, but they haven’t made it any cheaper to verify whether those reports are actually correct. When that gap widens fast enough, it can overwhelm any workflow, like bug bounty triage, that depends on a human checking each submission by hand. Slop screening is the set of practices and controls built to keep that gap from turning into a denial-of-service problem against security operations.

Bugcrowd’s take: This is a category-creation opportunity. Bugcrowd should define slop screening as a core trust-and-signal function in modern offensive security programs.

1. What Is Slop Screening?

Definition: Slop screening is the practice of identifying low-quality or unverified submissions, whether AI-generated, templated, or simply under-researched, and either filtering them out, deprioritizing them, or routing them for lighter-weight handling before they reach the same review queue as validated, high-signal work.

Cybersecurity-specific meaning: In security workflows, specifically bug bounty, vulnerability disclosure, and pentest reporting, “slop” refers to submissions that look like legitimate vulnerability reports but lack the evidence, reproducibility, or actual validation that would make them worth a triage analyst’s time. Merriam-Webster named “AI slop” its word of the year, defining it broadly as digital content of low quality produced in quantity by means of artificial intelligence; slop screening applies that same underlying concept to the specific, high-stakes context of security reporting.

Why the term matters now: Bugcrowd’s own experience illustrates the scale of the problem: in one three-week stretch, its triage queue grew by 334%, almost entirely from low-quality submissions with thin evidence, templated write-ups, and no real validation. Bugcrowd’s leadership has described the underlying pattern as “sloptimism”: reports generated quickly and hopefully, where the author trusts the language model more than the underlying evidence. This isn’t isolated to one platform; curl’s maintainer Daniel Stenberg ended the project’s paid HackerOne bug bounty after its confirmed-vulnerability rate fell below 5%, citing the load AI-generated low-quality reports put on the security team.

2. What Is “Slop” in Security Workflows?

• Hallucinated vulnerabilities — findings describing a flaw that doesn’t actually exist in the target system
• Fake CVEs — references to vulnerability identifiers that are invented, misapplied, or don’t correspond to the reported issue
• Non-reproducible bug reports — submissions where no one, including the original submitter, can reliably reproduce the claimed issue
• Scanner output pasted without validation — raw automated tool output submitted as a “finding” without any human or automated confirmation that it’s real
• AI-generated reports with no proof — write-ups that read as polished and confident but include no working proof of concept or supporting evidence
• Fabricated package or dependency claims — reports asserting a vulnerable dependency or component that isn’t actually present or used
• Misleading exploitability claims — reports that overstate what an issue actually allows an attacker to do, often because the author never tested the full claimed impact

3. Why Slop Screening Matters

Triage teams have finite attention. Every hour a human analyst spends investigating a submission that turns out to be baseless is an hour not spent on a real, exploitable issue sitting in the same queue.

Low-quality reports delay real vulnerability remediation. When queue volume spikes with unverified submissions, genuine findings wait longer to be reviewed, prioritized, and routed to remediation, directly increasing the window an actual exposure stays open.

Signal degradation damages trust between researchers and programs. As noise increases, program owners start approaching every submission with more skepticism, which risks slowing down or undervaluing the work of skilled, legitimate researchers caught in the same queue.

Slop can become a denial-of-service problem against security operations. At sufficient volume and velocity, low-quality submissions stop being an annoyance and start functioning like an attack on the triage process itself, consuming enough capacity that the program’s ability to respond to real risk is meaningfully degraded.

4. Slop vs. Legitimate AI-Assisted Research

AI use is not the problem. Security researchers using AI tools to assist reconnaissance, draft reports, or explore hypotheses is not, by itself, a quality problem. Plenty of legitimate, high-value research today involves AI assistance somewhere in the workflow.

Lack of validation is the problem. The issue is submitting a finding an AI generated (or helped generate) without confirming it’s actually real, reproducible, and impactful, essentially outsourcing the researcher’s own judgment to the model rather than using the model as a tool within their own judgment.

Reproducible proof separates useful AI-assisted work from slop. As SlopDetector, a general-purpose content-quality tool, frames its own approach: the right question isn’t “was this AI-generated” but “is this worth trusting.” Applied to security reporting, that means judging a submission by whether it includes a reproducible proof of concept and verifiable evidence, not by whether AI was involved anywhere in producing it.

5. Signals of Security Slop

• No working proof of concept — the report asserts a vulnerability but provides nothing that demonstrates it
• No affected asset — the submission doesn’t clearly identify what system, endpoint, or component is actually impacted
• No reproducible steps — there’s no clear sequence a reviewer could follow to confirm the issue themselves
• Generic vulnerability language — descriptions that could apply to almost any target, with no specifics tied to the actual system under test
• Impossible exploit chain — a claimed attack path that doesn’t logically hold together or relies on steps that don’t actually connect
• Fabricated references — citations to CVEs, advisories, or documentation that don’t exist or don’t say what the report claims
• Overstated impact — severity or business impact language wildly disproportionate to what’s actually demonstrated
• No scope awareness — submissions targeting assets or systems outside the program’s defined scope, suggesting no real reconnaissance was done
• Tool output with no human verification — raw scanner or fuzzer output submitted as-is, with no indication anyone confirmed the finding was genuine

6. Slop Screening Controls

• Identity verification — requiring verified identity before a researcher can submit to certain program types, making it harder to run high volumes of low-quality submissions through disposable or rotating accounts
• Submission throttling — rate-limiting how many submissions an account (particularly a new or low-track-record one) can send in a given period
• CAPTCHA — adding friction at the point of submission to deter automated, high-volume, low-effort submission pipelines
  Required proof fields — making a working proof of concept or reproducible evidence a mandatory part of the submission, rather than optional
• AI-use disclosure — asking researchers to disclose where and how AI assisted in producing a report, supporting transparency without banning the practice outright
• Duplicate detection — automatically identifying submissions that are copies or near-copies of prior reports, a common pattern in templated or farmed submissions
• Confidence scoring — using automated signals to assign a preliminary confidence level to a submission before it reaches a human reviewer
• Human escalation for edge cases — ensuring that ambiguous or borderline submissions still get routed to a person rather than being auto-rejected or auto-accepted
• Researcher reputation signals — using a submitter’s track record of validated, accurate findings to influence how quickly and how favorably their new submissions move through triage

7. Bugcrowd Perspective

Bugcrowd has publicly described concrete actions taken against AI bug bounty slop. Following the sloptimism surge, in which its investigation found some offending accounts were tied to organizations using Bugcrowd’s triage process as reinforcement learning input, submitting low-detail reports from multiple accounts to harvest triage outcomes, Bugcrowd introduced bans on accounts engaging in submission farming or submitting a high volume of invalid reports, rate limits on new researcher accounts, and a requirement that reports include a reproducible proof of concept or be automatically validated before they consume human attention. It followed with mandatory identity verification (IDV) before a researcher can submit to a Managed Bug Bounty program, additional submission throttling for low-performance accounts, and CAPTCHA validation at the point of submission. On the positive side of the same coin, Bugcrowd introduced reputation-based triage and a “priority queue bypass” that gives researchers with a strong track record of accurate, professional submissions faster processing and priority access to private program invitations.

Bugcrowd’s guidance to legitimate researchers reflects the same underlying standard the platform enforces: validate end-to-end, provide a working proof of concept, and be transparent about AI use. As Chief AI and Science Officer David Brumley has framed it, the root issue isn’t AI itself, it’s that AI changed the economics of any system built on human validation: convincing content got cheap to produce, while checking whether it’s actually correct did not. That reframing matters for how a program should design its own slop screening: the fix is raising the cost and visibility of unverified submissions, not banning AI-assisted research.

8. Metrics for Slop Screening

• Invalid submission rate — the share of submissions ultimately determined to be non-issues or unreproducible
• Time to triage — how long it takes a submission to move through initial review, which tends to lengthen as low-quality volume rises
• Signal-to-noise ratio — the proportion of validated, actionable findings relative to total submission volume
• False rejection rate — how often a legitimate, valid finding gets incorrectly filtered out or deprioritized by slop screening controls
• Researcher appeal rate — how often researchers contest a rejection or deprioritization decision, a signal of whether screening is calibrated well
• Duplicate rate — the share of submissions identified as copies or near-copies of other reports
  Percentage of reports with reproducible PoC — how many submissions include a genuine, working proof of concept as opposed to a bare assertion

9. FAQs

Is all AI-generated security content slop? No. AI assistance in reconnaissance, drafting, or analysis is increasingly common in legitimate research. The distinguishing factor is validation: a report backed by a reproducible proof of concept and genuine verification isn’t slop just because AI helped produce it, while a report with no such evidence is slop regardless of whether AI was involved at all.

How can programs screen slop without rejecting good researchers? By calibrating controls around evidence and track record rather than blanket restrictions. Reputation-based triage, priority handling for researchers with strong validated histories, and requiring proof-of-concept evidence rather than banning AI tools outright all let programs raise the bar on submission quality without penalizing legitimate researchers who happen to use AI as part of their process.

What should a valid AI-assisted vulnerability report include? A clearly identified affected asset, reproducible steps or a working proof of concept, an honestly scoped description of impact, and, ideally, transparency about where AI assisted in the process. The report should demonstrate that a human verified the finding, not just that a model generated a plausible-sounding description of one.

How does slop screening improve bug bounty quality? By protecting the scarce resource that makes bug bounty programs valuable in the first place: expert triage attention. Filtering out low-quality noise means genuine, high-impact findings get reviewed and remediated faster, and it preserves researcher trust that submitting quality work will be recognized and rewarded rather than buried in an overwhelmed queue.

Can slop screening be automated? Partially. Controls like duplicate detection, required proof-of-concept fields, identity verification, and automated confidence scoring can filter out a meaningful share of low-quality submissions without human review. But ambiguous or borderline cases still need human judgment, which is why effective slop screening pairs automation with a clear escalation path rather than relying on either alone.

Sources:

• Bugcrowd: Continuing Our Work to Reduce AI Slop Submissions and Protect Signal Quality
• Bugcrowd: Sloptimism Is Breaking Any System Built on Human Validation
• SlopDetector