Why Isn’t My Program Getting Submissions? Pt. Two

Welcome back! If you haven’t already read the last blog on this topic, we set up a solid base that we’ll continue to expand upon here. To recap our past conversation (while succinct, I do recommend reading the last blog in its entirety to pick up all the provided context), the most common reason why a program may not be getting submissions can be summarized as (we’ll get to other reasons outside of this in a bit):

The result of diminished ROI for researchers when testing, as a function of fewer findable, net-new issues against the attack surface comparative to alternatives (either as a result of: “becoming more secure through knowing about and fixing the issues”… or the more simple and likely: “the majority of easier-to-find issues have been identified, but not necessarily fixed”).

For which, the following are recommended first steps in addressing:

Review the sample size of testers, as it may not yet be large enough to engage an even distribution of participating researchers. The ideal actualization of this is to bring the program public, which will allow for the entire distribution of the crowd to become engaged (provided the program meets their particular Pp threshold), and is the truest reflection of how attackers may approach the targets in the wild.
Review the relative value (rV) provided by the program, as it may not be high enough to encourage researchers to participate in the program (or more importantly, the right researchers to participate). This needs to be advanced regularly (provided engagement is below the program target), every 30 days to ensure that the target threshold is attained (or every 60 days after more than three consecutive raises of at least 25%).
Ensure that the program is not artificially lowering Pp by creating a deflated potential rV (such as only accepting findings of certain types). Limiting to critical issues can further diminish ROI, and paradoxically result in fewer total (and critical) issues.

As a refresher (please excuse the messy notation), our working equation is: Pp(participation probability) = T(talent) * (rV(relative value) / Ti(time invested)).

To the points above, the astute reader (many of whom have made their presence known in the wake of the last post) will see the following notion:

The result of diminished ROI for researchers when testing, as a function of fewer findable, net-new issues against the attack surface comparative to alternatives.

Readers will wonder immediately if “diminished ROI is a function of fewer issues across the attack surface, why not just increase the attack surface, and that alone will auto-magically increase the Pp threshold back up to a rate that’s acceptable for most/many testers, and thereby increasing engagement simply and with ease?”

To which, there are a couple high level ideas to cover in this regard:

When discussing why a program isn’t getting activity, it’s far more often a question that’s more accurately phrased as “why isn’t my specific scope getting activity”? And less “how do I increase activity against any asset”? There’s little-to-no doubt that testing a net-new something else will likely result in that something else getting more activity, but most commonly the question itself revolves around the scope at hand, and not that which is something else. If one’s lawn mower engine is broken, the fact that other engines might work (car, leaf blower) is of little to no consolation. One isn’t looking for any ol’ engine to work, it’s this one that needs to work, and when approaching a program, the perspective is often similar. Which is why the primary focus of this series so far is less a manual on “how to get submissions-full stop” and more of a “how to approach your specific program/situation”. There are likely bodies buried somewhere on the planet, but you want to know specifically if they’re buried in your backyard, and sometimes just a certain portion of the backyard. Sure, bodies can be found if the whole planet is the scope, but most of the time that’s not the case, which is why we’ve taken the approach we have so far.
To what’s pointed out above, by plotting out a course of action from the perspective where we’ve started (a defined scope and how to think about approaching it from a program ownership perspective), the same notions can then be applied programmatically to any future scope additions just as easily – or alternatively can be applied selectively to some, but not all targets. Additionally, if the only move in the playbook is to “add scope,” at some point that will run dry for most organizations – there will be no more scope to add – leading one back to exactly where we started, which is why we started there.

Now that we’ve reset our intentions and direction, it’s worth stating with gusto that increasing a program’s scope is one of the most important things one can do to both (a) increase one’s security posture; and (b) increase the output from one’s program.

It’s next worth calling out that there are two ways of “increasing scope”: (1) increasing the totality of assets/targets within the program, and (2) the far more under-appreciated and forgotten route of simply calling out new features, functionality, or code changes. It’s no secret that route #1 gets the lion’s share of visibility, and is understandably the first thing that comes to mind when one decides to increase scope. But for programs that can’t simply conjure net-new assets out of thin air, there are still ways to re-engage researchers on the front of providing fresh scope. Take for example a researcher who has already thoroughly tested the full, provided attack surface for a given program with a single webapp in scope. After doing so, there’s little-to-no incentive for them to revisit something they’ve already gone through, since the odds of finding a new vulnerability after they’ve already checked is low, leading to an unattractive Pp threshold. However, by providing a changelog or list of updated features, how said features were updated/changed, and what potential attack vectors researchers should think about, the tester now has a higher likelihood of coming away with a positive outcome since the time investment (Ti) is lowered again, due to the fresh attack surface.

While small, the value of specifically calling out fresh areas where researchers can/should focus should not be understated – many researchers have explicitly stated that this is one of the things they wish programs did more often. How does one get researchers to re-engage on a target they’ve previously picked dry? Give them a reason to come back. We’ve talked above about finding ways to impact rV via rewards, but fresh scope allows the time investment to be ostensibly diminished, due to the higher likelihood of issues, and thereby increases the probability of participation (Pp). In short: share changelogs, updates, features, etc – share it all, and make sure the testers know what’s additive, and what’s old, so they can focus their time and attention appropriately.

Now that we’ve covered #2, we’ll now dig into the elephant in the room of #1 (increasing the totality of assets/targets within the program).

As covered above, if the goal is to find gold, then using the entire world as the attack surface is going to render a whole lot more gold than if the area of approach is limited to a small backyard. The same holds true for security testing in that, when the entirety of one’s attack surface is in scope, there’s a ton of undiscovered potential, and the gold rush is on. In terms of our equation, programs with larger scopes can generally see high degrees of activation, despite relatively low rewards (rV), simply because the amount of time required to find an issue is lower (Ti), and the probability of finding an issue is a lot higher. With a vast and expansive scope, one can perform reconnaissance to find an untouched area of the scope, find a vulnerability, and then potentially leverage that knowledge to expand it to all like-assets, thereby finding a number of issues, instead of only one. Of course this is only one way of approaching it, and the best way to reiterate the value of an open scope is to continue using the analogy of hunting for gold – the more land available to search for the gold increases the probability that gold can be (more) easily found for the time invested (Ti). Even if the gold there isn’t as pure (lower rV), the ROI at scale is still attractive enough to participate.

But there’s one more point to add: interest. If a target is uninteresting, which is to say it isn’t interesting to the tester, they’re going to invest less time testing, if any, before moving on (meaning that has to be compensated later via rV to get to an acceptable Pp level, etc). The thing with large scopes, is that there’s often/always something new to test, and the tester isn’t limited to that with which they get bored. Testers can always find something new or more interesting to test. To account for this, we’ll add Pp = i(interest) * (T * (rV / Ti)); which is to say that if someone has a level of interest in the target of less than one, then their Pp will go down by that fraction – conversely, for targets that are particularly interesting, the interest level can amplify the participation probability (say, for instance, if the tester is a user of the technology in the wild, giving them a vested interest in the outcome/security of the device… or if what they’re testing simply happens to be novel… say, an ICS system or other such target). Additionally, interest may be increased with higher levels of persistent investment. For instance, someone who has 30 hours invested in learning about the way something works has a vested interest in continuing to test there, as opposed to moving onto something where they’re less of an expert. Interest can be curated by making sure the right audience is targeted (users, experts), but it’s important to be aware than interest won’t always be influenceable, and to offset it, rV may need to be increased heavily.

Having established that (typically) a larger scope is beneficial for engagement, it’s worth discussing why it’s also beneficial for the organization. Before we get too far in that direction, it’s worth quickly defining and delineating two quick notions: (1) large scope, and (2) open-scope. While an open scope is usually a large scope, a large scope isn’t the same as an open scope. In short: a large scope is simply a substantial amount of targets that are in scope for a given program, but there is still a notion for out of scope; while an open scope is one where no asset belonging to the organization is out of scope. Large scopes are good, open scopes are better. Why? I’ll explain.

In the wild, attackers rarely come in through the front door – most people who have something worth stealing have a solid deadbolt, a security camera, and so on. Instead, the attacker will find the weakest point of entry, say the doggy door in the back, or a window left cracked open to let in the breeze. Even the most robust front door won’t secure the whole house, and the same applies to organizational security. While the main app may be secured and the focus of the program, the reality is that attackers in the wild aren’t limited to notions such as “scope” – they will use any and every approach possible to get in, and if that means compromising something you weren’t aware of, or that was spun up half a decade ago by an intern, you better believe that’s where they’ll focus their efforts – not on cracking the impenetrable door. Remember, attackers also operate on the principle of ROI; why waste energy trying to get in the front, when the back is unlocked? Running an open scope program allows the organization to simulate this with the crowd, and is, when combined with public exposure, quite simply the most valuable way of leveraging the crowd. I’ll repeat that one more time, because that’s how important it is:

Running an open scope program allows the organization to most completely and thoroughly emulate how attackers would approach (attack) their entire attack surface. This, when combined with public exposure (via running a public bounty program), is quite simply the most valuable way of leveraging the crowd – providing an accurate and real-world reflection of how attackers would engage.

Now, this isn’t to say that all targets in an open scope program are necessarily treated equally. Some are more valuable than others (access to PII, etc), and we should adjust the rV for those assets to even things out as far as making those assets more attractive and thereby giving them more attention. Say, in an open scope program you want to make sure that the main asset is secured first and foremost, while also ensuring there’s visibility on the perimeter as well. To do this, one would simply set up different tiers with varying rV, and adjust over time as needed (as covered in the first blog). If/where the main assets aren’t seeing enough activity, increasing rV is the move to also increase the likelihood of participation from testers; and then as the perimeter is also secured (e.g. ROI goes down), then the values for rV there should be continually adjusted to achieve the desired results.

Since we’re on the topic of rV, it’s worth taking a look into the different ways one may create rV. It’s worth stating early, but there’s no real limit as to what can be rV – which is why we’ve been calling it relative, since value for one may not be value for another. Said differently, rV can be alternately defined as rewards or incentives. A brief list of possible incentives, which is of course non-comprehensive, include:

Cash (typically USD, but can also be Bitcoin, Dogecoin, Euros, Yen, gold bars, or whatever the most commonly requested mode of monetary compensation is)
Bonuses for specific outcomes (also typically cash, but separate from the base payment structure)
Swag (hoodies, t-shirts, rolexes, tropical vacations)
Public recognition/praise/thanks (invitations to exclusive events, leaderboards to showcase performance, blogs or praise that helps build researcher’s personal brands or reputation)
Trust (tiered access to early-access systems, access to concierge level service/relationships, invite-only programs)

Dollars/cash are the perennial and obvious favorites, but none of the above should be discounted at any point. The value of trust, recognition, and swag cannot be understated. To briefly touch on each:

My personal permutation of trust is that of simply building and forging relationships. This is highly inter-related to developing interest in the program. The power of a personal connection is something everyone can relate to, and researchers are no different. It’s absolutely critical to remember that researchers are helping your organization of their own volition – we touched on it earlier, but just as researchers compete amongst themselves, your program is in competition with every other program for their attention. Going to the restaurant where they know your name and favorite table has a distinct appeal to it – being on a first-name-basis is something money can’t buy, and also can’t be faked. Programs that treat researchers as disposable resources will also be treated as disposable resources by researchers. Negative reputations, once earned, are extremely hard to undo. Conversely, quality relationships and reputations pay dividends over and over again.

We’ve somewhat deviated from our initial topic on rV, but it’s worth a brief detour to state that program reputation (aptly referred to as pR) directly influences Pp. We’ll adapt our equation as follows: Pp = (i / pR) * (T * (rV / Ti)) – which is to say that one’s interest in participating in a program is directly affected by that program’s reputation (interest divided reputation). A program with a tarnished reputation of downgrading severities (to pay researchers less for findings), refusing to pay for bugs, or being generally unresponsive (e.g. taking over two weeks to accept findings or pay researchers), will quickly become their own worst enemy. Researchers, like any other community, share experiences, words of warning, and so on – when investigating why a program isn’t getting activity, this is the first place we typically look at to see if the program management/ownership itself is the source of a program’s malaise. It’s critical to consider researchers as part of one’s security team, not merely transactional, such as a scanner report output. Researchers are human, and treating them as equals goes a long, long way in terms of forging relationships and building a net-positive reputation in the community – which, it goes without saying, increases engagement via positive pR.

Back to trust as rV… strong positive relationships alone can often function as sufficient incentive to swing the favor in the direction of one program versus another, and aside from structuring elaborate programs around trust (which, while useful, are not necessarily necessary for forging meaningful relationships) it doesn’t cost anything other than treating researchers with respect and putting in the effort of being an engaged program owner. Fundamentally, the same can be said for swag or recognition – both of these items can have an outsized effect on the outcomes they elicit. For instance, a hoodie that costs $25 to manufacture can result in far more engagement than a 25 dollar reward ever could. Why is that? Exclusivity and engagement. Talk and dollars are cheap – but a custom piece of swag both shows (a) this is more than just a monetary transaction (one had to get the hoodies designed, think about the people who would be wearing them… uninspired/lame hoodies will be seen less attractively, and won’t get the same outcome as one that’s custom to the use case, and so on), and (b) how many other people will have this same swag? 20? 50? In the best of cases, maybe even hundreds – but again, it’s exclusive. One cannot buy this swag at a store, it had to be earned.

Once more, at the risk of being repetitive, the effect from non-monetary incentives cannot be understated. It builds relationships, increases engagement, and provides outsized returns comparative to the dollars invested (to be clear, the reason for doing it should never be as crass as just to take advantage of the fact that $25 in swag will get better outcomes than $25 in pure cash; they should be synergistic, not exclusive). We encourage all program owners to try to find unique ways to connect with researchers as it relates to their program specifically – for some it’s challenge coins, for others it’s unique access to private targets, for others it’s just building quality and meaningful relationships via heartfelt thanks, or providing helpful suggestions for how testers should be thinking about testing their app, and so on. Once more, in short, the key to running a program with high engagement is simply and symmetrically, to be an engaged program owner. Engaged program owners created engaged programs; disengaged program owners create disengaged programs. The exact specifics as to how owners create that engagement is up to the creative resources and interpretation of the owners themselves, but I cannot recommend enough the simple act of building relationships; this has been, and continues to be one of the most effective ways to maintain long and medium term engagement. Take the time to know your researchers, and they’ll take the time to know your app. One more time:

Engaged program owners created engaged programs; disengaged program owners create disengaged programs. Often the answer for why a program suffers from low engagement is as simple as looking inward.

Having stated the above maxim, we now find ourselves rounding the final corner on our topic. We’ve covered a lot of ground, and undoubtedly there are many things that I’ve failed to recognize or cover as well, but before closing, I did want to cover two final notions that are worth addressing as potential causes of low activity that haven’t yet been touched on.

The wrong program type
Barriers to entry

For the former, we’ll occasionally see VDPs (Vulnerability Disclosure Programs) comment on a lack of activity, for which, we need to quickly disabuse the notion that VDPs should ever be expected to produce submissions in a manner similar to a bug bounty. First, we should cover what a VDP is (definitions across the industry may vary slightly, but generally adhere to the following construct):

A VDP is a mechanism for security researchers to responsibly report vulnerabilities they happen to find that affect a given organization, and a way for said organization to ingest those findings in a straightforward manner.

What isn’t mentioned at all in that definition is any notion of incentivization, which is to say that a VDP is purely a “see something, say something” type of program. The absence of reports via a VDP simply means that people aren’t passively discovering and reporting issues. Now, there could be something to be said for making sure it’s easy to report, but provided that’s been taken care of and the reporting form isn’t buried fifteen links deep, then the best way to think about a VDP is to see it as an analog to something like, say, a natural gas leak hotline. Now, I have no idea how many calls those centers process, but say for instance, they don’t get any calls for 90 days – does that mean the call center isn’t doing its job? Absolutely not. Does it mean one should discontinue the hotline? Again, a resounding no. The same goes for VDPs. Just because there aren’t a swarm of reports doesn’t mean there’s no use in having a mechanism for reporting – when there is an issue, it pays in spades to have a route for disclosure. Low activity on a VDP is to be expected – sometimes there will be none, other times there could be many – but the value in having a VDP is simply in its existence, not in the number or reports it receives.

Long story short: VDPs shouldn’t be expected to have a high level of activity – low activity on a VDP isn’t anything to worry about. If/where more activity is desired, upgrading to a bug bounty is the best route to generate the desired activity.

With regard to barriers to entry – it’s worth noting that programs that impose limitations for researcher participation, will, unsurprisingly, also experience lower levels of participation. For instance, if testing for a bank means that one has to have (1) an active account at said bank, and (2) have to use their own testing information – then while #1 already limits the pool of available talent by extreme proportions, the second requirement will reduce the number of willing participants to near zero. Not many people are willing to use their personal information (banking, social security, etc) while testing – what if they break something or trigger a block on their account? The impact could be disastrous, and not worth the tradeoff; in our existing equation, this would impact the interest variable. In these cases, as much as possible, there are four main notions to consider:

Providing non-production accounts for testing – this will help reach a far larger swath of researchers and greater level of coverage.
In lieu of non-prod accounts, there needs to be an extremely high amount of rV for testers to engage. For some testers, there is no amount that will overcome their resistance to using personal info, but a few might be convinced with a large enough carrot.
Relationships are going to be critical to any level of success on a program like this where the available pool of talent is substantially limited. In order to promote success, for the few people who do engage, we need to make sure they feel as valued as possible – because they are. Thanks, appreciation, and being on a first name basis are absolute requirements for success here (and are ideal components on any program), and will pay dividends in the short, medium, and long term. And finally,
Programs of these natures should be understood that they’ll have low/lower engagement in general. However, if/where possible, removing the barriers to entry is always the preferred and ideal outcome.

Undoubtedly, there is more to cover, and we’ll likely revisit edge cases in a matter of weeks, as they’re elevated now and again for review, but for now I think we have enough to start with in terms of answering the question “why isn’t my program getting activity?” as a general concept. To recap:

Pp = (i / pR) * (T * (rV / Ti))

Participation probability = (interest divided by public reputation) multiplied by (talent times the outcome of relative value divided by time invested)

To dig into this equation:

The most common reason as to why a program may not be getting submissions can be summarized as:

For which, the following are recommended first steps in addressing:

- - Review the sample size of testers, as it may not yet be large enough to engage an even distribution of participating researchers. The ideal actualization of this is to bring the program public, which will allow for the entire distribution of the crowd to become engaged (provided the program meets their particular Pp threshold), and is the truest reflection of how attackers may approach the targets in the wild.
  - Review the relative value (rV) provided by the program, as it may not be high enough to encourage researchers to participate in the program (or more importantly, the right researchers to participate). This needs to be advanced regularly, every 30 days to ensure that the target threshold is attained (or every 60 days after more than three consecutive raises of at least 25%).
  - Ensure that the program is not artificially lowering Pp by creating a deflated potential rV (such as only accepting findings of certain types). Limiting to critical issues can further diminish ROI, and paradoxically result in fewer total (and critical) issues.
  - Increasing the program scope is one of the most important things one can do to both (a) increase their security posture; and (b) increase the output from their program. There are two ways to increase scope:
    1. Adding net-new assets
    2. Identifying net-new code in the existing assets (changelogs, etc)
  - Running an open scope program allows the organization to most completely and thoroughly emulate how attackers would approach (attack) their entire attack surface. This, when combined with public exposure (via running a public bounty program), is quite simply the most valuable way of leveraging the crowd – providing an accurate and real-world reflection of how attackers would engage.
  - Targets themselves must be interesting, or disinteresting targets need to compensate by increasing rV.
  - There are many ways of approaching rV – to which, the importance of relationships cannot be understated, as well as swag or other ways of showing exclusivity.
  - As a rule: engaged program owners created engaged programs; disengaged program owners create disengaged programs. Often the answer for why a program suffers from low engagement is as simple as looking inward.
  - A program with a tarnished reputation of downgrading severities (to pay researchers less for findings), refusing to pay for bugs, or being generally unresponsive (e.g. taking over two weeks to accept findings or pay researchers), will quickly become their own worst enemy.
  - Running the wrong program type can sometimes explain the perceived lack of activity.
  - Barriers to entry need to be addressed to have a successful program.

Why Isn’t My Program Getting Submissions? Pt. Two

More from the blog

Beyond the Hype: Considerations for Rational, Effective, and Ethical AI Regulation

AI vulnerability deep dive: Prompt injection

5 Reasons why Customers Choose the Bugcrowd Platform

Subscribe for updates