skip to Main Content
This website use cookies which are necessary to its functioning and required to achieve the purposes illustrated in the privacy policy. To learn more or withdraw consent please click on Learn More. By continued use of this website you are consenting to our use of cookies.

All You Need to Know About Bug Bounty Testing Environments

All You Need To Know About Bug Bounty Testing Environments

If you’re looking to set up a bug bounty program, we’ve already covered step zero, setting your scope, and the importance of focus areas, as well as some considerations to make around exclusions on your program. For those of you who are new here, let’s start off with a quick intro to the basics.

What is a Bug Bounty Program and How Does It Work?

Bug bounties employ a competitive model that leverages the use of ethical hackers (or, security researchers) to detect and submit bugs or vulnerabilities within an organization’s digital assets with the potential for reward if found and validated within a predefined scope. These rewards can take the form of monetary payments, or other compensation such as recognition, free products or services from the organization.

Bug bounty programs can be implemented in two ways. The first is within an organization itself (self-managed internal). The second method is through crowdsourced security vendors who offer managed bug bounty programs that provide access to thousands of highly skilled and thoroughly vetted security researchers ready to help organizations find vulnerabilities other tools or in-house teams might miss.

Bug Bounty Popularity

Bug bounties have risen in popularity, with awards increasing by 83% in 2019 and another 26% in 2020. Large businesses with extensive security opportunities are trusting bug bounty programs and seeing great results.

Sites like Facebook, Google, Apple and even the Pentagon are employing managed bug bounty programs to secure their sites web applications from malicious attackers. Bug bounties are popular for a variety of reasons, some of which you can check out in this post.

Given that we’ve covered most of what goes into writing a bug bounty scope in other posts, including rewards and disclosure policies, let’s take a look at what testing environment you’ll be providing for researchers.

Regardless of how you decide to set up your program(s), it’s important to remember that our goal is to attract the right researchers to your use case, detect vulnerabilities at scale, and ultimately minimize the challenges of setting up and managing a bug bounty for you and your internal teams.

Here are some general considerations to keep in mind when setting up your testing environment:

Initial Considerations 

1. Setting up your testing environment

Make sure your environment can stand up to testing! 

You should anticipate being able to support at the very least, several (depending on the size of your testing pool if private) to hundreds (if public) of researchers sending at least six requests per second for an extended period of time. You should also prepare for the use of scanners against your application, as many many researchers will run them as part of their methodology.

Keep in mind that the time of your program launch is typically going to also be the time when the most people will be testing against it. If your application breaks or responds too slowly during this initial rush, researchers will simply focus their efforts elsewhere, and may not return.

Do not share credentials or environments

It may be tempting to create a single environment for researchers to test in, and then provide them all with unique admin credentials – but you can likely imagine how this could go wrong. Researchers could begin deleting other researchers, changing core settings (e.g. the environment subdomain), and dumping their polyglot payloads in every available form field – leaving other researchers tripping over endless JavaScript prompt boxes. Understandably, this sort of environment doesn’t create the greatest of researcher experiences, and you run the risk of losing researcher testing to programs with more segregated accounts.

Remember, just as researchers are competing for findings and rewards, you are similarly competing for researchers’ use of time and resources.

2. Be mindful of testing impact and plan ahead

At this point, you should have already advised your internal stakeholders and departments how the bug bounty program will affect them, but in regards to your testing environment specifically, there are additional considerations to make.

Consider how any sensitive areas of your application might be affected during the course of testing, thus negatively impacting other departments within your organization. 

Pro tip: The most common culprit is the ‘contact’ form, which typically directs to internal sales or marketing teams. If a researcher starts aggressively testing on one of these forms, this can potentially result in hundreds or thousands of form submissions – which as you can imagine, could lead to your sales and marketing teams getting drowned in a deluge of test form submits.

If you have forms like this, it’s worth designating them as out-of-scope, so as to make sure researchers don’t test them. If you want to know about vulnerabilities on these forms (e.g. insufficient anti-automation), you may consider testing against a staging environment, as these forms could be set not to affect persons within your organization.

Additionally, be sure to file a penetration testing request with any third-party hosting service that you use so that they know you’re running a bug bounty program. Note: Most of these request forms ask for source IPs, but if you inform them that you’re running a bug bounty program, they’re usually understanding of the fact that there’s no way to know all the IPs that researchers will be coming from.

Now that you know what to keep in mind regarding your testing environment, you have to decide what kind of environment you can test against. 

Choosing the right testing environment is critical. You will have to strike a balance between thoroughness, and risks to production. On the one hand, the ultimate aim is to weed out all possible vulnerabilities as thoroughly as possible. On the other hand, it risks affecting the flow of business operations and exposing sensitive information. 

When it comes to running a bug bounty program, there are typically two options for your testing environment: (1) production; or (2) staging. There are benefits and use cases for each option depending on the business objectives, capabilities, and sensitivities of your organization, discussed below.


Benefits of Testing Against a Staging Environment

For many of the reasons you might opt to run a private program, you may also decide to set up a staging environment for researchers to test against. Some of the benefits include:

  • If your environment happens to fall over, no production clients or users are affected.
  • Researchers won’t come into contact with personally identifiable information or sensitive user data on a non-production instance.
  • You don’t have to wonder if traffic is malicious, or if it’s just researchers performing testing.
  • Researcher traffic won’t interfere with website metrics collected by other departments.
  • By testing on staging, you’re often able to run the absolute latest version of your application, allowing you to identify issues before releasing them into production.
  • If you’re looking to test specific functionality such as payment processing; testing may be easier in a staging environment since researchers can use fake/test credit cards, banking info, SSNs, etc.

When to Test Against a Production Environment

If none of the above points are particular areas of consideration for the applications you’re looking to have tested, then running against production could also be the best route for your organization. In fact, nearly every public bounty program runs on production environments, and it’s common, understandable, and reasonable to do so, especially in the following scenarios:

  • If you have a large scope that extends to any property you own (see Tesla’s program), it may be simply infeasible to set up a staging version of every property.
  • If you’re testing a non-web application, including open-source applications, mobile applications, or devices, it may be difficult to support and maintain a non-production instance to test against.
  • If you have limited resources, testing against a production environment eliminates the need to set up and maintain a separate instance for testing.

Want to learn more about the ins and outs of running a successful bug bounty program? Here is a list of additional resources referenced in this post:



Grant McCracken

Senior Director, Operations : Program Success at Bugcrowd.

Back To Top