Have you ever asked yourself any of the following questions: is our engagement doing well? Do hackers like testing on our engagement? How do we compare to other engagements? If so, keep reading. In this post I’ll break down five easy to understand success indicators that will help you determine how your engagement is performing, in a matter of seconds.
Intro
Hi! My name is Matias, or Mati for short. I’ve been a Technical Customer Success Manager (TCSM) at Bugcrowd for almost three years now, and I’ve launched and managed hundreds of bug bounty engagements across multiple industries, such as finance, retail, computer software, and many others, taking them from zero submissions to multiple P1s (critical vulnerabilities) in a year. I’d like to say I’ve seen it all, but I never stop learning because the bug bounty space never stops evolving!
At some point, your TCSM has probably asked you about your success criteria: a metric used to determine what value you’re getting from your engagement according to your goals and needs. Some proven engagement metrics have been discussed at length here and here.
Defining a success criteria that best suits your engagement goals is the easiest and most effective way to ensure you thrive on our platform. A thoughtful success criteria can be tracked, measured, and adjusted, therefore making it obvious what value you’re receiving and how you may want to pivot. For example, if you define a successful engagement as 10 unique valid submissions every month, and last month you got 15, then all is well with the world. But if you only got seven submissions, you need to make some changes to your engagement according to the recommendations of your TCSM.
How do I determine my success criteria?
Not to worry! There are a few tricks I’ve learned over my time at Bugcrowd that I personally use to be able to quickly tell if an engagement has set a healthy success criteria and is doing well. I’ve found that by considering these few simple tricks, you can bet that your engagement is going to be successful in the long term:
- Rewards given in the last month/quarter/year. This means that what you have spent in a given period is going towards valid reports that are useful and valuable to you. If you are topping up the reward pool on a regular basis, that’s usually a good sign that your program is doing well. This is also the easiest and quickest way to get a pulse on your engagement success.
- Number of high and critical reports that have been accepted and rewarded in the last month/quarter/year. Everyone loves to see P1s and P2s, especially hackers. When they find one and it’s triaged as valid (and is then rewarded quickly), it’s likely that it will motivate hackers to hunt for more. It’s rare that a hacker finds a P1 on a target and decides to change engagements out of the blue. More often than not, they’ll stick to an engagement for a sufficient period of time to get familiar with its targets before moving on to the next.
- The same hackers are reporting vulnerabilities on a regular basis. You are probably already familiar with a few names on your engagement, and that’s a good thing! That means you have acquired the loyalty of a few top hackers in the crowd, and that happens because they feel respected and well treated when submitting reports to you. You respond quickly, reward fairly, and are looking for a collaboration and not just a service. Well done!
- Variety in hackers. The opposite can also be true if looked through a different perspective, especially on public engagements. When many different hackers are hunting on your targets and submitting vulnerabilities, it usually means that the brief is enticing and attracting talent. It stands out among other engagements and hackers want to test their skills against your assets. You want this variety of talent in your engagement, so embrace it. Maybe you can turn them into loyal testers in the future!
- Your processing queue is always greater than 0. Submissions simply don’t stop coming in! They may or may not be exactly what you want or expect, but it’s a great indicator of engagement from the crowd. If this happens as a consequence of a recommendation your TCSM gave you, even better! It means that it’s working, and that you can apply the recommendations again in the future to try and achieve the same or better results.
Even then, meeting some or all of the above points might still not be enough for you or your company’s goals. That’s okay, it’s a great opportunity to work with your TCSM towards defining a success criteria that works for you and iterating over different recommendations and changes to your engagement in order to meet that metric. Don’t hesitate to tell us exactly what you want as often as you can and we will gladly help you!
Conclusion
Nothing is definite. Scopes change, hackers come and go, and there isn’t a golden rule to follow that will guarantee that your engagement is the most successful engagement in the world. That said, hopefully the indicators I’ve shared with you can help you determine what it is that you’re looking for, and help you define the success metric that will bring you the most value.
