Each summer, members of the security community convene in Las Vegas for a week of talks, networking, and other activities at a series of conferences. At DEF CON specifically, a number of organizations host Capture the Flag (CTF) hacking competitions in which contestants either compete against each other trying to access other teams’ infrastructure while defending their own (“Attack with Defense”), or by racing to rack up the most points before the contest ends with answering standalone questions (“Jeopardy style”).
Last year at DEF CON 24, the Car Hacking Village (CHV) ran its own (Jeopardy-style) CTF for the first time, specifically targeting an “intro-level” difficulty in an effort to attract new people to the field of automotive security. Among the 20 actively participating teams, Team MK ended up taking home a Black Badge.
— Car Hacking Village (@CarHackVillage) August 7, 2016
This year, with many more organizations pitching in to help with CHVCTF (including Derive Systems sponsoring a brand new Ford F-150 truck as the grand prize), we had over 150 registrants and nearly 50 actively participating teams working on challenges that included remotely taking over and controlling a CAN-based RC car.
While we used the same Python/Flask-based CTF platform (CTFd) to manage teams, challenges, and scores that we did last year, both our method of deployment and experience running the infrastructure were much more interesting this year… For starters, our total cloud infrastructure bill amounted to less than half of last year’s!
Zappa, Lambda, and CTFd
Over the past few years, since the introduction of mainstream cloud-based code execution environments such as AWS Lambda, people have written a number of tools and frameworks to manage their usage and integration with other services. These include projects like Serverless for Node.js and Zappa for Python/Flask that allow developers to easily deploy web applications by programmatically triggering small standalone functions in order to serve requests, rather than relying on a traditional monolithic web server instance (or even a set of containerized web microservices running on that instance).
(Note that “serverless” is a somewhat misleading term; since standalone functions like AWS Lambdas are still running on and managing the creation of servers, so you can also think of these types of deployments as “Function-as-a-Service.”)
While this type of deployment isn’t ideal for all applications, in situations where it does make sense, you can often save a fair bit on cloud operating costs vs. using actual instances since both Lambda and API Gateway are priced based on volume and resource usage (compute memory/time and bandwidth respectively).
For a relatively lightweight web app like CTFd, even over the course of a larger CTF event, you can expect usage to fall well within the lower pricing tiers since individual requests require little RAM and usually less than a couple hundred milliseconds of compute time. For the entire month of July when we started taking signups through the end of DEF CON, this was our bill for Lambda and API Gateway usage:
Even a standalone t2.nano (the smallest AWS EC2 general compute instance) still would have cost around $4-5 for the same period, but that would have not been able to handle our peak loads. An appropriately-sized EC2 instance would have cost at least 10x the total listed above! Additionally, since API Gateway only starts throttling by default at 10k requests/second, other services such as the RDS instance would have created a bottleneck in the absence of rate limiting long before we had to worry about webserver-related performance issues. Overall, the RDS instances and other external services made up the majority of the total infrastructure cost.
While our work on this deployment of CTFd started at a company hackathon without a ton of time to prepare for production use at DEF CON, we enjoyed the challenge and had a better experience running the CHVCTF infrastructure than we did last year.
Stay tuned for the second part of this series, where we’ll talk about how we handled a last-minute fire…
Thanks to all of this year’s CHVCTF sponsors and participants, and congratulations to Team robamierdas for winning first place!
— Derive (@DeriveSystems) July 30, 2017
— Javier Vazquez Vidal (@fjvva) August 1, 2017