Impact of Capture The Flag (CTF)-style vs. Traditional Exercises in an Introductory Computer Security Class

The importance of possessing hands-on skills in addition to theoretical knowledge for cybersecurity researchers and professionals cannot be overstated. Computer security educators have therefore adopted various types of hands-on exercises in their classes, from traditional ones characterized by linear instructions and progress snapshots to full-scale Capture The Flag (CTF) challenges characterized by an underspecified solution path and a tangible token of success. Educators implementing CTF exercises have reported, often informally, that the exercises increase student motivation and/or engagement without harming learning outcomes relative to traditional exercises. In this work, we systematically compare exercises imbued with the essential characteristics of CTFs to traditional exercises in terms of student experience and learning outcomes in an undergraduate Introduction to Computer Security class. Quantitative and qualitative analyses of post-assignment surveys suggest that CTF-style exercises yielded significantly higher student motivation, and analysis of scores on exam questions relevant to each type of exercise showed no significant difference between exercise types. Our work agrees with previous studies showing the effectiveness of CTF-style exercises in increasing student motivation at no cost to objective learning outcomes, and helps establish the pedagogical validity of employing CTF-style exercises in an introductory computer security class.