How Johnny Experiences Phishing Warnings: A Qualitative Study Investigating the Impact of Design Decisions on the User

To this day, phishing remains one of the most critical and elusive threats in cybersecurity. Although detection technologies have evolved and improved, they have not kept pace with novel phishing strategies. Thus, when software cannot definitively identify phishing, the last line of defense rests with the user when they are asked to “think before you click”. The appeal is commonly accompanied by warning messages, supposedly providing the user with enough information and incentive to make an informed, secure decision. However, warning messages must be carefully crafted because their elements can considerably affect the user’s agency, trust, and decision-making. We selected four of the key design elements in warning messages: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">content</i>, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">placement</i>, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">level of friction</i>, and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">timing</i>. We conducted a qualitative study using think-aloud sessions with 18 participants. Each participant was presented with phishing scenarios, accompanied by warning messages that differ in regard to those four elements of design, followed by a post-session interview. Thematic analysis revealed 13 themes across the four elements and from the analysis, novel insights emerged. For instance, timing changes the context in which users frame their concern: rather than being concerned about the potential consequences of clicking —as the warning intends— they become suspicious of the app displaying the message, fearing it may invade their privacy and violate their security. Our findings form a basis for future research about how to design and implement mechanisms, such as warning apps, that are more adaptable, targeted, and potentially more effective in protecting users from phishing attacks.