Understanding practitioners’ challenges and requirements in the design, implementation, and evaluation of anti-phishing interventions

Research shows that the ineffectiveness of anti-phishing interventions can result from practitioners’ failure to consider end-users’ requirements in the intervention design, implementation, and evaluation. To assist practitioners in addressing usability issues, we reported 41 guidelines through a systematic Multi-vocal Literature Review (MLR). The usefulness of these guidelines in real-world scenarios remains uncertain until the involved challenges and requirements to implement them are investigated. (1) To investigate practitioners’ challenges in the design, implementation, and evaluation of phishing interventions in real-world settings; (2) to understand practitioners’ perspectives on our guidelines and how they can be made easily accessible to the practitioners. We interviewed 18 practitioners (intervention designers, security practitioners, and C-suite employees) from 18 organizations in 6 countries. (1) We identify 8 challenges in training content design, anti-phishing datasets, post-training knowledge assessment, and so on. We compare these challenges with the challenges identified from our MLR to demonstrate the ecological validity of the challenges found in MLR and derive a set of insights to overcome them; (2) we report practitioners’ feedback on our guidelines; (3) we gather actionable features on an envisioned tool to make these guidelines easily accessible. Conclusion: We provide 15 recommendations to improve the anti-phishing defense in the organisations.