A Supervised Evaluation Framework for Privacy Risk Scoring Models

The rise of online social networks (OSNs) has heightened concerns regarding user privacy, as sensitive attributes disclosed on profiles are increasingly susceptible to misuse, including identity theft and targeted manipulation. Each user's privacy risk varies based on the nature of the shared data and its intended audience. To quantify these risks, researchers have introduced privacy risk scores, inspired by credit scoring systems, to measure vulnerability to privacy violations. However, despite the proliferation of scoring models, their evaluation frameworks often rely on unsupervised methods, such as goodness-of-fit tests, which limit their practical reliability. To address the limitation, we propose SPR-EVAL, a supervised evaluation framework that systematically assesses the performance of privacy scoring models using various real-world attack scenarios, offering a more robust and actionable approach to privacy risk assessment. SPR-EVAL integrates simulations of various real-world privacy attacks as a core evaluation mechanism. The framework is adaptable to any OSN dataset and supports the incorporation of diverse privacy risk scoring models and privacy attacks. To validate the proposed framework, we conducted experiments on a real-world Facebook OSN dataset. The results highlight the effectiveness of SPR-EVAL in evaluating and comparing popular privacy scoring models under supervised conditions. By offering a rigorous supervised evaluation metric, SPR-EVAL overcomes the limitations of traditional unsupervised methods, representing a notable advancement in the domain of privacy risk scoring for OSNs.