Information Leakage Measures for Imperfect Statistical Information: Application to Non-Bayesian Framework

This paper analyzes the problem of estimating information leakage when the complete statistics of the privacy mechanism are not known, and the only available information consists of several input-output pairs obtained through interaction with the system or through some side channel. Several metrics, such as subjective leakage, objective leakage, and confidence boost, were introduced before for this purpose, but by design only work in a Bayesian framework. However, it is known that Bayesian inference can quickly become intractable if the domains of the involved variables are large. In this paper, we focus on this exact problem and propose a novel approach to perform an estimation of the leakage measures when the true knowledge of the privacy mechanism is beyond the reach of the user for a non-Bayesian framework using machine learning. Initially, we adapt the definition of leakage metrics to a non-Bayesian framework and derive their statistical bounds, and afterward, we evaluate the performance of those metrics via various experiments using Neural Networks, Random Forest Classifiers, and Support Vector Machines. We have also evaluated their performance on an image dataset to demonstrate the versatility of the metrics. Finally, we provide a comparative analysis between our proposed metrics and the metrics of the Bayesian framework.