Attribute mining of multisource and multimodal data from social media based on generalized scene graphs

In today's highly connected world, everyone leaves behind digital traces on the internet. While analyzing these individual data points might not reveal valuable sensitive information, combining and associating multiple types of data can uncover new sensitive attributes. This paper takes multi-modal data from social media as a case study to mine the underlying sensitive attributes. We propose a multi-modal data attribute mining method that includes feature representation of different modalities and modeling the relationships between features and sensitive attributes. Specifically, we introduce a generalized scene graph that represents features from both image and text modalities in a unified triplet form. Furthermore, we employ a Transformer to fuse information across different modalities and extract frequent feature representations associated with various sensitive attributes. Experimental results on the proposed dataset demonstrate that our method effectively establishes associations between extracted features and attributes from multi-source, multi-modal data and predicts the likelihood of sensitive attribute presence. This validates the potential for mining sensitive attributes from multi-source and multi-modal data in social media.