Essays on Multimodal Machine Learning and Social Media Analytics

Social media platforms are rapidly gaining popularity as the leading channels for both entertainment and marketing. Content on these platforms inherently encodes information across multiple modalities, such as images and text, or video and audio. Building on existing literature that explores the impact of attributes such as content and influencer characteristics on social media engagement, this dissertation focuses on how to utilize information from unstructured social media content and design to enhance engagement. In Essay one, I introduce a novel, deep learning-based measure called Music-Motion Synergy (MM Synergy) to assess the quality of dancing videos shared on social media. To develop this measure, I propose a new Context-Aware Multi-Task Multimodal Transformer (CA-MulT-MTL) model. The model addresses four key challenges, including: (1) capturing core dancing content with sequential unstructured data on audio and body movement, (2) understanding the interactions between multimodal elements, (3) simultaneously predict diverse engagement metrics that reflect management-relevant objectives, and (4) incorporating contextual video information. I instantiate the CA-MulT-MTL model based on a sample of 79, 588 short-form dancing videos on TikTok and find it outperforms state-of-the-art deep learning benchmark models. The model-derived MM Synergy scores are validated by human ratings and informative of consumer responses to video content. Furthermore, through an online controlled experiment, I establish the causal impact of MM Synergy levels on consumers’ video watch intention, purchase intention, and the probability of liking. This proposed scalable machine learning method serves as a useful decision-support tool for influencers to develop popular dance videos and empowers platforms to not only leverage MM Synergy as a quality control tool prior to publishing videos but also embed it into recommendation systems to enhance business outcomes and customer experiences. In Essay two, I examine how consumer engagement evolves following the adoption of co-posting—a novel form of co-branding strategy on social media. Specifically, I investigate how co-posting affects both content characteristics and subsequent consumer responses. Drawing on a unique dataset of 223,435 Instagram posts from 199 fashion brands, I employ a stacked difference-in-differences model combined with propensity score matching to account for selection bias. The results show that adopting co-posting leads to an approximate 11.5% increase in consumer engagement, as measured by likes and comments. Beyond engagement metrics, I find that co-posting is associated with higher image-text alignment and improved visual novelty in the brand’s subsequent posts, suggesting that co-posting may encourage brands to create more cohesive and creative content. These findings contribute to our understanding of collaborative branding strategies in digital environments and offer actionable insights for firms aiming to optimize content strategy through co-branding on social platforms.