MMInfluencer: A Multimodal AI Framework for Influencer Marketing Tasks Using Large Language Models

Influencer marketing involves brands partnering with influencers to promote products, often through combined text and image posts on social media platforms. This multimodal data is critical for identifying suitable products for effective marketing practice. In this study, we presented MMInfluencer, a novel framework that addressed the critical upstream challenge of knowledge extraction from unstructured multimodal social media data. Our framework operationalized communication theories, the heuristic-systematic model (HSM) and dual coding theory (DCT), to guide large language models (LLMs) in constructing a high-quality knowledge graph (KG) from the multimodal posts and biographies of 1000 Instagram influencers. Human assessment of the KG confirmed its relevance and completeness with accuracy scores of 0.89 and 0.75, respectively. We then examined the KG’s efficacy in various influencer marketing tasks using graph-based learning methods and retrieval-augmented generation (RAG) technology. The KG derived from multimodal data significantly improved product category prediction and recommendation tasks, with the Node2Vec model showing improvements of 20.47% (AUROC, p <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&lt;$</tex-math></inline-formula> 0.0001) and 1142.15% (NDCG@20, p <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&lt;$</tex-math></inline-formula> 0.0001), respectively. In contrast, the RAG method was ineffective, yielding an accuracy of just 0.1195 (<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\pm$</tex-math></inline-formula> 0.0162). Furthermore, the LLM-extracted KG significantly improved performance in challenging scenarios, where the best model achieved improvements of 155.51% (NDCG@20, p <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$=$</tex-math></inline-formula> 0.0170) for emerging influencers (zero-shot) and 1144.51% (NDCG@20, p <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&lt;$</tex-math></inline-formula> 0.0001) for influencers with scarce data (cold-start). Our research underscores the potential of leveraging multimodal data, LLMs, and graph-based learning methods for effective influencer marketing.