Mitigating Risks in Marketplace Semantic Search: A Dataset for Harmful and Sensitive Query Alignment

Semantic search engines have transformed user interaction with online marketplaces, creating a need for effective methods to moderate harmful and sensitive content.Existing approaches often struggle with ambiguous query intent, content classification challenges, and noisy data, making it difficult to ensure user safety while maintaining relevance in search results.To address these challenges, we introduce SHIELD, a synthetic dataset designed for classifying user queries into harmful, sensitive, and normal categories.SHIELD is generated using a large language model with a structured taxonomy, followed by automated filtering using a reward model to ensure data quality and relevance.To demonstrate SHIELD's utility, we evaluate three classification approaches: (1) BM25, a computationally efficient retrieval-based method; (2) a sentence transformer with FAISS, which improves classification by leveraging semantic embeddings; and (3) MoralBERT, a fine-tuned transformer model trained on SHIELD for direct query classification.We discuss the trade-offs among these methods in terms of accuracy, resource requirements, and explainability, highlighting their applicability in real-world semantic search systems.This work provides a foundation for developing AI-driven content moderation systems in semantic search, offering insights into the trade-offs between efficiency, accuracy, and explainability.The SHIELD dataset, pre-trained model, and generation details are publicly available to support future research and real-world deployment: https://github.com/flpspacek/SHIELD.Warning: This paper contains examples of language and content that some readers may find offensive.These examples are included solely to illustrate challenges in content moderation and to highlight the importance of ethical considerations in semantic search systems.