Semantic profile construction for explainable and semantically consistent multimodal fashion recommendation

Fashion recommendation requires heterogeneous visual and textual product content to be transformed into representations that are both retrievable and interpretable. We present FACTOR (Feature-Aware Contextual Optimization and Refinement), a framework that constructs six-field semantic profiles comprising category, subcategory, color, material, pattern, and cross-dataset style information, and combines them with image and multimodal retrieval. FACTOR integrates hierarchy-constrained category–subcategory decoding, cosine-based semantic retrieval, and late fusion, and is evaluated using both profile-derived consistency metrics and independent human judgments. FACTOR achieves category accuracy of 0.93, subcategory accuracy of 0.72, and semantic similarity of 0.85–0.87 for color, material, and pattern. Profile-based retrieval reaches weighted attribute consistency of approximately 0.99, exceeding image-only retrieval in semantic-profile agreement. In a 125-image in-domain audit, FACTOR achieved category/subcategory accuracy of 0.78/0.50 and multi-label F1 of 0.39, 0.22, and 0.28 for color, material, and pattern. LLaMA 3 metadata extraction achieved 0.64 precision (95% CI [0.58, 0.69]), identifying unsupported and missing labels as the main upstream risks. Hierarchy-constrained decoding reduced invalid category–subcategory pairs from 3.4% to 0% without new joint errors. In a matched ablation, the full FACTOR profile improved Human nDCG@5 over the base probe by 0.05 (95% CI [0.02, 0.09], Holm-adjusted p = 0.02), primarily through auxiliary refinement. On an independently judged 50-query benchmark, image-only FashionCLIP achieved the highest Human nDCG@5 (0.781), followed by native multimodal retrieval (0.707), rank fusion (0.674), and text-only retrieval (0.467). These findings show that structured semantic profiles provide strong semantic consistency and traceability, while visual representations remain essential for human-perceived relevance.