Vision-semantic alignment with hierarchical consistency constraints for multi-attribute museum artifact classification

The digitization of museum collections has produced vast multimodal data, yet existing automated classification methods treat classification attributes as independent flat categories and overlook the hierarchical taxonomic structures embedded in museum cataloging systems, frequently yielding predictions that are taxonomically illegitimate. We present a multimodal hierarchical learning framework for multi-attribute museum artifact classification that combines vision-semantic alignment with hierarchical consistency constraints. The framework extracts visual and textual features through a ViT and BERT dual-encoder, aligns the two modalities in a shared space via a contrastive objective trained on cultural-heritage data, adaptively fuses them through a gating mechanism, and performs joint hierarchical prediction across multiple taxonomy trees using cascaded conditional classification heads. The core contribution is a hierarchical consistency loss that explicitly penalizes incompatible prediction distributions between adjacent levels during training, complemented by a constrained decoding strategy that guarantees the structural legitimacy of output paths at inference. On the Metropolitan Museum of Art Open Access dataset, the framework achieves an average hierarchical F1 of 85.2%, a gain of 6.9 percentage points over the strongest multimodal baseline, and reduces the model’s learned hierarchical violation rate to 2.8%, with constrained decoding eliminating the remaining violations at inference. On the SILKNOW silk textile corpus, ten epochs of fine-tuning surpass the reported baselines under the evaluated transfer setting. Ablations and visualization analyses further clarify the contribution of each component and the model’s internal decision-making behavior.