Handwritten Signature Verification via Multimodal Consistency Learning

Multimodal handwritten signatures usually involve offline images and online sequences. Since in real-world scenarios, different modalities of the same signature are generated simultaneously, most research hypothesizes that the different modalities are consistent. However, attacks launched on a partial modality (e.g., only tampering on the image modality) of signature data are commonly seen, and will cause the inter-modal inconsistency. In this paper, we propose and analyze the multimodal security and attack levels for handwritten signatures, and provide a multimodal consistency learning method to detect different levels of attacks of signatures. The modalities include not only traditional offline and online data, but also videos capturing hand movements. We collect a number of triple modal signatures to address the scarcity of public handwritten video datasets. Then, we extract hand joint sequences from videos and utilize them to analyze subtle multimodal consistency with the online modality. We provide extensive experiments for the consistency between online and offline signatures, as well as between online signatures and movement videos. The verification involves distance-based and classification-based fusion models, showing the most effective discriminative networks for attack detection and the superiority of consistency learning.