HumanVLM: Foundation for Human-Scene Vision-Language Model

Human-scene vision-language tasks are increasingly prevalent in diverse social applications, yet recent advancements predominantly rely on models specifically tailored to individual tasks. Emerging research indicates that large vision-language models (VLMs) can enhance performance across various downstream vision-language understanding tasks. This study introduces a domain-specific Large Vision-Language Model, Human-Scene Vision-Language Model (HumanVLM), designed to provide a foundation for human-scene Vision-Language tasks. Specifically, (1) we create a large-scale human-scene multimodal image-text dataset (HumanCaption-10M) sourced from the Internet to facilitate domain-specific alignment; (2) develop a captioning approach for human-centered images, capturing human faces, human behavior, and backgrounds, and construct a high-quality Human-Scene image-text dataset (HumanCaption-HQ, about 311k pairs) that contain as much detailed information as possible about human; (3) Using HumanCaption-10M and HumanCaption-HQ, we train a HumanVLM. In the experiments, we then evaluate our HumanVLM across various downstream tasks, where it demonstrates superior overall performance among multimodal models of comparable scale, particularly excelling in human-related tasks and significantly outperforming similar models, including Qwen2-VL and ChatGPT-4o (as shown in Figure 1). HumanVLM, alongside the data introduced, will stimulate the research in human-around fields. All codes, data and model checkpoints are available at: https://github.com/ddw2AIGROUP2CQUPT/HumanVLM ; https://huggingface.co/OpenFace-CQUPT • Constructing a large-scale and high-quality human-scene image-text dataset. • Training a general Vision-Language Model for Human-Scene, achieving the best overrall performance. • Releasing all data, model checkpoints, and the codebase for model training.