Reprogramming LLMs for Multimodal Time Series Forecasting

Human pose prediction plays a vital role in applications such as human-computer interaction, robot control, and intelligent surveillance. Traditional methods primarily rely on single-modal inputs such as skeletal keypoints or RGB video, which often fall short in comprehensively understanding complex human motions. In this thesis, we propose a novel multimodal Time-LLM framework that fuses structured skeletal data (in CSV format) with monocular video information for future pose prediction. Building upon the Time-LLM architecture, we introduce a visual patch reprogramming module that maps spatiotemporal visual features into discrete embedding representations understandable by large language models (LLMs). These visual embeddings are concatenated with prompt tokens and skeletal time-series embeddings to form a unified input sequence for the LLM, thereby enabling cross-modal joint reasoning. Our method fully leverages the strengths of pre-trained LLMs in temporal modeling and generalization, demonstrating powerful representational and reasoning capabilities on nonlinguistic sequence prediction tasks—without the need for fine-tuning. Finally, we propose exploring the framework’s potential applications in broader domains such as self-driving, multimodal biometrics, security surveillance, and abnormal behavior detection as part of future work. These directions remain conceptual and are not experimentally validated in this thesis.