Irregularly Sampled Time Series Classification via Fusing Language Models and Multi‐Channel Image Representations

ABSTRACT Multivariate time series have broad applications in domains such as healthcare, finance, and climate science. However, irregular sampling often leads to severe data gaps and misaligned variables across dimensions, making time series forecasting under such conditions particularly challenging. Common approaches typically fill in missing values via interpolation before applying classic time series models—such as Transformers—for prediction. Yet these methods suffer from two major drawbacks: first, conventional interpolation techniques often fail to restore missing data accurately, resulting in imputed values that diverge substantially from the true observations; second, common models struggle to learn both the temporal dynamics of individual variables and the interdependencies among multiple variables in a unified manner. To address both challenges, this paper proposes a novel strategy. First, it uses a fixed template to convert time series data into text prompts and utilizes a large language model to extract text embeddings. Second, it employs a message passing mechanism to fully exploit the correlations among variables in an irregularly sampled series, yielding more reasonable imputations. Next, it transforms the time series into multi‐channel images, a representation that not only preserves the temporal dynamics of each variable but also captures the interactions among different variables at each time point. Finally, it adopts a dual‐modal alignment method for text and time series data based on multi‐head attention to learn the time series information and classify it. Experimental results on the PAM, P12, and P19 datasets demonstrate that the proposed method achieves superior predictive performance compared to common approaches for irregularly sampled time series classification.