Optimizing Cross-Modality Alignment Module for Audio Large Language Models

Extending text-based Large Language Models (LLMs) to multimodal settings often requires an alignment module to bridge representations across different modalities. While various alignment strategies have been explored, their comparative effectiveness remains underexamined, particularly across different task types. In this work, we analyze representative alignment modules in the context of audio instruction-following tasks, comparing their strengths and limitations across different task settings. Based on these insights, we introduce a layer-wise attention pooling network (LWAPN) as an optimization strategy to capture multi-scale feature representations from the audio encoder more effectively. We further compare LWAPN with alternative fusion methods and find it performs strongly across most tasks. Further analysis reveals that different tasks exhibit varying sensitivities to fusion strategies, highlighting the importance of method selection. These results underscore the practical advantages of our approach and suggest promising directions for future improvements in audio LLMs design.