Advancing Financial Standards Comprehension through Domain-Specific MoE Architecture

This paper presents a novel Mixture of Experts (MoE) approach for optimizing Large Language Models in the interpretation of financial and accounting standards within Task 4 of the FinRL Contest 2025, focusing on the Common Domain Model, Markets in Financial Instruments Directive Operational Framework, and eXtensible Business Reporting Language. Operating under a 45GB VRAM hardware constraint, domain-specific data preparation was implemented using DeepSeek V3, and a Qwen 0.5B model was converted to MoE architecture with 16 experts, totaling 5.51B parameters. The router was initialized with random weights while experts retained pretrained knowledge of the instruct model. The research demonstrates that targeted fine-tuning with MoE architecture produced mixed results, significantly improving performance in recall related tasks, while reducing performance of more complex ones compared to a baseline model Qwen7B.