Evaluation of Vanilla and RAG-Supported LLM Comprehension of the EU AI Act

This paper presents a comparative benchmarking study evaluating the regulatory comprehension of four open-source large language models, TinyLlama-1.1B-Chat, Gemma-2B-Instruct, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3, on the EU Artificial Intelligence Act (Regulation EU 2024/1689). A custom benchmark of 100 multiple-choice questions was constructed and classified across three dimensions, namely cognitive difficulty, knowledge type, and regulatory mechanism, with a balanced answer key distribution. Each model was evaluated under vanilla inference, relying exclusively on parametric knowledge, and retrieval-augmented generation (RAG), in which relevant Act passages are retrieved using a FAISS-indexed sentence embedding pipeline with standardized top three chunk retrieval applied uniformly across all models. Vanilla accuracy ranges from 22.0% for TinyLlama-1.1B to 80.0% for Llama-3.1-8B, indicating that larger models perform better. Under the retrieval configuration employed, RAG improves performance of the evaluated models, with gains from 3.0 percentage points for TinyLlama-1.1B to 17.0 for Gemma-2B and 10.0 for both 7B–8B models. Questions related to procedural knowledge, governance and enforcement emerge as weaknesses in the baseline, vanilla setting, partially mitigated by retrieval. Qualitative analysis identifies two RAG failure modes: retrieval failures, where the embedding mechanism returns informationally insufficient passages, and integration failures, where the correct passage is retrieved but not correctly exploited. The study contributes an empirical characterization of small and medium language model regulatory comprehension and a reusable 100-question benchmark.