Bridging ERP Complexity Through Retrieval-Augmented Generation: Design and Evaluation of an Intelligent Question-Answering System for SMEs

Purpose/Background: Small and medium-sized enterprises (SMEs) that deploy Enterprise Resource Planning (ERP) systems face a persistent paradox: although ERP centralises organisational data, frontline users frequently lack the technical expertise to navigate complex menu structures, preventing efficient information retrieval for decision-making. This study presents a preliminary Research and Development (R&D) effort to design, build, and qualitatively evaluate a prototype intelligent question-answering system that connects to Odoo ERP through application programming interfaces (APIs) via Retrieval-Augmented Generation (RAG), enabling natural-language access to real SME business data. Methods: A single-organisation R&D case study was conducted at an SME in Khon Kaen Province, Thailand. The development cycle comprised problem analysis and requirement specification, system architecture design, prototype construction, integration and deployment, iterative testing and refinement, and multidimensional evaluation. The prototype was implemented with Chainlit (conversational interface), FastAPI (orchestration and tool-calling layer), and Odoo XML-RPC/JSON-RPC APIs (structured data retrieval). A fixed set of 20 test questions spanning four complexity levels (easy, moderate, complex, out-of-scope) was evaluated by three automated tools (OpenAI Evals, DeepEval, Ragas), by real-task verification against live ERP data, by five domain experts using 5-point Likert-scale questionnaires, and by three end users from the case-study organisation, who additionally completed the System Usability Scale (SUS). Given the very small expert and user samples, all human evaluation results are reported descriptively as preliminary, exploratory indicators rather than as statistically generalisable measures. Results: Automated evaluation achieved indicative pass rates of 95.00% (OpenAI Evals, 19/20), 90.00% (Ragas, 18/20), and 85.00% (DeepEval, 17/20). Descriptive expert feedback (n = 5) yielded an overall mean of 3.82 (high level), and descriptive end-user feedback (n = 3) yielded an overall satisfaction mean of 4.33 (highest level). The SUS score was 66.67/100, sitting at the boundary between ‘OK’ and ‘Good’ and revealing a divergence between high stated satisfaction and lower confidence in independent system use (item 9, raw mean = 2.33) and a stronger perceived need for expert assistance (item 4, raw mean = 2.67). These results are interpreted as preliminary diagnostic signals for further development rather than as confirmatory evidence. Conclusions: This preliminary R&D study suggests that a RAG-based ERP chatbot can meaningfully simplify ERP data access for SME users, while exposing persistent gaps in multi-step reasoning, user confidence, and data privacy boundaries that must be addressed in subsequent development cycles. The SUS pattern, in particular, suggests a ‘novelty effect’ in which users are enthusiastic about the natural-language interface yet remain anxious about correctness and stability during real tasks. Originality/Value: This work contributes a transparent, replicable preliminary R&D blueprint that combines (i) live API-mediated RAG over structured ERP data, (ii) a complementary multi-tool automated evaluation set (OpenAI Evals + DeepEval + Ragas), (iii) descriptive expert and end-user feedback, and (iv) SUS-based usability assessment, all documented for a single Thai SME using Odoo 18. The study explicitly positions itself as an early step toward larger, multi-site, and on-premise deployments of trustworthy ERP-integrated conversational agents.