Designing an Amharic Legal Question-Answering System Using Retrieval-Augmented Generation

This paper documents the design, implementation and planned evaluation of an Amharic legal question-answering prototype built on retrieval-augmented generation (RAG), and reports no performance results because the evaluation set it requires has not yet been built. Legal text in Amharic is hard to search, and few retrieval tools support the language. The prototype combines Scrapify, a Python tool that collects and normalizes Amharic web and PDF content, with a Zufan notebook that extracts the Amharic text of the Constitution of the Federal Democratic Republic of Ethiopia and defines a LangChain retrieval chain over a Chroma vector store that answers questions with Gemini 2.0 Flash. We describe each pipeline stage as implemented, report descriptive statistics for the 38-page constitutional corpus, and list correctness and reproducibility issues found by auditing the two public notebooks. We then specify a controlled evaluation protocol covering retrieval quality (Recall@k, MRR, nDCG), answer correctness, faithfulness, citation accuracy, abstention on unanswerable questions, and latency. The contribution is a documented baseline and a pre-specified protocol that lets later work measure the reliability of Amharic legal RAG rather than infer it from demonstrations.