Digital Resources to Long-Term Agent Memory: Evaluating Memory Formation and Retrieval Across Heterogeneous Personal Knowledge
The ability to retain and retrieve information from heterogeneous digital resources is an important requirement for long-term memory in intelligent agents. This paper evaluates different memory representation and retrieval approaches for organizing personal knowledge collected from multiple digital sources. The evaluated dataset contains 100 resources across five categories: webpages, video transcripts, GitHub repositories, research papers, and personal notes. Of these, 96 resources were successfully processed. A question set containing 150 questions was used, with 30 questions allocated for development and 120 for testing. Five approaches were compared: BM25 retrieval, dense retrieval-augmented generation (RAG), summary-based memory, structured-fact memory, and graph-linked memory. The evaluation considered retrieval effectiveness, answer accuracy, evidence support, fact extraction quality, provenance preservation, and storage and retrieval efficiency. Among the evaluated approaches, graph-linked memory achieved the highest retrieval performance, with a Recall@5 of 0.83, MRR of 0.70, and nDCG@5 of 0.76. It also achieved the highest answer accuracy at 77%. Structured-fact memory achieved the highest fact extraction F1-score of 0.90 and provenance score of 0.98, while requiring less storage and providing lower median retrieval latency than graph-linked memory. These findings illustrate the trade-offs between retrieval effectiveness, memory representation quality, and computational efficiency when building long-term memory from heterogeneous personal knowledge.
Authors
- vansh
- kishore sabavath
- devansh
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-15
- DOI
- https://doi.org/10.5281/zenodo.24241304
- Primary Topic
- Information Retrieval and Search Behavior
- Type
- article
- Field-Weighted Citation Impact
- 0.00