Ribot: A Language Model That Reads Memory Networks It Never Saw

A language model gets new knowledge in two ways: in its context, or by training. The context is limited, and training is expensive and changes the whole model. I introduce Ribot: a language model with a small memory network as one of its parts. The memory network holds data in its own weights, and it is trained on it alone, without the rest of the model. The rest of the model is trained once to read memory networks, with swap training: a hundred or more frozen memory networks are swapped during its training, so the only way to answer is to read the one plugged in. Trained on made-up facts, Ribot reads memory networks it never saw at 92.4% exact, and 0% with the memory removed. Trained on ordinary words, it reads 500 facts from one memory network at 80% to 90% exact, with nothing in its prompt, about as accurate as retrieval (RAG) over the same facts. With the right fact alone in the prompt, the original model, SmolLM2-135M, is still more accurate at this size. The models and code are released. Models and code: https://huggingface.co/aikronus-labs/ribot-135m (version 0.2)