MEMORYLLM: Towards Self-Updatable Large Language Models

Existing Large Language Models (LLMs) usually remain static after deployment,which might make it hard to inject new knowledge into the model. We aim tobuild models containing a considerable portion of self-updatable parameters,enabling the model to integrate new knowledge effectively and efficiently. Tothis end, we introduce MEMORYLLM, a model that comprises a transformer and afixed-size memory pool within the latent space of the transformer. MEMORYLLMcan self-update with text knowledge and memorize the knowledge injectedearlier. Our evaluations demonstrate the ability of MEMORYLLM to effectivelyincorporate new knowledge, as evidenced by its performance on model editingbenchmarks. Meanwhile, the model exhibits long-term information retentioncapacity, which is validated through our custom-designed evaluations andlong-context benchmarks. MEMORYLLM also shows operational integrity without anysign of performance degradation even after nearly a million memory updates. Ourcode and model are open-sourced at https://github.com/wangyu-ustc/MemoryLLM.