Instruction-tuned Language Models Are Better Knowledge Learners

In order for large language model (LLM)-based assistants to effectively adaptto evolving information needs, it must be possible to update their factualknowledge through continued training on new data. The standard recipe for doingso involves continued pre-training on new documents followed byinstruction-tuning on question-answer (QA) pairs. However, we find that LLMstrained with this recipe struggle to answer questions, even though theperplexity of documents is minimized. We found that QA pairs are generallystraightforward, while documents are more complex, weaving many factualstatements together in an intricate manner. Therefore, we hypothesize that itis beneficial to expose LLMs to QA pairs before continued pre-training ondocuments so that the process of encoding knowledge from complex documentstakes into account how this knowledge is accessed through questions. Based onthis, we propose pre-instruction-tuning (PIT), a method that instruction-tuneson questions prior to training on documents. This contrasts with standardinstruction-tuning, which learns how to extract knowledge after training ondocuments. Extensive experiments and ablation studies demonstrate that PITsignificantly enhances the ability of LLMs to absorb knowledge from newdocuments, outperforming standard instruction-tuning by 17.8