Smaller Language Models Are Capable of Selecting Instruction-Tuning Training Data for Larger Language Models

Instruction-tuning language models has become a crucial step in aligning themfor general use. Typically, this process involves extensive training on largedatasets, incurring high training costs. In this paper, we introduce a noveltraining data selection based on the learning percentage of the samples. Weassert that current language models possess the capability to autonomouslyselect high-quality training data, leading to comparable or improvedperformance compared to training on the entire dataset. Our experiments spandifferent-sized models, revealing that this characteristic holds for modelsranging from 1B (small) to 13B (large) in size. Moreover, we demonstrate aninteresting finding that the data hardness transfers across model sizes, and asmaller 350M model can effectively curate high-quality training data with hardsamples for a larger 13B model, resulting in an equally or superiorinstruction-tuned model compared to training on the complete dataset. Utilizingopen-sourced OPT and Llama-2 models up to 13B in size, two publicly availableinstruction-tuning training datasets and evaluated by both automatic metrics humans, our paper introduces a novel approach to training data selection,showcasing a more efficient alternative.