LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks

LoRA employs lightweight modules to customize large language models (LLMs)for each downstream task or domain, where different learned additional modulesrepresent diverse skills. Combining existing LoRAs to address new tasks canenhance the reusability of learned LoRAs, particularly beneficial for taskswith limited annotated data. Most prior works on LoRA combination primarilyrely on task-level weights for each involved LoRA, making different examplesand tokens share the same LoRA weights. However, in generative tasks, differenttokens may necessitate diverse skills to manage. Taking the Chinese math taskas an example, understanding the problem description may depend more on theChinese LoRA, while the calculation part may rely more on the math LoRA. Tothis end, we propose LoRA-Flow, which utilizes dynamic weights to adjust theimpact of different LoRAs. The weights at each step are determined by a fusiongate with extremely few parameters, which can be learned with only 200 trainingexamples. Experiments across six generative tasks demonstrate that our methodconsistently outperforms baselines with task-level fusion weights. Thisunderscores the necessity of introducing dynamic fusion weights for LoRAcombination.