SparQ Attention: Bandwidth-Efficient LLM Inference
CoRR(2023)
摘要
Generative large language models (LLMs) have opened up numerous novel
possibilities, but due to their significant computational requirements their
ubiquitous use remains challenging. Some of the most useful applications
require processing large numbers of samples at a time and using long contexts,
both significantly increasing the memory communication load of the models. We
introduce SparQ Attention, a technique for increasing the inference throughput
of LLMs by reducing the memory bandwidth requirements within the attention
blocks through selective fetching of the cached history. Our proposed technique
can be applied directly to off-the-shelf LLMs during inference, without
requiring any modification to the pre-training setup or additional fine-tuning.
We show how SparQ Attention can decrease the attention memory bandwidth
requirements up to eight times without any loss in accuracy by evaluating Llama
2 and Pythia models on a wide range of downstream tasks.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要