Source research

Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

Lingling Xu, Haoran Xie, S. Joe Qin, Xiaohui Tao, Fu Lee Wang · 2026 · 115 citationsRead the paper

With the continuous growth in the number of parameters of the Transformer-based pretrained language models (PLMs), particularly the emergence of large language models (LLMs) with billions of parameters, many natural language processing (NLP) tasks have demonstrated remarkable success. However, the enormous size and computational demands of these models pose significant challenges for adapting them to specific downstream tasks, especially in environments with limited computational resources. Parameter-Efficient Fine-Tuning (PEFT) offers an effective solution by reducing the number of fine-tuning parameters and memory usage while achieving comparable performance to full fine-tuning. The demands for fine-tuning PLMs, especially LLMs, have led to a surge in the development of PEFT methods, as depicted in Fig. 1. In this paper, we present a comprehensive and systematic review of PEFT methods for PLMs. We summarize these PEFT methods, discuss their applications, and outline future directions. Furthermore, extensive experiments are conducted using several representative PEFT methods to better understand their effectiveness in parameter efficiency and memory efficiency. By offering insights into the latest advancements and practical applications, this survey serves as an invaluable resource for researchers and practitioners seeking to navigate the challenges and opportunities presented by PEFT in the context of PLMs.

What the full text shows

Shown in the lab

ProPELTadapter achieved optimal average performance on GLUE, outperforming full fine-tuning (FT) for both RoBERTa-base and RoBERTa-large while using minimal parameters. — 1.30% higher than RoBERTa-base (FT) and 1.65% higher than RoBERTa-large (FT) using 1.50% of trainable parameters (RoBERTa-base/large on GLUE benchmark)
QLoRA significantly reduces GPU memory usage compared to full fine-tuning, especially as model size increases. — 66.66% reduction for LLaMA-7B and 76.86% reduction for LLaMA-13B (LLaMA-7B/13B-Alpaca)
LoRA and (IA)3 maintain comparable performance to full fine-tuning on translation tasks with drastically fewer parameters. — (IA)3: 0.03% parameters (0.16 BLEU higher for T5-base); LoRA: 0.39% parameters (0.36 BLEU higher for T5-base) (T5-base on WMT16 En-Ro dataset)
Full fine-tuning of LLaMA models produces better accuracy than PEFT, though PEFT methods greatly reduce parameter counts. — (IA)3 performs 2-4% lower than FT; LoRA/QLoRA perform ~2% lower than FT (LLaMA-7B/13B on 5-shot MMLU test set)
Certain PEFT methods (sequential adapter, MAM adapter, Child-TuningD, ProPELT) unexpectedly increase GPU memory usage compared to full fine-tuning. — Sequential and MAM adapters use ~3x more memory than FT (RoBERTa-base/large)
Prompt-tuning with a virtual marker length of 20 is the most parameter-efficient but the least performant. — Average performance ~10% lower than full fine-tuning (RoBERTa-base/large on GLUE benchmark)

Setup: Experiments conducted on A800 GPU using RoBERTa (base/large), T5 (base/large), and LLaMA (7B/13B-Alpaca) on GLUE, WMT16 En-Ro, and MMLU benchmarks.
How: The authors conducted a systematic review and taxonomy of PEFT methods (additive, partial, reparameterized, hybrid, and unified). They performed comparative experiments measuring accuracy, trainable parameter counts, and peak GPU memory usage across various PLMs.
Limitations: Some PEFT methods (e.g., sequential adapter) increase memory consumption relative to full fine-tuning. · Full fine-tuning performance on LLaMA is extremely sensitive to the learning rate. · Performance fluctuations observed in 5-shot MMLU accuracy during training for LoRA and QLoRA.

Summarized by Gemma 4 from the open-access full text · 13 Oct 2026. Read it