PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

Summary

PiSSA (Principal Singular values and Singular vectors Adaptation) is a drop-in LoRA replacement that initializes adapters with the principal components of the pretrained weights instead of Gaussian noise, freezing the residual. It converges faster and beats LoRA consistently — e.g., +5.16% on GSM8K with Mistral-7B — and its quantized variant QPiSSA beats QLoRA.

Keywords

peft · lora · fine-tuning · quantization

Topics

peft, lora, fine-tuning, quantization

Research notes

• Method: Same architecture as LoRA, but A/B adapters are initialized with the principal singular vectors/values of the original weight matrix W, while the residual (W minus principal components) is frozen. Fine-tuning updates the principal components rather than a random 'noise & zero' adapter.
• Key findings: Consistently outperforms LoRA under identical setups across 12 models (184M–70B), 5 NLG + 8 NLU tasks; Mistral-7B on GSM8K: 72.86% vs LoRA 67.7% (+5.16%); QPiSSA on LLaMA-3-70B GSM8K: 86.05% vs QLoRA 81.73%; smaller quantization error in early stages; Fast SVD initialization takes seconds — negligible cost to switch from LoRA
• Limitations: Requires an SVD of the pretrained weights at setup (cheap but an extra step); gains demonstrated mainly on math/NLU benchmarks — breadth on other task families less established.
• v4 (Apr 2025) is the current arXiv version. Compatible with quantization (QPiSSA).