Self-Policy Distillation via Capability-Selective Subspace Projection
- Type
- other
- Venue
- arXiv / HKUST / University of Chicago / University of Cambridge
Summary
SPD SVD-extracts a low-rank K/V subspace from gradients on correctness-defining tokens (answer span / assertions), hooks those projections during self-generation, then LoRA-SFT on the raw hooked completions. No teacher, reward, or filter. Up to 13% over SSD/PSR and 16% over base across code/math/QA on five instruct backbones; QA-calibrated SPD also lifts math/code OOD. Correctness-aligned loss beats full-sequence loss (MBPP 11.9→25.5). Calibration works with ~50 examples. No official code on abs.
Keywords
self-distillation · spd · kv-projection · lora · qwen2.5 · llama · hkust · uchicago · cambridge
Topics
self-distillation, representation steering, KV subspace
Research notes
- Primary: arxiv abs (cs.CL). ArXiv HTML states perpetual non-exclusive license. Shang HKUST, Zhao UChicago, Liang Cambridge; Hao affiliation not stated on abs. Zhao/Liang joint last. Correspondence hl589@cantab.ac.uk, zhuokai@uchicago.edu, ytshang@ust.hk. No official code on abs. HF has no paper page (API 404). Discord posted PDF. Uses public MBPP/CodeAlpaca/GSM8K/SVAMP/MMLU/BBH; self-generated corpus is per-run, not a standalone public release, so no datasets_local row. License field left blank per catalog convention.