Reverse That Number! Decoding Order Matters in Arithmetic Learning

Daniel Zhang-Li, Nianyi Lin, Jifan Yu, Zheyuan Zhang, Zijun Yao, Xiaokang Zhang, Lei Hou, Jing Zhang, Juanzi Li

Type: other
Venue: arXiv / Tsinghua University / Renmin University of China

Summary

LEFT (Little-Endian Fine-Tuning): reverse digit order so carries are local. Addition/subtraction drop CoT; multiplication keeps a reversed step-by-step. Llama2-13B, 5K examples/op on digits 5–12: overall acc 94.4 vs Scratchpad-Detailed 83.3 using 3.04M vs 11.0M train tokens (+11.1 pp overall; multiply 88.5 vs 52.8). Add/sub use ~1/20 the tokens of the detailed scratchpad baseline at similar accuracy. Code/data https://anonymous.4open.science/r/RAIT-9FB7/.

Keywords

left · little-endian · arithmetic · decoding-order · llama2 · tsinghua · scratchpad · cot

Topics

arithmetic, decoding order, LLM reasoning

Research notes

Primary: arxiv abs (cs.CL; also cs.AI). License CC BY-NC-ND 4.0 on HTML at check. Equal contrib Zhang-Li/Lin; corresponding Juanzi Li. Tsinghua CST; Zhang (Xiaokang) and Jing Zhang at Renmin University of China. Correspondence {zlnn23,linny20}@mails.tsinghua.edu.cn, lijuanzi@tsinghua.edu.cn. Code/data on abs is anonymous.4open.science/r/RAIT-9FB7/ (no public GitHub named on abs). HF has no paper page (API 404). Discord posted HTML v1. Training set is synthetically generated 15K/3K arithmetic items, not a substantial hosted corpus, so no datasets_local row.