PowLU: An Activation Function for Stable Pre-Training of LLMs

Summary

PowLU(x)=x·x^{m/(√x+1)}·sigmoid(x) for x>0 (same as SwiGLU for x≤0); m=3 default. Proved continuous, differentiable, monotone for 0<m<10, and ~linear (not quadratic) as x→+∞. Scaling-law MoE 26M–368M activated matches SwiGLU loss. Ling 7.9B/600B tokens and 124B/800B stay competitive vs SwiGLU and SwiGLU-Clip on knowledge/reasoning/math/code while cutting outliers and FP8 loss spikes after ~76k steps. No official code on abs.

Keywords

powlu · swiglu · activation · training-stability · ling · ant-group · moe · fp8

Topics

activation functions, LLM pretraining stability, SwiGLU

Research notes

• Primary: arxiv abs (cs.CL/LG). Ling Team, Ant Group; correspondence jiangpeijie.jpj / lingyao.zzq@antgroup.com. No official code on abs. HF has no paper page (API 404). Discord posted tweet + abs. Pretrains Ling MoE on unnamed corpus; no standalone public dataset, so no datasets_local row. ArXiv license widget not visible in converted abs HTML, so paper license left blank.