Muon quietly "starved" 25% of its neurons: Aurora's repair resulted in a hundredfold increase in data efficiency.

According to Beating's monitoring, Tilde Research discovered a hidden flaw in Muon, the optimizer used by leading models such as DeepSeek V4, Kimi K2.5, and GLM-5: it permanently kills over a quarter of the neurons in the MLP layer early in training. The team designed and open-sourced an alternative optimizer, Aurora, to address this issue. A 1.1B model, using only about 100B tokens, outperformed Qwen3-1.7B, trained with 36T tokens, on language understanding benchmarks like HellaSwag and Winogrande. The problem lies in a mathematical characteristic of Muon's handling of the MLP weight matrix. Early in training, some neurons happen to receive weak gradient signals. Traditional optimizers like AdamW normalize parameters one by one, naturally smoothing out these differences; however, Muon's orthogonalization step propagates the weak signals unchanged. Weak neurons continue to receive weak updates, becoming increasingly silent, creating a vicious cycle of "the strong getting stronger." By training step 500, over a quarter of the neurons had essentially died, wasting parameter capacity. Previous improvements, NorMuon, mitigated this by forcibly flattening the update magnitude per row, but at the cost of breaking the orthogonality of the update matrix (orthogonalization ensures the most efficient update per step, a core advantage of Muon), resulting in reduced optimization accuracy. Aurora sets "uniform updates" and "orthogonality" as joint constraints, satisfying both through alternating iterations: ensuring each neuron gets a fair learning opportunity without sacrificing update accuracy. Untuned Aurora incurs only 6% more computational overhead than Muon and can be directly replaced. In the modded-nanoGPT optimization benchmark, Aurora broke the current best record with 3175 steps. Aurora's advantages also amplify with increasing MLP width; the higher the expansion coefficient, the more significant the improvement. The code and the 1.1B pre-trained model are both open-source.