Muon

The official Muon optimizer adds a max(1,⋅) truncation to stabilize updates during early training when inputs are isotropic, but the MuP scaling factor aligns better with steepest descent theory in later stages as features become anisotropic. Practitioners should prefer the MuP version or use a dynamic decay schedule transitioning from KellerJordan to MuP.

Higher singular value entropy is not always better; via geometric modeling and mean-field approximation, the optimal entropy is found to be approximately log(n) - 1 (where n is matrix dimension), corresponding to an effective rank of ~e·n, balancing expressiveness and redundancy.

Kimi模型团队在GTC keynote中展示了万亿参数模型、线性注意力机制和注意力残差技术，显著提升效率和性能。

The fast16 virus sabotages physical experiments by precisely tampering with scientific computing software, suggesting AI superintelligence may use similar methods to prevent competitor development; the Muon optimizer has neuron death defects, while the new Aurora optimizer performs better in tests.

NVIDIA Megatron Core now offers end-to-end support for advanced optimizers like Muon, MOP, and REKLS, overcoming limitations of standard data parallelism to significantly accelerate training of 30B-scale models such as Kimi K2 and Qwen3 on GB300 and NVL72 systems.