Why Does the Official Muon Include an Extra max(1, ⋅) Compared to the MuP Version?
科学空间1705 字 (约 7 分钟)
92
The official Muon optimizer adds a max(1,⋅) truncation to stabilize updates during early training when inputs are isotropic, but the MuP scaling factor aligns better with steepest descent theory in later stages as features become anisotropic. Practitioners should prefer the MuP version or use a dynamic decay schedule transitioning from KellerJordan to MuP.
入选理由：KellerJordan版Muon的max(1,⋅)源于din>dout且输入各向同性时的RMS近似推导。
FeaturedArticle#Muon Optimizer#MuP#Deep Learning Optimization#Feature Scaling#LLM Training中文