LatentMoE

NVIDIA today launches Nemotron 3 Ultra, a 550B-parameter open model built on the same architecture as Nemotron 3 Super, optimized for long-running AI agents. It employs LatentMoE to quadruple the number of experts at the same inference cost, introduces multi-token prediction to boost single-user inference speed, and is released under the Linux Foundation’s Open MDW license to enable enterprise deployment.

Kimi K3通过线性注意力、多教师蒸馏和开源MoE架构实现2.8T参数扩展，显著提升模型效率与可扩展性。

K3模型通过Stable LatentMoE和SiTU-GLU改进MoE稳定性，结合Per-Head Muon优化Attention机制，实现效果与效率的平衡。