📄 中文摘要
通过在TorchAO中使用MXFP8 MoE训练原语，Llama4 Scout的训练速度实现了超过30.2%的提升，同时收敛性与bfloat16相当。这一成果是在GB200集群上进行的，显示出MXFP8技术在模型训练中的潜力，达到了理论速度的约81%。该研究为大规模模型训练提供了新的思路，尤其是在资源受限的环境中，优化训练效率具有重要意义。
MXFP8 MoE训练：在GB200集群上使用TorchAO和TorchTitan实现Llama4 Scout相较于BF16的1.3倍训练加速
📄 English Summary
MXFP8 Training for MoEs: 1.3x training speedup vs BF16 for Llama4 Scout on GB200 cluster using TorchAO and TorchTitan
A recent demonstration showcased a training speedup of over 30.2% for Llama4 Scout using MXFP8 MoE training primitives in TorchAO, achieving equivalent convergence to bfloat16. This was conducted on the GB200 cluster, highlighting the potential of MXFP8 technology in model training, reaching approximately 81% of the theoretical speedup. This advancement offers new insights for large-scale model training, particularly in resource-constrained environments, emphasizing the importance of optimizing training efficiency.