📄 中文摘要
广义点积注意力（GDPA）是一种标准点积注意力（SDPA）的变体，通过替换软最大化操作来提升性能。GDPA 旨在解决在 GPU 训练内核中遇到的实际挑战，尤其是在处理大规模数据时。该方法通过引入新的内核设计，优化了计算效率和内存使用，显著提高了模型训练的速度和准确性。GDPA 的灵活性使其能够适应不同的应用场景，展现出在多种任务中的优越表现。实验结果表明，GDPA 在多个基准测试中均优于传统的点积注意力机制，展示了其在深度学习领域的广泛应用潜力。
广义点积注意力：应对 GPU 训练内核中的现实挑战
📄 English Summary
Generalized Dot-Product Attention: Tackling Real-World Challenges in GPU Training Kernels
Generalized Dot-Product Attention (GDPA) is a variant of Standard Dot-Product Attention (SDPA) that enhances performance by replacing the softmax operation. GDPA addresses real-world challenges encountered in GPU training kernels, particularly when handling large-scale data. By introducing a new kernel design, it optimizes computational efficiency and memory usage, significantly improving the speed and accuracy of model training. The flexibility of GDPA allows it to adapt to various application scenarios, demonstrating superior performance across multiple tasks. Experimental results indicate that GDPA outperforms traditional dot-product attention mechanisms in several benchmark tests, showcasing its broad applicability in the field of deep learning.