11-19 更新 实时
首页/ 算力/ 正文

Zyphra ZYFA:AMD MI355X 的 FP8 注意力提速 1.76 倍

算力 来源:MarkTechPost 2026-11-07 01:28 0 阅读 机器翻译

Zyphra 发布了 ZYFA(Zyphra FlashAttention on AMD),这是一组面向 AMD Instinct MI355X GPU 的 FP8 和 BF16 预填充注意力 kernel。ZYFA 是 Zyphra 用于长上下文预填充的 FP8 注意力引擎,代码以 Apache 2.0 协议开源。在真实模型激活值上,其 FP8 kernel 比 AMD 默认 kernel 库 AITER 快 1.16-1.76 倍。在头维度为 128 时,ZYFA 的 FP8 误差比 AITER 低 62-78%。

ZYFA 以技术论文加代码的形式由 Zyphra 发布。它为 Zyphra Optimization Engine(ZOE)的注意力引擎提供支持,而 ZOE 驱动着 Zyphra Cloud。BF16 kernel 用 HIP 编写,FP8 kernel 则是 Python 生成器输出的手工排布汇编代码。

FP8 矩阵指令的运行速度是 BF16 峰值的 2 倍。但 FP8 只能加速 QKᵀ 和 PV 这两个矩阵乘积。两者之间的 softmax 在向量单元上运行,两种精度下速度相同。

对于 32 个 query 对 128 个 key 的情况,FP8 矩阵运算需要 1,024 个周期。精确的 FP8 softmax 需要 896 个向量周期。因此 kernel 需要近乎完美的重叠才能让矩阵单元保持忙碌。Zyphra 的线程轨迹显示,AITER 的矩阵流水线在 31.7% 的周期里处于空闲状态。

MI355X 不会在硬件层面跟踪矩阵指令的完成情况,因此必须靠指令顺序来形成这种重叠。

ZYFA 采用了 3 项技术:

e4m3 的位模式(bit pattern)按整数读取时,值每翻一倍就会增加 8。所以 2^x 的编码大致是 8x 加一个常数。lean P 用 1 次乘加和 1 次整数转换写出每个 FP8 概率。这使每块的 softmax 向量周期从 896 降到 384。

每个 SIMD 由 2 个 wave 共享。一个负责发出矩阵指令,另一个计算概率,然后互换角色。借助 lean P,矩阵流水线在稳态周期中有 86% 的时间处于忙碌状态,而使用 exact P 时只有 60%。

每个 64-key 概率块在舍入前会按 2 的幂进行缩放。CDNA4 的缩放矩阵指令可零开销地消除该因子。ZYFA 还会除以舍入后概率之和,而该和每个块需要额外 1 条矩阵指令来计算。

译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Zyphra ZYFA: FP8 FlashAttention for AMD MI355X, Up to 1.76x Faster)

Zyphra has released ZYFA (Zyphra FlashAttention on AMD), a set of FP8 and BF16 prefill attention kernels for AMD Instinct MI355X GPUs. ZYFA is Zyphra’s FP8 attention engine for long-context prefill, and the code is open source under Apache 2.0. On real model activations, its FP8 kernels run 1.16-1.76x faster than AITER, AMD’s default kernel library. At head dimension 128, ZYFA’s FP8 error is 62-78% lower than AITER’s.

TL;DR

  • Framework: It is a kernel library. Causal forward (prefill) attention in FP8 e4m3 and BF16, head dim 128 and MLA shape (192/128).
  • Runs on: AMD Instinct MI355X only (CDNA4, gfx950), ROCm 7.2.3, PyTorch 2.12. Drop-in flash_attn_func interface.
  • Performance: 1.16-1.76x AITER’s FP8 throughput on real activations; BF16 roughly ties (1.01-1.02x).
  • Best: 2,672 TFLOP/s FP8 at 32K tokens, 53% of the FP8 datasheet peak (AITER: 39%).
  • Worst: Only 1.16x on Mistral-7B activations; FP8 clock drops to 1,628 MHz vs AITER’s 2,108 MHz.

What is ZYFA?

ZYFA is shipped in the form of a technical paper along with code from Zyphra. It powers the attention engine of the Zyphra Optimization Engine (ZOE), which runs Zyphra Cloud. The BF16 kernels are written in HIP. The FP8 kernels are hand-scheduled assembly emitted by a Python generator.

Why does FP8 attention underperform on MI355X?

FP8 matrix instructions run at 2x the BF16 peak. But FP8 only speeds up the 2 matrix products, QKᵀ and PV. The softmax between them runs on the vector unit at the same speed in both precisions.

For 32 queries against 128 keys, FP8 matrix work takes 1,024 cycles. The exact FP8 softmax takes 896 vector cycles. So the kernel needs near-perfect overlap to keep matrix units busy. Zyphra’s thread traces show AITER’s matrix pipe idle for 31.7% of cycles.

The MI355X does not track matrix instruction completion in hardware, so instruction order must create that overlap.

How does ZYFA work?

ZYFA uses 3 techniques:

Lean P: one multiply-add instead of an exponential

An e4m3 bit pattern, read as an integer, grows by 8 each time the value doubles. So the code of 2^x is roughly 8x plus a constant. Lean P writes each FP8 probability with 1 multiply-add and 1 integer conversion. This cuts softmax vector cycles per block from 896 to 384.

Two-wave ping-pong schedule

2 waves share each SIMD. One issues matrix instructions while the other computes probabilities, then they swap. With lean P, the matrix pipe is busy 86% of steady-state cycles, up from 60% with exact P.

Per-block scaling and a rounded row sum

Each 64-key block of probabilities is scaled by a power of 2 before rounding. CDNA4’s scaled matrix instruction removes the factor at no cost. ZYFA also divides by the sum of the rounded probabilities, computed with 1 extra matrix instruction per block.

How fast is ZYFA?

Real-activation FP8 speedups at 16K and sustained clocks are 1.16x (Mistral-7B), 1.17x (Qwen3-8B) and 1.76x (DeepSeek-V2-Lite). On Gaussian benchmarks at 32K, the head-dim-128 FP8 kernel reaches 2,672 TFLOP/s against AITER’s 1,944. At the MLA shape it reaches 2,464 against 1,237, a 1.99x ratio.

On captured Mistral-7B and Qwen3-8B layers, ZYFA delivers 1.94 and 1.91 TFLOP/J. AITER delivers 1.60 and 1.56.

Why are real-model gains smaller than benchmark gains?

The MI355X clock depends on how many operand bits flip. AITER lets more small probabilities become zeros or subnormals. Those switch fewer bits, so its clock runs higher. On the Mistral-7B layer, ZYFA runs at 1,628 MHz while AITER runs at 2,108 MHz. ZYFA still needs fewer cycles overall. Zyphra team recommends benchmarking on real activations at sustained clocks.

How accurate is ZYFA’s FP8 attention?

Against fp64 attention, ZYFA’s FP8 error is 0.22x AITER’s on Mistral-7B and 0.38x on Qwen3-8B. It sits within a few percent of Composable Kernel and Triton. In full Mistral-7B, KL divergence from fp64 is 3.4×10⁻³ with ZYFA and 5.9×10⁻³ with AITER. ZYFA’s outputs were also bitwise identical across 7,200 launches.

Most full-model FP8 error comes from quantizing Q, K and V. A Hadamard rotation of Q and K roughly halves that input error. On Mistral-7B it adds about 0.1% to attention time.

How does ZYFA compare with other attention kernels?

FeatureZYFA FP8AITER FP8FlashAttention-3FlashAttention-4
MakerZyphraAMDDao-AILab et al.Dao-AILab et al.
Target GPUAMD MI355XAMD MI300X to MI355XNVIDIA H100/H800NVIDIA B200 (plus Hopper)
Headline throughput2,672 TFLOP/s FP81,944 TFLOP/s FP8 (Zyphra-measured)Close to 1.2 PFLOP/s FP81,613 TFLOP/s BF16
Share of peak53% FP839% FP8 (Zyphra-measured)Not disclosed (FP8)71% BF16
Softmax approachLean P, no exponentialHardware exponentialOverlap via async Tensor CoresSoftware-emulated exponential
FP8 accuracy claim62-78% lower error than AITERBaseline2.6x lower error than baseline FP8Not disclosed
ScopeCausal prefill forward onlyPrefill, decode, MLA, pagedForward and backward, FP8 forwardForward and backward
LicenseApache 2.0MITBSD-3-ClauseBSD-3-Clause

Cross-vendor numbers come from different inputs and clocks. Zyphra did not measure NVIDIA GPUs.

Key Takeaways

  • FP8 attention on MI355X is softmax-bound, not matrix-bound.
  • Lean P swaps the exponential for 1 multiply-add.
  • Real-model gains are 1.16-1.76x, smaller than benchmarks show.
  • FP8 error drops 62-78% vs AITER at head dim 128.
  • Prefill only: no decode, no paged KV, MI355X only.


Check out the paper, the Zyphra blog post and the GitHub repo. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

The post Zyphra ZYFA: FP8 FlashAttention for AMD MI355X, Up to 1.76x Faster appeared first on MarkTechPost.

正文保留英文原文,标题与摘要由机器翻译。

本文由采集器自动同步

原文发布于 MarkTechPost(Zyphra ZYFA: FP8 FlashAttention for AMD MI355X, Up to 1.76x Faster),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗