Liquid AI 发布 LiquidCodec,74M 参数流式音频编解码器
Liquid AI 发布了 LiquidCodec,即其 LFM3-Audio-2B 和 LFM3-Omni-2B 模型背后的音频编解码器。LiquidCodec 音频编解码器是一个约 74M 参数的流式语音 Token 化器。它把语音转换成语言模型可以预测的 Token,再把这些 Token 转换回音频。Liquid AI 有 2 个目标:易于预测的 Token,以及快速转换回语音。权重已发布在 Hugging Face 上。
LiquidCodec 是 Liquid AI 的语音编解码器,可将 24 kHz 音频压缩为 12 条 Token 流。根据 LFM3-Audio-2B 的模型卡,每个 80 ms 帧会变成 12 个离散 Token。它取代了 Kyutai 的 Mimi,后者曾被 Liquid AI 用于 LFM2-Audio-1.5B。Mimi 有 96.2M 参数。
整个流水线是编码、量化、解码。Liquid AI 在编码器和解码器中都使用了其 LFM3 架构的模块。这两部分是对称的,且编码器完全因果,适合流式处理。
量化采用分组有限标量量化(GFSQ),而非残差向量量化(RVQ)。在 RVQ 中,每个码本编码的是前一个码本遗漏的信息。GFSQ 的各组相互独立,因此全部 12 个码本都可以由 MaskGIT 风格的 transformer 并行预测。
解码器直接预测频谱系数。这避免了对可学习的转置卷积上采样的依赖,而后者是波形解码器中常见的一项开销。
训练分 2 个阶段进行。第 1 阶段使用了覆盖 107 种语言的 264,000 小时语音,采用波形损失和频谱损失,未使用对抗损失。第 2 阶段在 55,000 小时的全频带语音(24 kHz 或更高)上微调,并引入对抗判别器。Liquid AI 称这消除了第 1 阶段后残留的金属音伪影。最后一个仅针对解码器的量化感知训练(QAT)步骤面向边缘推理。
Liquid AI 在 Apple M4 Max MacBook Pro 的 1 个 CPU 线程上以 FP32 精度测量了速度。LiquidCodec 的编码速度约为实时的 361 倍,解码约为实时的 87 倍。这些均为其自行报告的数据。
实时因子(RTF)是处理时间除以音频长度,因此数值越低越好。LiquidCodec 的编码 RTF 为 0.0028,而 Mimi 为 0.0361,NanoCodec 为 0.2145,X-Codec 2 为 0.7359。解码 RTF 为 0.0115,三者分别为 0.0329、0.2543 和 0.0379。也就是说,其编码 RTF 比 Mimi 低约 13 倍,解码 RTF 低约 2.9 倍。
Liquid AI 使用了 8 个测试集,包括 LibriSpeech test-clean、Seed-TTS 的英语和普通话,以及 5 种 FLEURS 语言(德语、法语、普通话、日语、韩语)。基线为 12 个码本下的 X-Codec 2、NanoCodec 和 Mimi。
*由 Liquid AI 在其 LiquidCodec 博客中测量。Mimi 在 12 个码本下测量。
该编解码器通过 Hugging Face Transformers 加载,需设置 trust_remote_code=True。调用 encode() 可获取音频码,调用 decode() 可获取波形。根据模型卡,原生的 LiquidCodecStreamingCache 支持逐块解码。模型卡中的示例使用 CUDA 与 bfloat16。
面向边缘使用场景,Liquid AI 列出了用于 llama.cpp 的 Q4_0、Q4_K_M 和 Q8_0 GGUF 版本。GGUF 仓库随 LFM3-Audio-2B 和 LFM3-Omni-2B 一同发布。采用仅解码器 QAT 的 Q4_0 将解码器占用缩小 3.5 倍。LibriSpeech ViSQOL 仅从 4.319 微降至 4.292。
需要与我们合作推广你的 GitHub 仓库、Hugging Face 页面、产品发布、网络研讨会等?请联系我们。
《Liquid AI 发布 LiquidCodec:一款 74M 参数的流式音频编解码器》一文首发于 MarkTechPost。
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Liquid AI Releases LiquidCodec, a 74M-Parameter Streaming Audio Codec)
Liquid AI has released LiquidCodec, the audio codec behind its LFM3-Audio-2B and LFM3-Omni-2B models. The LiquidCodec audio codec is a ~74M-parameter streaming speech tokenizer. It turns speech into tokens a language model can predict, then converts those tokens back into audio. Liquid AI had 2 goals: easy-to-predict tokens and fast conversion back to speech. The weights are on Hugging Face.
TL;DR
- Size: ~74M parameters (73.6M on the model card); 24 kHz audio, 12.5 Hz frames, 12 codebooks.
- Runs on: a single CPU thread (tested on an Apple M4 Max laptop, FP32); CUDA GPUs via Transformers; Q4_0, Q4_K_M and Q8_0 GGUF builds for llama.cpp.
- Performance: best ViSQOL and lowest mel distance on all 8 test sets versus X-Codec 2, NanoCodec and Mimi.
- Best: ~361x real-time encoding on 1 CPU thread (encode RTF 0.0028 vs Mimi’s 0.0361).
What is LiquidCodec?
LiquidCodec is a Liquid AI speech codec that compresses 24 kHz audio into 12 token streams. Each 80 ms frame becomes 12 discrete tokens, per the LFM3-Audio-2B model card. It replaces Kyutai’s Mimi, which Liquid AI used in LFM2-Audio-1.5B. Mimi has 96.2M parameters.
How does LiquidCodec work?
The pipeline is encode, quantize, decode. Liquid AI uses blocks from its LFM3 architecture in both the encoder and decoder. The 2 halves are symmetric, and the encoder is fully causal, which suits streaming.
Quantization uses grouped finite scalar quantization (GFSQ) instead of residual vector quantization (RVQ). In RVQ, each codebook encodes what the previous one missed. GFSQ groups are independent, so all 12 codebooks can be predicted in parallel by MaskGIT-style transformers.
The decoder predicts spectral coefficients directly. This avoids learned transposed-convolution upsampling, a common cost in waveform decoders.
How was it trained?
Training ran in 2 stages. Stage 1 used 264,000 hours of speech across 107 languages. It applied waveform and spectral losses, without adversarial losses. Stage 2 fine-tuned on 55,000 hours of full-band speech (24 kHz or higher) with adversarial critics. Liquid AI says this removed metallic artifacts left after stage 1. A final decoder-only quantization-aware training (QAT) step targets edge inference.
How fast is LiquidCodec?
Liquid AI measured speed in FP32 on 1 CPU thread of an Apple M4 Max MacBook Pro. LiquidCodec ran at ~361x real time for encoding and ~87x for decoding. These are self-reported figures.
Real-time factor (RTF) is processing time divided by audio length, so lower is better. LiquidCodec’s encode RTF is 0.0028, against 0.0361 for Mimi, 0.2145 for NanoCodec and 0.7359 for X-Codec 2. Decode RTF is 0.0115, against 0.0329, 0.2543 and 0.0379. That is about 13x lower encode RTF and 2.9x lower decode RTF than Mimi.
How does LiquidCodec compare on quality benchmarks?
Liquid AI used 8 test sets. These were LibriSpeech test-clean, Seed-TTS English and Mandarin, and 5 FLEURS languages (German, French, Mandarin, Japanese, Korean). Baselines were X-Codec 2, NanoCodec and Mimi at 12 codebooks.
- Reconstruction: LiquidCodec had the best ViSQOL and lowest mel distance on all 8 sets. On LibriSpeech, ViSQOL was 4.319 versus 4.219 (NanoCodec), 4.119 (X-Codec 2) and 4.031 (Mimi).
- Text-to-speech: Liquid AI trained identical TTS models on each codec’s tokens. Data (101,000 hours, 6 languages), architecture, compute and batch order were fixed. LiquidCodec scored the highest UTMOSv2 on all 8 sets and the best WER/CER on 4 of 8. LibriSpeech WER was 3.15%, against 6.27% for Mimi. Results held with a Qwen3-1.7B backbone as well as LFM3-1.7B.
- Where it trails: NanoCodec scored higher PESQ (WB) on 7 of 8 sets. It also led on LibriSpeech speaker similarity (SECS 0.836 vs 0.785).
LiquidCodec vs Mimi vs NanoCodec vs X-Codec 2
| Feature | LiquidCodec | Mimi | NanoCodec 1.78 kbps | X-Codec 2 |
|---|---|---|---|---|
| Developer | Liquid AI | Kyutai | NVIDIA | HKUST Audio |
| Parameters | 73.6M | 96.2M | 62M | 0.8B |
| Sample rate | 24 kHz | 24 kHz (repo) | 22.05 kHz | 16 kHz |
| Frame rate | 12.5 Hz | 12.5 Hz | 12.5 fps | 50 tokens/s |
| Codebooks | 12 | 8 at 1.1 kbps (blog) | 13 | 1 (docs) |
| Quantizer | GFSQ | RVQ (docs) | FSQ | Single-layer FSQ (docs) |
| Bitrate | Not disclosed | 1.1 kbps | 1.78 kbps | Not disclosed |
| Causality | Causal encoder; decoder not disclosed | Fully causal (Kyutai) | Non-causal encoder, causal decoder | Not disclosed |
| License | “other” on HF; blog links LFM License | CC-BY 4.0 | NVIDIA Open Model License | CC-BY-NC 4.0 |
| Encode / decode RTF, M4 Max CPU* | 0.0028 / 0.0115 | 0.0361 / 0.0329 | 0.2145 / 0.2543 | 0.7359 / 0.0379 |
| LibriSpeech mel distance* (lower is better) | 0.120 | 0.160 | 0.122 | 0.123 |
| LibriSpeech PESQ (WB)* | 2.640 | 2.615 | 2.942 | 2.434 |
| LibriSpeech TTS UTMOSv2* | 3.31 | 2.89 | 3.19 | 3.27 |
*Measured by Liquid AI in its LiquidCodec blog. Mimi measured at 12 codebooks.
How can developers deploy LiquidCodec?
The codec loads through Hugging Face Transformers with trust_remote_code=True. Call encode() to get audio codes and decode() to get a waveform. A native LiquidCodecStreamingCache supports chunk-by-chunk decoding, per the model card. The card’s examples use CUDA with bfloat16.
For edge use, Liquid AI lists Q4_0, Q4_K_M and Q8_0 GGUF builds for llama.cpp. GGUF repos ship with LFM3-Audio-2B and LFM3-Omni-2B. Q4_0 with decoder-only QAT shrinks the decoder footprint by 3.5x. LibriSpeech ViSQOL dips only from 4.319 to 4.292.
Key Takeaways
- ~74M-parameter streaming codec, 12.5 Hz, 12 codebooks, 24 kHz.
- ~361x real-time encode, ~87x decode on 1 CPU thread.
- Best ViSQOL, mel distance and TTS UTMOSv2 on all 8 test sets.
- NanoCodec still wins PESQ (WB) on 7 of 8 sets.
- Q4_0 decoder is 3.5x smaller with minimal quality loss.
Check out the technical blog, the model weights on Hugging Face, and LFM3-Audio-2B and LFM3-Omni-2B. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
The post Liquid AI Releases LiquidCodec, a 74M-Parameter Streaming Audio Codec appeared first on MarkTechPost.
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 MarkTechPost(Liquid AI Releases LiquidCodec, a 74M-Parameter Streaming Audio Codec),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。