第二代 Optane 内存条,纯 CPU 跑 2.8 万亿参数 AI 模型

Reddit 上一位家庭实验室(homelab)搭建者用已停产的二代 Optane PMem 200 内存,跑起了 Moonshot AI 的 2.8 万亿参数 Kimi K3 模型,推理使用的是 40 核第三代 Xeon Scalable CPU,而非 GPU。这套配置运行温度很高,这位搭建者的建议是「把主动 DIMM 散热视为必需」,他给内存条加装了两个 12W 鼓风机风扇来控制温度。该项目是今年早些时候另一位用户第一代 Optane 搭建方案的后续,即便采用 1-bit 量化,速度也只有缓慢的 0.85 tokens/秒。
在我们看来,Kimi K3 绝不是为家用而设计的,Moonshot AI 推荐使用数据中心推理引擎来运行它,即便是社区版最小的量化版本也至少需要 610GB 内存。而改用二手 Optane,是家用环境下达到这一容量的少数途径之一。DRAM 价格飞涨更是雪上加霜,这位搭建者表示,Optane 内存的成本只有同等容量方案的「一小部分」。
Intel 于 2020 年推出面向第三代 Xeon Scalable CPU 的 Optane 持久内存 200 系列(PMem 200),容量有 128GB、256GB 和 512GB 三档。在双路 Xeon 平台上,它的有效 MT/s 速率高于第一个项目所用的第一代(PMem 100),平均可提供最多高出 32% 的内存带宽。Optane 可运行在内存模式(Memory Mode)或 App Direct 模式下,这两套搭建所用的平台均支持这两种模式。
Memory Mode 让 Optane 充当系统内存、由 DRAM 担任其缓存,这是第一次搭建时采用的方式;而 App Direct 则把 Optane 暴露为一个独立的持久化区域,由软件直接映射,这是第二次使用的方式。据第一次搭建的创作者 u/APFrisco 说,这样可以“更好地控制模型张量存放在哪里”,他计划尝试这一做法。
用 Optane 替代 DRAM 确实会损失性能,再加上更高的 3-bit 量化,速度进一步降至 0.75 Token/秒。有评论者指出这非常慢,按大约 1 Token/秒估算,一个 5 万 Token 的任务要花 13 个小时;但也有人指出,过夜批处理对速度并没有那么在意。这两个速度都可以与第一次搭建的大约 4 Token/秒作对比,不过那次除了 Memory Mode 之外,搭建者还用一块 RTX 3060 承担了模型的一部分,用的是 2-bit 量化,以及更小、更早的 1 万亿参数 K2.5 模型。
最新的这位搭建者还必须应对散热问题:在持续推理时,Optane DIMM 会变得“极其烫”,搭建者称这种状况可能危及相邻的 DRAM。这种发热与 Intel 自己的规格相符——厂商要求以 2.4 至 2.7 米/秒的强制气流为这些模组散热,而这正是数据中心里才有的条件。通常你不会让这种级别的气流直接吹向内存条。
在预期的运行条件下,每根 Optane 模块的功耗为 12W 或 15W,与容量无关。如果这套配置使用四根内存条,就意味着紧挨着系统内存的位置要散发高达 60W 的热量。这类模块在 83 摄氏度时开始降频,在 85 摄氏度时触发热跳闸关机;相比之下,典型的 DDR4 服务器内存的额定温度为 85 摄氏度,扩展温度范围最高可达 95 摄氏度。要让系统保持正常运行,必须排出大量热量。
给出的性能数据是调优前的——也就是说,尚未对线程数或 NUMA 亲和性进行调优——因此还有改进空间。也有评论者问到,是否可以针对最常用的专家采用带 DRAM 缓存的 Memory Mode。搭建者计划加入 GPU,并将模型的一部分加载到 DRAM 中,作为与 Optane 并行的更快存储层级,“后续还会有更多基准测试”。
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Homelabber uses second-gen Optane memory sticks to run a 2.8-trillion-parameter AI model on CPU alone)
A homelab builder on Reddit has run Moonshot AI’s 2.8-trillion-parameter Kimi K3 AI model on discontinued second-gen Optane PMem 200 memory, using a 40-core 3rd-gen Xeon Scalable CPU and not a GPU for inference. The build runs hot, and the builder’s advice is that you “treat active DIMM cooling as mandatory,” with dual 12W blower fans aimed at the modules to get temperatures under control. The project, a follow-up to a separate user’s first-gen Optane build earlier this year, runs at a sluggish 0.85 tokens per second even with 1-bit quantization.
Second-gen Optane PMem server running Kimi K3 (2.8T params) on CPU only, plus a critical thermal warning that could save your RAMfrom r/homelab
Kimi K3 is absolutely not built for home use, in our estimation, with Moonshot AI recommending data center inference engines to run it, and even the smallest community quantization needs at least 610GB of RAM. Using secondhand Optane, instead, is one of the few ways to reach that capacity at home. It doesn’t help that DRAM prices have skyrocketed, and the builder says the Optane memory costs “a fraction” of the equivalent.
Intel launched its Optane Persistent Memory 200 Series (PMem 200) in 2020 for its 3rd-gen Xeon Scalable CPUs in 128GB, 256GB, and 512GB capacities. On two-socket Xeon platforms, it runs at a higher effective MT/s rate than the first generation (PMem 100) that was used on the first project, and it can deliver an average of up to 32% more memory bandwidth. Optane can run in either Memory Mode or App Direct Mode; both builds had platforms allowing for each.
Memory Mode has the Optane act as system memory with DRAM as its cache, the method used in the first build, while App Direct exposes Optane as a separate, persistent region that the software maps directly, the mode used in the second. This can give “more control over where the model’s tensors live,” according to the first build’s creator, u/APFrisco, who planned to try it.
Computer build using Intel Optane Persistent Memory - Can run 1 trillion parameter model at over 4 tokens/secfrom r/LocalLLaMA
Performance does take a hit by going with Optane over DRAM, and with the higher 3-bit quantization, the speed went down even more to 0.75 tokens per second. One commenter points out that this is very slow, with a 50k-token task estimated to take 13 hours at about 1 token per second, but another notes that overnight batches don’t care as much about speed. Both speeds compare to the first build’s roughly 4 tokens per second, although in that case the builder had an RTX 3060 handling part of the model in addition to Memory Mode, a 2-bit quantization, and the smaller, earlier 1-trillion-parameter K2.5 model.
The latest builder also had to deal with heat output, with the Optane DIMMs getting “extremely hot” with sustained inference, creating a situation the builder says can endanger neighboring DRAM. That heat is consistent with Intel’s own specs, as the manufacturer rates the modules with forced airflow of 2.4 to 2.7 meters per second for cooling, which is what you expect to find in data centers. You might not normally have this level of airflow pointed directly at your memory sticks.
The expected operating conditions put each Optane module at 12W or 15W, regardless of capacity. If the build used four sticks, that would mean up to 60W of heat placed right next to the system memory. Throttling on these occurs at 83 degrees Celsius, with a thermal trip shutdown at 85 degrees, compared to the 85 degrees Celsius and up to 95 degrees in an extended range that typical DDR4 server memory is rated for. A lot of heat has to be removed from the system to keep things running.
The performance numbers given are pre-tuning — this means without tuning thread counts or NUMA affinity — so there is room for improvement, and one commenter also asked about using Memory Mode with DRAM caching for the most-used experts. The builder plans to add GPUs and to also load parts of the model into DRAM as a faster tier alongside the Optane, with “more benchmarks to come.”
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 Tom's Hardware(Homelabber uses second-gen Optane memory sticks to run a 2.8-trillion-parameter AI model on CPU alone),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。