律师八卡V100服务器将DeepSeek从2-bit提升至近NVFP4精度

一位律师用八块二手 Nvidia V100 GPU 搭建了私人 AI 服务器,如今带来了更新:把约 80% 的工作负载从 2-bit 提升到 4-bit NVFP4 精度,并加入 1TB 内存和一块 RTX 3090。该服务器运行 DeepSeek V4.1 Flash,并配有一套法律 harness——即他围绕模型封装的工作流与检查机制——用于自己的法律业务。代价是什么?性能:代码输出从每秒 160–215 个 Token 降到 150–180 个 Token,不过新配置遭遇的崩溃问题已经消失。崩溃的元凶最初被归咎于 PCIe,但很可能是电源(PSU)故障。
这位律师原本只是 Reddit 上一个不起眼的硬件极客律师,而他对最初那篇突发报道的回应让他的身份浮出水面。他是南卡罗来纳州哥伦比亚市的一名出庭律师。ProprietaryLegalAI 在我们的第一篇报道下评论了三次,并附上代码仓库、网站和 LinkedIn 链接,由此确认他是 Nick Mermiges,PLI Labs 和 proprietarylegal.ai 的创始人。他的另一家公司 PLI 是一家技术咨询公司,而非律所,如今已将私有化、本地部署的 LLM 服务列入其业务范围。
软件部分他是借助 AI 完成的,具体是 Claude Code——由它编写内核(kernel),并在他的指导下跑基准测试。Opus 5.5 和 Fable 5.2 模型负责优化、自定义内核和重新量化;他表示,自己对 Fable 构建自定义 Volta 内核的能力感到惊讶,也惊讶于它能重新量化他的法律 harness 所触发的专家(experts)。大约从 2 月到 7 月,他围绕 Claude 搭建了一套可用的法律 harness,而他尝试的本地模型版本都失败了;但随后 DeepSeek V4.1 Flash 问世,他成功将其移植过来,尽管他说它不如前沿模型。
他重申“文件从不离开办公室”,并将系统的内存从 256GB 升级到 1TB,还添置了一块较新的 GPU,尽管 V100 已进入“借来的时间”——R580 是最后一个支持它们的 Nvidia 驱动分支。正如上次所列,该项目的意图表明他希望这套系统能分担他更多的工作负载,因此这次升级加上精度调整是一次合理的升级。他此前提出的让系统同时处理多个请求的目标也已实现。
我们此前报道的初版“Volta sPLIt”方案如今已变成“Volta Max uPLIft”,依然保留着那套巧妙的字母设计——他公司的首字母缩写是 PLI,即 Proprietary Legal Intelligence。他如今声称达到“接近 NVFP4 的精度”,本周早些时候他还在我们的论坛上把 NVFP4 描述为“实质上就是全精度”。NVFP4 是 Nvidia 的 4 位浮点格式,Nvidia 也发布了自己的 DeepSeek V4.1 Flash 的 NVFP4 版本;该项目的 README 称,这一版本在每权重 4.25 位下与该模型官方权重的精度相当。
DeepSeek V4.1 Flash 由“专家”构成,这些专家是模型中为每个词完成大部分工作的部分。该模型此前在这一精度下装不下,因为 40 层中每层有 384 个路由专家,总计 15,360 个。据他估算,按官方精度计算,这约为 269 GiB,而八块 32GB 的 V100 在加载任何其他内容之前总共只有 256 GiB。在 2 位版本中,权重为 248.8GB;新版本的 Hugging Face 包约为 556GB。
变通做法是让显卡上的每个专家都保持 2-bit,再把使用最频繁的专家替换上来,直到显卡装不下为止;在这个案例中,最终约有 38.7%,也就是 5,949 个专家,恰好以 NVFP4 精度存放。“使用最频繁”在这里指的是对四类文本混合统计得出的计数,四类文本权重相同,其中一类是他自己的法律类测试框架流量。
回报在于这些专家承担了大部分工作。在每次只处理一个请求的测试中,80% 到 86% 的解码路由——即模型生成内容时对专家的调用——落在 NVFP4 专家上,通用文本最低,源代码最高。对他所运行的双请求配置,预测值为 78% 到 85%。其余 15% 到 22% 的调用仍会落到 2-bit 专家上。
以 NVIDIA 自己版本的该模型作为参照来衡量,他取得了不错的结果。相同下一 Token 比例(即某个构建选出的下一 Token 与参照相同的频率)从 92.2% 升至 95.1%。KL 散度(衡量某个构建的概率分布与参照的偏离程度)从 0.196 nats 降至 0.079 nats。在衡量发生变化的单选题答案时,在一项标准多项选择知识测试的 513 道题中,答案不同的题目从 79 道降到仅 8 道。
这衡量的不是准确率——三种情况下的准确率大致相同——而是保真度。这些测试是在该构建的首个版本上运行的,并未在他现在运行的新双请求配置上进行。
这些改动带来的结果:保真度更高,但性能水平更低。代码输出慢了 3% 到 30%,具体取决于提示长度,法律写作慢了 9% 到 23%。在构建之前,他希望在法律和散文上保持约每秒 80 个 Token;基准测试结果为法律 86 到 106,散文 78 到 86。好处是,同时运行两个请求可将总输出提高到每秒 159.0 个 Token,而单个请求为 118.9。在每个提示长度下,首个 Token 也到达得更早,在 256K Token 时为 108.4 秒对 118.7 秒。
不过,每个请求都需要一个“槽位”,这是有代价的。每个槽位都在卡上保留自己的状态,这为更少的 NVFP4 专家留出空间:一个、两个和三个槽位分别为 6,215、5,949、5,663。这意味着每增加一个槽位,可能会将另外一到两个点的调用推回 2-bit,从而可能降低输出质量。
提示长度
等待首个 Token,2-bit / 接近 NVFP4
代码(Token/s),2-bit / 接近 NVFP4
法律写作(Token/s),2-bit / 接近 NVFP4
1.2 / 1.0 秒
215.0 / 150.1
117.6 / 105.5
17.2 / 16.0 seconds
200.5 / 165.1
116.5 / 93.4
59.2 / 54.3 seconds
182.8 / 151.4
110.9 / 87.3
118.7 / 108.4 seconds
160.0 / 154.6
103.1 / 93.5
构建者的实测数据,11 月 5 日(2-bit)与 11 月 12 日(接近 NVFP4);贪心解码,关闭思考,一次一个请求。
本周升级到更大内存,采用八条 128GB 模块,频率 2,933 MT/s;原先的八条 32GB 模块因 ECC 错误只能运行在 DDR4-2666。内存升级后,主机报告出现两个 NUMA 节点,而两次启动都在板间首次传输时挂起,因此他在软件中重新调整了板卡顺序。GPU 仍与此前一样被限制在 1,522 MHz。
至于电源,他采用了常见的排查方式,逐一追查 PCIe 插槽、板卡和线缆以缩小原因范围,最终得出结论:“大概是其中一个电源坏了。”他还加装了更多风扇,但温度毫无变化,这只能说明气流并不是问题所在。他更换了供电,之后九次长对话全部读取正常。另外,别评判他的线缆理线水平;任何“建议都是欢迎的,但会被无视”——在这种情况下,也只能这么评论了。
他已经有两台 DGX Spark 以 1M 上下文运行同一模型,用于智能体工作,并可同时跑三到四个会话,而他并不打算就此止步。现在的问题归结为扩展。另有八块 V100 和一台配备 512GB 3,200 MT/s ECC 内存的第二台服务器蓄势待发。第二台机器将运行另一个高精度 DeepSeek,用作额外的智能体或并行执行任务。他还计划把 DeepSeek 的小型草稿模型从 V100 迁到 RTX 3090 上,不过目前尚未实施。
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Lawyer’s eight-V100 AI server moves DeepSeek from 2-bit to near-NVFP4 precision)
A lawyer with a private AI server built on eight used Nvidia V100 GPUs returned with an update, moving about 80% of its work up from 2-bit to 4-bit NVFP4 precision and bringing in 1TB of RAM and an RTX 3090. The server uses DeepSeek V4.1 Flash with a legal harness, the workflows and checks he wraps around the model, for his law practice. The cost? Performance, with code output dropping from 160–215 to 150–180 tokens per second, although the crashes that hit the new build are gone. The culprit behind the crashes, originally blamed on PCIe, was likely a bad PSU.
Lawyer's V100 homelab, update 2: the 8x V100 box now runs DeepSeek at near-NVFP4 precision with a 524K context, a 3090 moved in, RAM went to 1 TB, and I blamed PCIe for what was probably one bad PSUfrom r/homelab
Once a random hardware nerd lawyer on Reddit, the lawyer's responses to the original breaking story illuminated his identity. He’s a trial lawyer in Columbia, South Carolina. ProprietaryLegalAI commented on our first story three times, linking to a code repository, site, and LinkedIn that identify him as Nick Mermiges, founder of PLI Labs and proprietarylegal.ai. His other firm, PLI, is a technology consultancy, not a law firm, and now lists private, on-prem LLM serving as a service.
He built the software side with AI, specifically Claude Code, which wrote the kernels and ran the benchmarks with his direction. The Opus 5.5 and Fable 5.2 models handled the optimizations, custom kernels, and re-quantizing; he noted he was surprised at how well Fable built custom Volta kernels, and how it re-quantizes the experts his legal harness triggers. From about February to July, he built a legal harness around Claude that worked, while his attempts at a local-model version failed, but then DeepSeek V4.1 Flash arrived, and he was able to port it successfully, even if he says it’s not as good as frontier.
He reaffirms that “the documents never leave the office” and upgraded the system from 256GB to 1TB of RAM and tossed in a newer GPU, even as the V100s stay on “borrowed time” with R580, the last Nvidia driver branch to support them. The project’s intentions, as listed last time, suggested he wanted the system to be able to offload more of his workload, so this and the precision change are a reasonable upgrade. His previously stated goal of wanting the system to handle more than one request at a time has also been met.
The original “Volta sPLIt” build we covered has transformed into the “Volta Max uPLIft,” still with the clever lettering as his company’s initials are PLI — Proprietary Legal Intelligence. He now claims “near-NVFP4 precision,” having described NVFP4 on our forum earlier this week as “essentially full precision.” NVFP4 is Nvidia’s 4-bit floating-point format, and Nvidia publishes its own NVFP4 version of DeepSeek V4.1 Flash; the project’s README says that version matches the precision of the model’s official weights at 4.25 bits per weight.
DeepSeek V4.1 Flash is built from “experts,” the parts of the model that do most of the work for each word. This model didn’t fit at that precision before, as 384 routed experts, in each of 40 layers, counts to 15,360 in all. At official precision, this comes out to about 269 GiB, by his estimate, and eight 32GB V100s have 256 GiB in total before anything else is loaded. With the 2-bit build, the weights were 248.8GB; the new build’s Hugging Face package is about 556GB.
The workaround is to keep every expert on the cards at 2-bit and swap the most-used ones up until the cards are full, which in this case came out to about 38.7%, or 5,949 experts, at precisely NVFP4. “Most-used,” in this context, refers to a count from a mix of four kinds of text, which are weighted equally, one of which is his own legal harness traffic.
The payoff is that those experts do most of the work. In testing with one request at a time, between 80% and 86% of decode routes, the calls to an expert made while the model writes, land on the NVFP4 experts, lowest for general text and highest for source code. The prediction for the two-request setup he runs is 78% to 85%. The remaining 15% to 22% of calls would still go to a 2-bit expert.
Measured against Nvidia’s own version of the model as the reference, he was able to get good results. Same next token, or how often a build picks the same next token as the reference, went up to 95.1% from 92.2%. For KL divergence, which measures how far a build’s probabilities are from the reference’s, it went down to 0.079 nats from 0.196. In measuring multiple-choice answers that change, across 513 questions from a standard multiple-choice knowledge test, it went from 79 different to just 8.
This does not measure accuracy — which was about the same across all three cases — but fidelity. Those tests ran on the build’s first release, but not on the new two-request setup he now runs.
The result of the changes: higher fidelity but at a lower level of performance. Code output is 3% to 30% slower, depending on the prompt length, and legal writing is 9% to 23% slower. Before the build, he hoped to maintain about 80 tokens per second on legal and prose; the bench came in at 86 to 106 for legal and 78 to 86 for prose. The upside is that running two requests at once raises total output to 159.0 tokens per second, versus 118.9 with a sole request. The first token also arrives sooner at every prompt length, 108.4 seconds versus 118.7 seconds at 256K tokens.
Each request needs a “slot,” though, and this comes at a cost. Every slot keeps its own state on the cards, which leaves room for fewer NVFP4 experts: 6,215, 5,949, 5,663 for one, two, and three slots, respectively. This means each added slot may push another one to two points of the calls back to 2-bit, which could reduce output quality.
Prompt length | Wait for first token, 2-bit / near-NVFP4 | Code (tokens/s), 2-bit / near-NVFP4 | Legal writing (tokens/s), 2-bit / near-NVFP4 |
|---|---|---|---|
1K tokens | 1.2 / 1.0 seconds | 215.0 / 150.1 | 117.6 / 105.5 |
32K tokens | 17.2 / 16.0 seconds | 200.5 / 165.1 | 116.5 / 93.4 |
128K tokens | 59.2 / 54.3 seconds | 182.8 / 151.4 | 110.9 / 87.3 |
256K tokens | 118.7 / 108.4 seconds | 160.0 / 154.6 | 103.1 / 93.5 |
Builder’s measurements, Nov. 5 (2-bit) and Nov. 12 (near-NVFP4); greedy decoding, thinking off, one request at a time.
The move to more RAM came this week with eight 128GB modules at 2,933 MT/s, originally eight 32GB modules that had to be run at DDR4-2666 due to ECC errors. After the RAM upgrade, the host reports two NUMA nodes, and two boots hung at the first transfer between the boards, so he reordered the cards in software. The GPUs remain capped at 1,522 MHz, as before.
As for the PSU, he took the common approach of chasing PCIe slots, cards, and cables to narrow down the cause, finally concluding it was “probably one bad PSU.” He also added more fans, but with no temperature changes, this only indicated that airflow was not the problem. He changed the power delivery, and nine of nine long chats read clean afterward. Also, don’t judge his cable management skill; any “advice [there] is welcome and will be ignored,” which is the only comment to make in that situation.
He also already has two DGX Sparks running the same model at 1M context for agent work, and three or four sessions at once, and he isn’t stopping there. Now it comes down to scaling. Eight more V100s and a second server, with 512GB of 3,200 MT/s ECC RAM, are waiting in the wings. The second box would run another high-precision DeepSeek, used as an extra agent or to run tasks in parallel. The plan is also to move DeepSeek’s small draft model off the V100s and onto the RTX 3090, which hasn’t happened yet.
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 Tom's Hardware(Lawyer’s eight-V100 AI server moves DeepSeek from 2-bit to near-NVFP4 precision),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。