律师用八块二手 V100 GPU 搭建私有 AI 服务器

一名律师在 Reddit 上发帖,介绍自己用八块 2018 年前后的二手 NVIDIA Tesla V100 数据中心 GPU 搭建的私人 AI 服务器,这样案件文件就永远不必离开办公室。这套系统使用两块可插 4 张卡的 SXM2 NVLink 基板、四个 PEX 交换芯片,以及随处可见的 SlimSAS 线缆——外加一个烧毁的电源接口——搭建者如今能运行开放权重的 DeepSeek V4.1 Flash 模型,所有专家都驻留在 GPU 内存中,速度最高达每秒 215 个 Token。搭建者购入时,每块四卡基板的价格约为 2,400 美元。
这八块 Tesla V100-SXM2-32GB 卡每块配备 32GB 的 HBM2、带宽 900 GB/s,它们插在 SXM2 而非 PCIe 插槽上——SXM2 是 NVIDIA 面向数据中心 GPU 的夹层插槽——并分装在两块基板上。每块基板内部,四块卡全部通过 NVLink 互联——NVLink 是 NVIDIA 的 GPU 直连链路——每块基板合计 128GB,但两块基板之间仅通过 PCIe 通信。每块基板通过 SlimSAS 线缆、经两块 PEX(PLX)交换卡连接到主机,总共四个交换芯片,每个交换芯片通过自有的 x16 上行链路,以 PCIe 3.0 x8 的速率带动两块 GPU。
不过,文中的价格已成为过去:二手 V100 SXM2 32GB 卡目前在 eBay 上的挂牌价约为 900 美元,是帖子中每块约 400 美元的两倍多,而且挂牌信息显示价格仍在攀升。这套系统还已经配备了其他一些必要部件:一块 Gigabyte Threadripper Pro 主板,搭配与之匹配的 12 核 3945WX CPU,8 条 32GB DDR4 RDIMM,一个 NUMA 节点,以及两个电源、总功率 2,800 W,接在普通办公室的独立电路上。这是一个真实案例,说明二手数据中心硬件可以成为把大量高速 VRAM 集中在一处的廉价途径。
这台服务器目前每天用于案件文件的初筛工作:它读取一个记录文件夹,提取出事实、日期和数字,加以汇总,并对常规文书起草做第一版尝试。“任何东西不经过真正的审核都不会发出去”,总体而言这是一条不错的“人在回路”策略。它也有助于保障隐私,适用于客户文件不能发送到云端或前沿模型的案件。检索增强生成(RAG)数据库和工具调用已接入;操作者可以通过 SSH 使用类似 Discord 的界面来操控该系统,在这里是从一台 Windows PC 上操作。目前它一次只能处理一个请求。
DeepSeek V4.1 Flash 在这里被压缩到大部分为 2 比特,其权重总计仍有 248.8GB。该模型的“专家”——即为每个词承担大部分工作的部分——完全驻留在 GPU 内存中,而约 110GB 的查找表(称为 Engram 表)则留在系统的普通内存里。从内存中取数每步约耗时 0.12 毫秒,搭建者表示这几乎不影响速度数据。
模型前半部分层(0–19)在第一块 SXM2 板上运行,后半部分层(20–39)在第二块板上运行。每块板都通过各自的 NVLink 处理自己的高负载流量,因此每步只有一次交接需要跨越两块板之间较慢的 PCIe 链路。在此前的一篇帖子中,搭建者发现,把每一层都拆分到全部八张卡上会让速度“断崖式下跌”,原因正是那一跳 PCIe。
对于这个规模的模型来说,性能好得惊人。代码生成最快,可达每秒最多 215 个 Token,因为 DeepSeek 内置的一个小型草稿模型会向前猜测五个 Token,而代码很容易猜中。法律文书写作——也就是这台服务器的用途所在——速度可达每秒最多 118 个 Token。
长文档需要更长时间才能开始响应:该模型读取提示词的速度约为每秒 2,200 个 Token,因此在 128K Token 时,首个 Token 要等大约一分钟后才会出现。它最多可接受 524K Token 的上下文。在该帖子发布的三天前,同样的模型在四块这类卡上每秒仅能处理 36–38 个 Token。性能跃升来自第二块板卡,在理想情况下,这大概能使性能翻倍,而搭建者将剩余提升归功于自定义内核和预填充工作。完整细分见表格。
提示词长度
等待首个 Token
代码(Token/秒)
法律写作(Token/秒)
17.2 秒
59.2 秒
118.7 秒
组装者的测量数据,11 月 5 日;贪心解码,输出 1,024 个 Token,两次运行的平均值。
这并不只是把所有东西接好、通上电那么简单。主机两次自行重启,都是在连接某块 GPU 板卡的链路上出现致命 PCIe 错误;两次都发生在功耗上限调高、GPU 频率不加限制的情况下。在这种状态下,这些卡的功耗达到 192–232 W,超过了 180 W 的上限。最有可能的猜测——尽管“尚未证实”——是某块板卡在负载尖峰期间停止响应。
在组装者限制频率并移除一块电源接口有烧灼痕迹的板卡后,重启就停止了;“各位,检查一下你们的接口。”现在每块卡的上限设为 150 W,即其 300 W 额定值的一半,图形时钟上限为 1522 MHz。根据测试,把上限设为 200 W 并没有带来明显的性能提升。
系统内存也出现了无法纠正的 ECC 错误,组装者把内存从 DDR4-3200 降到 DDR4-2666 后,问题就消失了。最近一次检查时,有一块 GPU 是以 x4 而非 x8 连接的,组装者打算重新插拔它;一位评论者在自己 V100 上遇到的同类问题,最终追溯到一根坏的线缆。
这些折腾是值得的,因为这位律师现有的 LLM 工作流中估计有 50–60% 现在可以在这台机器上运行,而不必调用前沿模型。帮助弥合差距的是第二台 DeepSeek 服务器,跑在两台 DGX Spark 上,两套系统彼此交接任务。这套配置看起来确实省钱,至少与 API 定价相比是这样,尽管组装者说它其实并不更快。不过,一旦实现“自包含”——这也是目标所在——它就能保护客户数据。
这位搭建者还补充说,V100 主机可能还会接入一条 240 V 电路。除了供电问题之外,该系统在 Nvidia R580 驱动分支上也已时日无多,因为这是最后一个支持 Volta 的驱动分支。目前,这位律师承认这主要还是个爱好,但未来可能成为一项商业想法。下一步是一次处理两三个请求,这一功能已在测试系统上搭建并检查过,但尚未在服务器上运行。
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Lawyer builds private AI server with eight used Nvidia V100 GPUs)
A lawyer posted on Reddit about their private AI server built from eight used 2018-era Nvidia Tesla V100 data center GPUs so case files never have to leave the office. With two 4-card SXM2 NVLink boards, four PEX switches, and SlimSAS cables everywhere — plus one burnt power connector — the builder now runs the open DeepSeek V4.1 Flash model with every expert in GPU memory, at up to 215 tokens per second. The four-GPU boards cost about $2,400 apiece when the builder bought them.
Lawyer's V100 homelab, update: two 4-card SXM2 NVLink boards, four PEX switches, SlimSAS everywhere, one burnt power connector, and a 250 GB model running entirely in VRAM at 200+ tok/sfrom r/homelab
The eight Tesla V100-SXM2-32GB cards, each with 32GB of HBM2 at 900 GB/s, plug into SXM2, Nvidia’s mezzanine socket for data center GPUs, rather than PCIe slots, and are split across two baseboards. Inside each board, all four cards are NVLinked — NVLink is Nvidia’s direct GPU-to-GPU link — for 128GB per board, but the two boards only communicate over PCIe. Each board connects to the host through two PEX (PLX) switch cards over SlimSAS cables, four switches in all, with each switch handling two GPUs at PCIe 3.0 x8 speeds over its own x16 uplink.
The given prices are already in the past, though: used V100 SXM2 32GB cards are now listed on eBay for about $900, more than double the roughly $400 apiece from the post, and listings suggest prices are still climbing. The system also already has some other necessary components: a Gigabyte Threadripper Pro motherboard with a 12-core 3945WX CPU to match, 8x 32GB DDR4 RDIMM, one NUMA node, and two PSUs, 2,800 W in total, are on separate regular office circuits. This is a real-world case for used data center hardware as a cheap route to getting lots of fast VRAM together in one place.
The server is currently in daily use for first-pass work on case files: it reads a folder of records, pulls out facts, dates, and numbers, summarizes them, and takes a first swing at routine drafting. “Nothing goes out the door without a real review,” which is a good human-in-the-loop policy in general. It also helps ensure privacy for cases where client files can’t be sent to a cloud or frontier model. RAG databases and tool calls are wired in; the operator can drive the system through a Discord-style interface over SSH, in this case from a Windows PC. For now, it handles only one request at a time.
DeepSeek V4.1 Flash is compressed to mostly 2 bits here, and its weights still total 248.8GB. The model’s “experts,” the parts that do most of the work for each word, live entirely in GPU memory, while about 110GB of lookup tables, called Engram tables, stay in the system’s regular RAM. Fetching from RAM costs about 0.12 milliseconds per step, which the builder says barely affects the speed numbers.
The first half of the model’s layers (0–19) runs on the first SXM2 board and the second half (20–39) on the second. Each board handles its heavy traffic over its own NVLink, so only one hand-off per step has to cross the slower PCIe link between the boards. In an earlier post, the builder found that splitting every layer across all eight cards made speeds “fall off a cliff” because of that PCIe hop.
Performance is surprisingly good for a model this size. Code comes out fastest, at up to 215 tokens per second, because a small draft model built into DeepSeek guesses five tokens ahead, and code is easy to guess. Legal writing, which is what the server is for, runs at up to 118 tokens per second.
Long documents take longer to get going: the model reads prompts at about 2,200 tokens per second, so at 128K tokens, the first token arrives after about a minute. It can accept up to 524K tokens of context. Three days before the post, the same model on four of these cards managed just 36–38 tokens per second. The jump came with the second board, which could presumably double performance under ideal circumstances, and the builder credits custom kernels and prefill work for the rest. See the table for the full breakdown.
Prompt length | Wait for first token | Code (tokens/s) | Legal writing (tokens/s) |
|---|---|---|---|
1K tokens | 1.2 seconds | 215 | 118 |
32K tokens | 17.2 seconds | 200 | 117 |
128K tokens | 59.2 seconds | 183 | 111 |
256K tokens | 118.7 seconds | 160 | 103 |
Builder’s measurements, Nov. 5; greedy decoding, 1,024 tokens out, mean of two runs.
It wasn’t as simple as hooking everything up and turning on the power. The host reset itself twice with a fatal PCIe error on the link to one GPU board, both times with a higher power limit and uncapped GPU clocks. In this state, the cards drew 192–232 W, past the 180 W limit. The best guess, although “not proven,” is that a board stops responding during load spikes.
The resets stopped after the builder capped the clocks and removed one board whose power connector had a burn mark; “Check your connectors, people.” Each card is now capped at 150 W, half of its 300 W rating, with the graphics clock capped at 1522 MHz. A 200 W cap brought no appreciable performance gains, according to tests.
The system memory hit uncorrected ECC errors, too, which went away after the builder dropped the RAM from DDR4-3200 to DDR4-2666. One GPU was connected at x4 instead of x8 at last check, and the builder plans to reseat it; one commenter traced the same kind of problem on their own V100 to a bad cable.
The pain was worth the payoff as an estimated 50–60% of the lawyer’s existing LLM workflows can now run on the box instead of frontier models. Helping bridge the gap is a second DeepSeek server on two DGX Sparks, where the two systems hand work off to each other. The setup does appear to save money, at least when compared to API pricing, although the builder says it’s not really faster. However, it will protect client data “once it is self contained, which is the goal.”
The builder added that the V100 box might also get a 240 V circuit. Beyond power, the system is also on borrowed time with the Nvidia R580 driver branch, which is the last to support Volta. For now, the lawyer admits that it’s mostly a hobby, but in the future it may be a business idea. The next step is to handle two or three requests at once, something built and checked on a test system but not yet on the server.
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 Tom's Hardware(Lawyer builds private AI server with eight used Nvidia V100 GPUs),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。