Together GPU 集群自动修复:从故障到健康节点

推出 Polymorphic Inference,首先从 Together Compiler 开始 →
推出 Together Post-Training:模型更好,运行更快,成本更低 →
推出 Together Link:在你已在使用的 harness 中运行开放模型 →
开源 AI 技术栈:模型、推理、路由器、harness 与工具 →
GLM-5.3 与 GLM 5.3 Flash 在 DeepSWE 上的对比 →
⚡ 按需 B200 现已在 Together GPU 集群上线 →
以 API 形式提供的高性能推理
面向批处理工作负载的推理
基于 Token 的容量,附带 SLA
专属模型推理
自定义硬件上的推理
专属容器推理
自定义模型的推理
在您的运行框架中运行开放模型
随时间不断优化的推理
探索热门开源模型
大规模可靠 GPU 集群
前沿规模的自定义基础设施
为 AI 构建开发环境
安全存储模型权重与数据
定义你自己的 RL 与 SFT 循环
启动托管式 SFT 与 DPO 任务
微调顶级开源模型
面向生产级 AI 的系统研究
我们所有的研究出版物
Together AI 技术文档
我们的最新新闻与博客文章
我们的开源演示应用
实用实施指南
用开放模型构建更好的应用
在你的 harness 中运行开放模型
浏览我们的活动日历
开源 ROI 计算器
以 API 形式提供的高性能推理
面向批处理工作负载的推理
基于 Token 的容量,附带 SLA 保障
专用模型推理
在自定义硬件上运行推理
专用容器推理
面向自定义模型的推理
在自有框架中运行开放模型
随时间不断优化的推理
探索顶级开源模型
大规模可靠 GPU 集群
前沿规模的自定义基础设施
为 AI 构建开发环境
安全存储模型权重与数据
定义你自己的 RL 与 SFT 循环
启动托管式 SFT 和 DPO 任务
微调顶级开源模型
面向生产级 AI 的系统研究
我们的全部研究论文
Together AI 技术文档
我们的最新新闻与博客文章
我们的开源演示应用
实用实施指南
用开放模型构建更好的成果
在你的 harness 中运行开放模型
浏览我们的活动日历
开源 ROI 计算器
Together GPU 集群的自动修复:从 GPU 故障到节点恢复健康
健康检查发现问题,平台选择最轻量的修复方式,而由你决定哪些操作无需人工介入即可执行,精细到单个 Xid 错误码。
在集群规模下,GPU 故障是常态。一次 Xid 错误、一次双比特 ECC 错误、一块 GPU 从 PCIe 总线上掉线,或是一条 InfiniBand 链路抖动,都可能让分布式训练任务停滞,或使一个推理副本宕机。察觉到出了问题通常是最容易的部分。真正困难的是从故障发生到节点重新恢复健康之间的所有环节。
今年早些时候,我们宣布推出全自动节点修复(node remediation)功能。我们的 GPU 集群客户以及我们内部的推理平台一直在使用这项能力,以提升工作负载韧性并最大限度减少停机时间。客户非常喜欢这项能力,所以我想花点时间拆解这套系统,讲清楚它是如何工作的,以及我们为什么这样设计。
我们首先实现了"被动健康"检查系统,它会持续监控每一个节点——无论是在你的 VM 内部,还是在其底层的宿主机上。当某项检查发现故障时,自动修复会推荐最轻量的修复操作,并向你展示该建议背后的支撑证据。你可以事先逐个故障地决定:某项修复是等待批准后执行,还是自行运行。你还可以决定自动修复在运行中的工作负载完成前最多等待多久。你自己的检测器只需在节点上写入一条注解,就能通过同一条流水线发起修复。
可用性。目前已正式可用(GA)的包括:被动健康检查、修复建议、自动修复策略(含按故障和按节点的授权)、等待空闲控制、升级阶梯、安全护栏,以及 API 和控制台修复。预览版:基于节点标签的修复合约,它让你可以接入自有的检测系统,例如 NVSentinel 或自定义控制器。我们持续新增检测信号和修复动作,以确保最大覆盖范围和最短恢复时间。
为什么 NotReady 还不够
大多数节点自动修复都依赖单一信号:节点进入了 NotReady 状态。无论节点是真的坏掉了,还是只是短暂失联,这个信号看起来都一样。当某个存储或网络事件同时波及大量节点时,每个受影响的节点都会跌落至 NotReady。我们的系统内置了熔断机制,以防止节点被大规模替换。NotReady 也不会说明节点为何故障,因此没有依据在重启和更换新机器之间做出选择。
所以自动修复遵循一条规则:没有失败的检查作为依据,就不进行任何节点修复;而每一次修复都会附带其支撑证据。
自动修复流水线:来自 VM 内部和宿主机的信号,汇入你集群内的 Detect、Decide 和 Repair 环节;迁移走的宿主机则交由 Together 基础设施上的 Re-validate 环节处理。
该流水线包含四个阶段:检测故障、决定是否以及何时采取行动、修复节点、重新验证机器。前三个阶段在你的集群上运行。第四个阶段在我们这一侧运行,运行在你的节点刚刚腾出的那台物理宿主机上。来自你自己的检测工具的节点标签请求从 decide 阶段进入,并与我们自己的请求得到完全相同的处理。
1. 从 hypervisor 两侧进行检测
Together GPU 集群中的 GPU 以直通方式交给你的 VM,以实现裸金属性能。这意味着 GPU 故障可以从 VM 内部看到,而宿主机与 fabric 的故障则从宿主机侧可见。因此,检测同时在 hypervisor 两侧运行,并汇聚到同一处。
在你的 VM 内部,各类 exporter 监控:
内核日志,用于发现 Xid 错误以及 GPU 从 PCIe 总线上掉线
DCGM,用于发现 ECC 双比特错误、行重映射失败、PCIe 重放率和热降频
Node Problem Detector,用于发现只读文件系统、XFS 关闭、内核死锁以及 kubelet 或 containerd 反复重启
在 Slurm 上,exporter 会上报被标记为 DOWN 或 DRAIN 的节点。
在主机上,我们监控 InfiniBand rail 状态和链路抖动(link flapping),并将其归属到拥有这些 rail 的 VM。我们还监控主机可达性,因此一台宕掉的主机会在你的节点上显示为一个问题,而不是一片沉默。
每条告警都会附带一份稳定的摘要、一段通俗语言的描述,以及一份机器可读的证据负载。对于 Xid,就是每个错误码及其 NVIDIA 助记名;对于热降频,就是峰值温度和降频次数。控制平面原样透传这些字符串,因此控制台、API(以 linked_alerts 形式)和你的通知始终说同一件事。
我们把“已观测”与“可操作”区分开。每个与修复相关的告警都会被记录下来,并将你的节点标记为 Degraded。只有带有动作映射的信号才会创建修复。例如磁盘压力会出现,但不会触发重启,因为重启解决不了它。没有映射的告警会被计数并记录,绝不会被悄悄丢弃,而是转交我们的支持团队调查。
2. 策略:我们决定做什么,你决定是否做以及何时做
平台负责的部分。每个信号都映射到能修复它的最轻量动作,因此你无需知道哪个 Xid 代表什么:
是否修复由你决定。在 Repairs(修复)标签页中,自动修复策略默认为「修复前批准」:每条建议都会等待人工处理,由人来接受或覆盖该操作。你可以把默认值切换为「完全自动」,但这并非一刀切的开关。「修复操作」清单按所触发的修复对故障分组,你选择其中哪些可以放心无人值守运行。在 Xid 分组内,你可以按错误码逐一设置授权。一种常见配置是自动批准 Xid 119 和 95 的重启,而对迁移保持门控。如果你希望让一小部分故障自动运行、其余都等待你审核,Xid 清单在「修复前批准」模式下同样生效。
何时修复同样由你决定。修复会排空(drain)节点,因此「等待空闲」会一直挂起修复,直到节点真正空闲。在 Kubernetes 上,这意味着节点上没有任何正在运行的 pod 申请 GPU。在 Slurm 上,这意味着调度器报告该节点没有任何资源分配。最大等待时长用于限定挂起时间,默认 2 小时,上限 24 小时。「不中断正在运行的任务」则完全取消这一限制。请把等待时长设置为至少等于你的检查点间隔。如果等待时限到期,修复将继续进行,并记录该超时。
每次修复都会记录是谁发起的(自动修复为 Together Health Checker)、是谁批准的(具体用户名或「自动批准」)、由哪条规则决定,以及时间。
3. 自带检测器
运行大规模 GPU 集群的团队已经有自己信任的检测手段:DCGM 仪表盘、NVSentinel、自研扫描器、任务级检查。每一次修复,无论来源为何,都会在我们的控制平面中变成同一条记录。这让容量检查、审计和安全防护机制有了唯一的归口。
接入方式有三种:
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Auto repair for Together GPU Clusters: from GPU fault to healthy node)
Introducing Polymorphic Inference, starting with Together Compiler →
Introducing Together Post-Training: Better models, faster runs, lower costs →
Introducing Together Link: Run open models inside the harness you already use →
The Open Source AI Stack: models, inference, routers, harnesses & tools →
GLM-5.3 vs. GLM 5.3 Flash on DeepSWE →
⚡ On-demand B200s now available on Together GPU Clusters →
High-performance inference as APIs
Inference for batch workloads
Token-based capacity with SLAs
Dedicated Model Inference
Inference on custom hardware
Dedicated Container Inference
Inference for custom models
Run open models in your harness
Inference that gets better over time
Explore Top Open-Source Models
Reliable GPU clusters at scale
Custom infrastructure at frontier scale
Build development environments for AI
Store model weights & data securely
Define your own RL and SFT loop
Launch managed SFT and DPO jobs
Fine-tune top open-source models
Systems research for production AI
All our research publications
Technical docs for Together AI
Our latest news & blog posts
Our open-source demo apps
Practical implementation guides
Build better with open models
Run open models in your harness
Explore our events calendar
Open-source ROI calculator
High-performance inference as APIs
Inference for batch workloads
Token-based capacity with SLAs
Dedicated Model Inference
Inference on custom hardware
Dedicated Container Inference
Inference for custom models
Run open models in your harness
Inference that gets better over time
Explore Top Open-Source Models
Reliable GPU clusters at scale
Custom infrastructure at frontier scale
Build development environments for AI
Store model weights & data securely
Define your own RL and SFT loop
Launch managed SFT and DPO jobs
Fine-tune top open-source models
Systems research for production AI
All our research publications
Technical docs for Together AI
Our latest news & blog posts
Our open-source demo apps
Practical implementation guides
Build better with open models
Run open models in your harness
Explore our events calendar
Open-source ROI calculator
Auto repair for Together GPU Clusters: from GPU fault to healthy node
Health checks find the fault, the platform picks the lightest fix, and you decide what runs without a person in the loop, down to individual Xid codes.
At cluster scale, GPU faults are a reality. An Xid error, a double-bit ECC error, a GPU falling off the PCIe bus or a flapping InfiniBand link can stall a distributed training job or take an inference replica down. Spotting that something is wrong is usually the easy part. The hard part is everything between the fault and a healthy node again
Earlier this year, we announced the launch of our fully Automatic node remediation features. Our GPU clusters customers and our internal Inference platform has been using this capability to help drive workload resiliency and minimize down time. Our customers love this capability, and we wanted to take some time break the system down and explain how it works and why we designed it this way.
We first implemented “Passive health” checks system, that watches every node continuously, both inside your VM and on the host underneath it. When a check finds a fault, auto repair recommends the lightest action to fix and shows you supporting evidence behind the recommendation. You decide up front, fault by fault, whether a repair waits for approval or runs on its own. You also decide how long auto repair waits for running workload to finish. Your own detectors can open repairs through the same pipeline by writing a single annotation on a node.
Availability
Generally available today: passive health checks, repair recommendations, the auto-remediation policy (including per-fault and per-node consent), wait-for-idle controls, the escalation ladder, the safety rails, and API and console repair.
Preview: node-label based remediation contracts, which let you bring your own detection systems such as NVSentinel or custom controllers.
We are constantly adding new detection signals and remediation actions to ensure maximum coverage and minimum time to recovery.
Why NotReady is not enough
Most node auto-repair keys on a single signal: the node went NotReady. That signal looks the same whether a node is broken or just briefly unreachable. When a storage or network event touches many nodes at once, every impacted node drops to NotReady. Our system has a circuit breaker built in to prevent mass replacement of nodes. NotReady also does not say why the node failed, so there's no basis for choosing between a reboot and a new machine.
So auto repair follows one rule: no node repair occurs without a failed health check behind it, and every repair carries its supporting evidence.
Auto repair pipeline: signals from inside the VM and from the host feed Detect, Decide, and Repair inside your cluster; migrated hosts go to Re-validate on Together infrastructure.
The pipeline has four stages: detect a fault, decide whether and when to act, repair the node, and re-validate the machine. The first three stages run against your cluster. The fourth runs on our side, on the physical host your node just vacated. Node-label requests from your own detection tooling enter at the decide stage and get exactly the same treatment as ours.
1. Detection from both sides of the hypervisor
GPUs in Together GPU Clusters are passed straight through to your VM for bare-metal performance. That means GPU faults are visible from inside the VM, while host and fabric faults are visible from the host. Detection therefore runs on both sides of the hypervisor and lands in one place.
Inside your VM, exporters watch:
the kernel log, for Xid errors and GPUs falling off the PCIe bus
DCGM, for ECC double-bit errors, row-remap failures, PCIe replay rate, and thermal throttling
Node Problem Detector, for read-only filesystems, XFS shutdowns, kernel deadlocks, and repeated kubelet or containerd restarts
On Slurm, an exporter reports nodes marked DOWN or DRAIN.
On the host, we watch InfiniBand rail state and link flapping, attributed to the VM that owns those rails. We also watch host reachability, so a dead host shows up as a problem on your node rather than as silence.
Every alert ships a stable summary, a plain-language description, and a machine-readable evidence payload. For an Xid, that's each code and its NVIDIA mnemonic. For thermal throttling, it's the peak temperature and throttle count. The control plane passes those strings through untouched, so the console, the API (as linked_alerts), and your notifications always say the same thing.
We keep observed separate from actionable. Every repair-relevant alert is recorded and marks your node Degraded. Only signals with an action mapping create a repair. Disk pressure, for example, shows up but doesn't trigger a reboot, because a reboot won't fix it. An alert with no mapping is counted and logged, never silently dropped, and routed to our support team for investigation.
2. Policy: we decide what, you decide whether and when
What is owned by the platform. Each signal maps to the lightest action that fixes it, so you never need to know which Xid means what:
Whether is yours to decide. In the Repairs tab, the Auto-remediation policy defaults to Approve before repair: every recommendation waits for a person, who can accept or override the action. You can switch the default to Fully automatic, but it isn't a blanket switch. The Repair actions checklist groups faults by the repair they trigger, and you select the ones you trust to run unattended. Inside the Xid group, you can set consent per code. A common setup auto-approves reboots for Xid 119 and 95 and keeps migrations gated. The Xid checklist also works under Approve before repair, if you want a short list of faults to run automatically while everything else waits for your review.
When is yours as well. A repair drains the node, so Wait for idle holds it until the node is actually idle. On Kubernetes, that means no running pod on the node is requesting a GPU. On Slurm, it means the scheduler reports no allocation on the node. Maximum wait bounds the hold, with a default of 2 hours and a cap of 24. Do not interrupt running jobs removes the bound entirely. Set the wait to at least your checkpoint interval. If the bound expires, the repair proceeds and the timeout is recorded.
Every repair records who requested it (Together Health Checker for automatic repairs), who approved it (a named user or Auto-Approved), which rule decided, and when.
3. Bring your own detector
Teams running large GPU fleets already have detection they trust: DCGM dashboards, NVSentinel, in-house scanners, job-level checks. Every repair, whatever its source, becomes the same record in our control plane. That gives capacity checks, audit, and safety rails exactly one home.
There are three ways in:
API. Create a repair against an instance with a mode, an optional client-supplied ID for idempotency, and optionally the health check run or alert you're citing as evidence. You can also list, get, approve, reject, and cancel repairs.
Console. The ⋮ menu on any node opens the Repair dialog with the detected issues and the recommended action.
Node labels (Preview). Any detector with RBAC permission to write node metadata can open a repair by adding one annotation and one label to the node. The annotation carries the detector name, a stable request ID, the repair mode, when the fault was detected, and a human-readable reason.
Our cluster operator mirrors the node label up to the control plane. The repair is recorded with the detector as an external requester, so it's always attributable and can never impersonate a user. Detector name plus request ID is the idempotency key across the cluster, finished repairs included, so a detector that re-asserts the same fault on every scan can't reboot a node twice. Authorization is your cluster's own RBAC, with no API key and no onboarding ticket.
Annotation requests can ask for Reboot, Quick reprovision, or Migrate. Remove stays human-only. Like a console click, these requests skip the approval queue, but they still pass the same evidence gate, capacity check, and budgets as everything else. A per-cluster cap means a buggy detector causes limited damage rather than a catastrophe.
4. Repair actions and the escalation ladder
Every repair follows the same lifecycle: cordon, wait (per your policy), drain, act, rejoin. The console shows the node moving through Cordoning → Draining → Repairing / Migrating → Joining → Running.
The actions, from lightest to heaviest:
Reboot. Restarts the VM in place. Local /scratch and /tmp data survive.
Quick reprovision. Rebuilds the VM from a clean image on a healthy host. Local data is lost.
Migrate to new host. Rebuilds the VM on different hardware and sends the old host to re-validation.
Remove. Pulls the node for RMA. Always human-initiated.
The same fix doesn't loop. If a fault comes back on the same machine after the same action has already run twice, the next recommendation moves one rung up. A third round of Xid reboots becomes a reprovision, and a third reprovision becomes a migration. The ladder is tracked per machine, cluster, alert, and Xid code. Attempts older than a week don't count, and neither do cancelled or self-resolved recommendations.
Two tries per rung, then one rung up. Migration needs critical-alert evidence.
Migrations are evidence-gated. Migrate is the one action that consumes real capacity. It requires either a critical alert currently firing on the node or a critical alert on the same machine within the last ten days. A request that meets neither condition runs as a Quick reprovision instead.
Host power cycle (Preview). Some faults, Xid 79 among them, need the GPU to lose power, which a VM reboot can't do even though the hardware may be fine. Rather than consuming a new host, we can power-cycle the one you're on. No BMC credentials exist anywhere in the system. A boot-ID check prevents a retry from cycling a host that has already come back.
Clusters that need a guaranteed replacement window can reserve hot spare capacity, so migrations draw from that reserve rather than general availability. Ask your account team for details.
5. Rails that assume the alert storm will happen
A bad driver push, a storage outage, or a network partition can make dozens of nodes look sick at once. Node-level repair can't fix a cluster-level cause, so the rails are layered and independent:
Persistence. A signal must hold for a per-signal duration before it becomes a recommendation.
Cooldown. After any repair, a node gets roughly 30 minutes before a new recommendation can be generated.
Concurrency budget. The budget is sized from cluster node count, and the operator runs one repair per node pool at a time.
Circuit breaker. When one cluster generates a burst of correlated automatic repairs, the breaker trips. Many nodes failing at once points to a cluster-level cause, and repairing nodes one by one will not fix it.
One target per repair. The schema enforces it, so a bug can't fan out across nodes.
Auto-resolve. A recommendation whose alert clears before anyone acts disappears on its own.
The control plane is replicated, and every write is idempotent and checked against the state it expects. Two replicas racing on the same node produce one repair, not two.
6. The machine re-validates itself
When a node migrates off a machine, or is removed from one, that machine shouldn't go straight back into the pool, and it shouldn't sit in a queue waiting for someone to read diagnostics either. As soon as the old VM is gone, the machine moves into validation automatically. A test VM runs the same GPU, NCCL, InfiniBand, and storage checks that every brand-new machine passes before joining the fleet. No person touches the machine between your node leaving and the verdict landing.
Only an explicit failure files a hardware ticket.
There are three verdicts:
Passed. The machine returns to the pool. No ticket is filed and no one is paged, so a transient fault stays transient.
Failed. A hardware ticket is filed automatically with per-test results and logs attached. The machine stays out until it's fixed and passes again.
Aborted. The checks couldn't run, for example because the host was reallocated or the test VM couldn't be placed. The machine is restored to its prior state, and nothing is condemned.
Only an explicit failure produces a ticket.
Built to extend past one accelerator
Repair decisions key on alerts, nodes, and machines, not on the GPU vendor. The GPU type matters for placement and capacity, but recommendations, policy, escalation, budgets, triggers, and re-validation don't special-case it. The vendor-specific parts are detection (the Xid and DCGM exporters) and the diagnostic battery. Bringing up a new accelerator means adding those pieces, and everything downstream carries over.
Open your cluster in the console and go to the Repairs tab. Pending recommendations and history are there, with the Auto-remediation policy at the top. Each remediation in the API carries its linked_alerts, so your own tooling sees the same evidence the console does.
NVSwitch coverage. SXid faults, coalesced per node, with readable descriptions.
Status back on the node. Repair progress written back as node metadata, so external detectors can close the loop where they opened it.
One health surface. Click from a failed check to the repair it caused and back, plus a per-node view of health, events, and repair history.
Tell us what fails on your clusters that we don't catch yet. That list drives what we build next.
Auto repair for Together GPU Clusters: from GPU fault to healthy node
Introducing preemptible compute: the same compute, half the price
Together AI and Y Combinator partner to launch the first dedicated GPU cluster for the YC community
New in Together GPU Clusters: Reliability and control for production GPU clusters
Capacity without conflict: A guide to multi-tenant GPU cluster design for AI-native teams
New in Together GPU Clusters: Autoscaling, observability, and self-healing
Inside multi-node training: How to scale model training across GPU clusters
Announcing General Availability of Together Instant Clusters, offering ready to use, self-service NVIDIA GPUs
Bringing 100,000 GPUs to Europe
Together AI Powers Pioneers at GTC: NVIDIA Blackwell GPUs, Instant GPU Clusters, and A Full-Stack for AI Innovation
Introducing Together Instant GPU Clusters Accelerated by NVIDIA GPUs, with Self-Service Provisioning in Minutes
Supercharging NVIDIA H200 and H100 GPU Cluster Performance With Together Kernel Collection
A practitioner's guide to testing and running large GPU clusters for training generative AI models
Growing to 20 exaflops, Together GPU Clusters help startups and enterprises accelerate generative AI development
Optimizing Training Workloads for GPU Clusters
Together Compiler: Continual Compilation for AI inference
From Static to Polymorphic Inference: A new era for production open-source inference systems
Auto repair for Together GPU Clusters: from GPU fault to healthy node
Together Post-Training: frontier-grade model shaping, the way that fits you
Meta Muse Spark Open is live on Together AI
Expanding our enterprise inference capacity with IBM Cloud and NVIDIA
Together Link: frontier-quality open models in the harness you already use
How to train your own Jev for $17
Canary rollouts: upgrade models in production without downtime
How a global fintech scaled coding agent traffic with Dedicated Model Inference
Migrating from closed to open source models, Together
Together AI expands fine-tuning service with more models, live metrics, and finer controls
原文为英文,正文由自动抓取并保留原文。
原文发布于 Together AI Blog(Auto repair for Together GPU Clusters: from GPU fault to healthy node),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。