Buoyant 发布 Kinference 推理控制平面
Linkerd 背后的公司 Buoyant 发布了 Kinference,这是一个用于在 Kubernetes 上构建推理平台的开源项目。Kinference 是面向 Kubernetes 和 vLLM 的推理控制平面。它位于一组 vLLM worker 之前,决定每个请求的去向。它在一层中实现路由策略、Token 速率限制、KV 缓存感知的负载均衡和按租户的指标。对于需要向众多内部客户提供多种开放权重模型服务的开发团队来说,它瞄准的是模型服务器之上的一层。该项目在 GitHub 上采用 Apache 2.0 许可。当前版本为 v0.1.0-alpha.7。
Kinference 是一个策略驱动的控制平面,把 Kubernetes 加上 vLLM 变成多租户推理平台。Buoyant 的 FAQ 说得很简单:vLLM 是数据平面,Kinference 是控制平面。
该系统有 2 个部分:流量网关和推理处理器(inference handler)。GitHub README 指出,处理器对外暴露 gRPC 接口。它接收请求,返回选定的端点以及可能被修改的请求体。它与 Buoyant Gateway 这类代理配合使用。
Kinference 会监听 vLLM 的 KV 缓存事件和引擎指标。它目前使用 NVIDIA 的 dynamo-kv-router crate 来处理缓存事件。配置位于 5 个 CRD 中:GatewayBackend、InferencePolicy、InferencePool、InferenceProvider 和 InferenceWorkerSet。
推理策略是声明式规则,规定每个请求如何被路由或被拒绝。规则可以基于租户、请求的模型、GPU 池状态和语义标签进行匹配。项目网站上的示例策略涵盖 5 种模式:
大多数规则以 minimize: PredictedTTFT 结尾。Kinference 利用缓存重叠和队列长度,预测每个候选端点的首 Token 时间(time-to-first-token)。策略存放在 git 中,可以用 kinfctl CLI 离线校验。
Buoyant 发布了一项针对原生 Kubernetes 负载均衡的基准测试。每个测试组使用 3 个 vLLM 0.31.0 Pod,每个 Pod 运行在 1 块 L40S GPU 上。模型为 Qwen3.8-27B-FP8。负载由 GuideLLM 0.8.1 生成。基线是 ClusterIP Service,由 kube-proxy 为每条连接随机选择一个 Pod。
空载延迟结果(TTFT 中位数):
端到端延迟中位数在共享前缀场景下降 45%,在多轮对话场景下降 32%。在对照流量上,p95 TTFT 仍从 21.46 s 降至 12.33 s。Buoyant 将这一结果归因于队列感知负载均衡。
容量测试衡量的是有效吞吐:即满足 TTFT SLO 的每分钟请求数。共享前缀场景的峰值有效吞吐为 103.0 对 14.0 请求/分钟,约为 7.4 倍。多轮对话场景为 82.7 对 23.8,约为 3.5 倍。Service 基线在所有测试并发下均未达到 SLO。
这些数字均为自报数据。基线是随机路由,而非 llm-d 或 Dynamo 之类的其他缓存感知路由方案。Buoyant 也指出,其 SLO(8 s 和 4 s)适用于离线智能体,而非聊天场景。
Kinference 的差异化角度在于治理:策略、别名、供应商故障回退和租户核算。llm-d 和 Dynamo 则在纯服务性能、分离式架构和引擎覆盖范围上处于领先。
按照入门指南,安装共需 5 个步骤。团队先安装 Gateway API v1.5.1 CRD,再安装 Buoyant gateway Helm chart,然后安装 Kinference chart。Kinference 提供 Prometheus 指标和 Grafana 仪表盘。指标包括 TTFT、ITL、端到端延迟以及已缓存与未缓存的提示 Token,大多按租户维度统计。
想与我们合作推广你的 GitHub 仓库、Hugging Face 页面、产品发布或线上研讨会等?欢迎联系我们
“Buoyant 发布 Kinference:面向 Kubernetes 和 vLLM 的开源推理控制平面”一文首先出现在 MarkTechPost 上。
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM)
Buoyant, the company behind Linkerd, has released Kinference, an open source project for building inference platforms on Kubernetes. Kinference is an inference control plane for Kubernetes and vLLM. It sits in front of a fleet of vLLM workers and decides where each request goes. It applies routing policies, token rate limits, KV-cache-aware load balancing and per-tenant metrics in one layer. For dev teams serving many open-weight models to many internal clients, it targets the layer above the model server. The project is Apache 2.0 licensed on GitHub. The current release is v0.1.0-alpha.7.
TL;DR
- Size: v0.1.0-alpha.7, written in Rust, Apache 2.0, configured through 5 Kubernetes CRDs.
- Runs on: any GPU-enabled Kubernetes cluster running vLLM. Buoyant’s benchmark used 3 L40S GPUs per pool.
- Performance: large gains on prefix-heavy traffic versus plain Kubernetes Service routing (self-reported).
- Best: median TTFT fell from 6.88 s to 0.31 s (95%) on a shared-prefix workload.
- Worst: no median TTFT gain on prompts without shared prefixes (6.92 s vs 6.83 s).
What is Kinference?
Kinference is a policy-driven control plane that turns Kubernetes plus vLLM into a multi-tenant inference platform. Buoyant’s FAQ puts it simply: vLLM is the data plane and Kinference is the control plane.
How does Kinference work?
The system has 2 parts: a traffic gateway and an inference handler. The GitHub README states that the handler exposes a gRPC interface. It takes a request and returns the chosen endpoint plus a possibly mutated body. It pairs with a proxy such as the Buoyant Gateway.
Kinference watches vLLM KV-cache events and engine metrics. It currently uses NVIDIA’s dynamo-kv-router crate to process cache events. Configuration lives in 5 CRDs: GatewayBackend, InferencePolicy, InferencePool, InferenceProvider and InferenceWorkerSet.
What are inference policies?
Inference policies are declarative rules for how each request is routed or rejected. Rules can match on tenant, requested model, GPU pool state and semantic tags. Example policies on the project site cover 5 patterns:
- Simple to cheaper: route low-complexity prompts to a smaller model, unless cache overlap makes the large model faster.
- Burst to cloud: skip workers with over 2 queued prompts, then fall back to providers like Bedrock.
- PII stays local: remove external providers when a PII tag fires, and fail closed with
onEmpty: Reject. - Batch on idle GPUs: send batch requests only to workers under 70% KV-cache usage.
- Model aliasing: publish one stable name, such as
standard-model, backed by several models.
Most rules end with minimize: PredictedTTFT. Kinference predicts time-to-first-token for each candidate endpoint, using cache overlap and queue length. Policies live in git and can be validated offline with the kinfctl CLI.
How fast is Kinference in benchmarks?
Buoyant published a benchmark against plain Kubernetes load balancing. Each arm used 3 vLLM 0.31.0 pods, each on 1 L40S GPU. The model was Qwen3.8-27B-FP8. Load came from GuideLLM 0.8.1. The baseline was a ClusterIP Service, where kube-proxy picks a random pod per connection.
Unloaded latency results (median TTFT):
| Workload | Kubernetes Service | Kinference | Change |
|---|---|---|---|
| Control (no shared prefix) | 6.92 s | 6.83 s | No significant difference |
| Shared-prefix | 6.88 s | 0.31 s | about -95% |
| Multiturn | 1.70 s | 0.59 s | about -65% |
Median end-to-end latency fell 45% on shared-prefix and 32% on multiturn. On control traffic, p95 TTFT still dropped from 21.46 s to 12.33 s. Buoyant attributes this to queue-aware balancing.
The capacity test measured goodput: requests per minute meeting a TTFT SLO. Peak goodput was 103.0 vs 14.0 req/min on shared-prefix, about 7.4x. On multiturn it was 82.7 vs 23.8, about 3.5x. The Service baseline never met the SLO at any tested concurrency.
These numbers are self-reported. The baseline is random routing, not another cache-aware router like llm-d or Dynamo. Buoyant also notes its SLOs (8 s and 4 s) suit offline agents, not chat.
How does Kinference compare with llm-d, Dynamo and Production Stack?
| Feature | Kinference | llm-d | NVIDIA Dynamo | vLLM Production Stack |
|---|---|---|---|---|
| Backer | Buoyant | CNCF sandbox (Red Hat, Google Cloud, IBM, CoreWeave, NVIDIA) | NVIDIA | vLLM project |
| License | Apache 2.0 | Apache 2.0 | Apache 2.0 | Apache 2.0 |
| Maturity | v0.1.0-alpha.7 | v0.7 (May 2026) | 1.x (1.5.1 containers) | Reference stack |
| Engines | vLLM (SGLang, TensorRT-LLM planned) | vLLM, SGLang | vLLM, SGLang, TensorRT-LLM | vLLM |
| KV-cache-aware routing | Yes, predicted TTFT | Yes, prefix-cache and predicted latency | Yes, KV-aware router | Session-based; prefix-aware listed as WIP |
| Declarative routing policy CRD | Yes (InferencePolicy) | Not documented in README | Not documented in README | Not documented in README |
| Burst to external providers | Yes (InferenceProvider) | Not documented in README | Not documented in README | Not documented in README |
| Disaggregated prefill/decode | Not disclosed | Yes | Yes | On roadmap |
| Multi-tenant controls | Per-tenant metrics; rate limits planned | Flow control for multi-tenant serving | Gateway-level rate limiting via GAIE | Not documented in README |
| Main result | -95% median TTFT vs kube-proxy (3x L40S) | 3x throughput, 2x faster TTFT vs round-robin (4x MI300X) | 2x faster TTFT with KV-aware routing (Baseten) | Not disclosed |
Kinference’s distinct angle is governance: policies, aliases, provider fallback and tenant accounting. llm-d and Dynamo lead on raw serving performance, disaggregation and engine coverage.
Deployment and availability
Installation takes 5 steps per the getting started guide. Teams install Gateway API v1.5.1 CRDs, the Buoyant gateway Helm chart, then the Kinference chart. Kinference ships Prometheus metrics and Grafana dashboards. Metrics include TTFT, ITL, end-to-end latency and cached versus uncached prompt tokens, mostly per tenant.
Key Takeaways
- Kinference is an Apache 2.0 control plane for vLLM on Kubernetes, from Linkerd maker Buoyant.
- Declarative InferencePolicy rules handle model choice, provider bursting, PII routing and batch admission.
- Self-reported median TTFT fell 95% on shared-prefix traffic versus kube-proxy.
- Peak goodput rose about 7.4x on shared-prefix and 3.5x on multiturn.
- It is alpha: rate limiting, classifiers and Anthropic API support are not yet shipped.
Check out the GitHub repo, docs, launch post and benchmark report. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
The post Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM appeared first on MarkTechPost.
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 MarkTechPost(Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。