腾讯 WeVisOne-VL 统一检测、OCR 与分割
腾讯微信视觉团队发布了 WeVisOne-VL,这是一款开放视觉语言模型,提供 2B 和 4B 两种规模。WeVisOne-VL 视觉语言模型在单一模型中即可完成检测、指代、计数、指向、OCR、分割和 Alpha 抠图。像素输出由一个 960 万参数的解码器生成,因此无需单独的 SAM。当查询的目标不存在时,它还会回答“None”。对开发者而言,这可以用一个模型替代检测器、OCR 引擎和分割器。
WeVisOne-VL(“WeChat Vision All-in-One”)是腾讯基于 Qwen3-VL 构建的以感知为核心的模型。它面向细粒度定位,而非开放式对话。用户可选择输出粒度:点、框、二值掩码或连续 Alpha 蒙版。
每项任务都被建模为结构化的下一 Token 预测。模型新增了坐标 Token <0> 到 <1000>,用于归一化的框和点。目标不存在时输出 None,而不是猜测一个框。
对于像素任务,模型会在每个预测框之后输出一个 <OBJ> Token。根据模型卡,其隐藏状态会成为像素解码器的查询。该解码器由 3 部分组成:
一个知识图谱驱动的数据引擎在超过 1 亿张图像上完成了标注,约 36 亿个框。SAM3、Rex-Omni 和一款自研检测器提出候选区域,由 Qwen3-VL 进行校验。
训练使用了 3 个 SFT 阶段。第 1 阶段在 9000 万个样本上对齐输出格式。第 2 阶段将输入提升至 1,960 个视觉 Token,并将每次回复的框数上限提高到 500 个。第 3 阶段训练像素解码器,随后在 1.3 亿个样本上针对抠图进行专门化训练。最后一个 GRPO 阶段使用了 17.8 万个经不确定性过滤的样本,具体见研究论文。
以下所有数字均为自报结果,采用贪心解码,每张图像最多使用 1,960 个视觉 Token。检测、指代和 OCR 使用在 IoU 0.5 至 0.95 上平均的 F1。
2B 模型在大多数任务上落后于 4B 模型。但在 P3M-500-NP 上,其框提示抠图得分仍更高,为 6.6 MAD。
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Tencent WeVisOne-VL Unifies Detection, OCR, Segmentation and Matting)
Tencent’s WeChat Vision team has released WeVisOne-VL, an open vision language model in 2B and 4B sizes. The WeVisOne-VL vision language model handles detection, referring, counting, pointing, OCR, segmentation and alpha matting in one model. Pixel outputs come from a 9.6M-parameter decoder, so no separate SAM is needed. It also answers “None” when a queried object is absent. For developers, that can replace a detector, an OCR engine and a segmenter with one model.
TL;DR
- Size: 2B (2.44B params) and 4B (4.83B params) on Hugging Face, and a 9.6M-parameter pixel decoder. Trained with up to 8,192 tokens of context; deployment context length is not disclosed.
- Runs on: A single GPU through Transformers or vLLM in BF16. The package also lists NPU and CPU devices. VRAM needs are not disclosed. A community GGUF ran object-level tasks on a 6 GB RTX 2060.
- Performance: Beats the same-scale VLM baselines in Tencent’s tables on most of 30+ benchmarks (self-reported).
- Best: 71.3 F1 Mean on Dense200 dense detection, versus 58.7 for LocateAnything-3B.
- Worst: 52.6% rejection rate on PR-Bench, so many absent-target queries still get a box.
What is WeVisOne-VL?
WeVisOne-VL (“WeChat Vision All-in-One”) is a perception-focused model from Tencent built on Qwen3-VL. It targets fine-grained localization rather than open-ended chat. Users choose the output granularity: points, boxes, binary masks or continuous alpha mattes.
How does WeVisOne-VL work?
Every task is cast as structured next-token prediction. The model adds coordinate tokens <0> to <1000> for normalized boxes and points. An absent target produces None instead of a guessed box.
For pixel tasks, the model emits an <OBJ> token after each predicted box. Its hidden state becomes a query for the pixel decoder, according to the model card. The decoder has 3 parts:
- Visual Feature Adapter: reuses multi-level Qwen3-VL ViT features instead of a second image encoder.
- Box-Guided Query Decoder: 6 deformable cross-attention blocks seeded by the
<OBJ>state and the box. - Dense Prediction Head: refines output at S/4, S/2 and S/1 into a full-resolution mask or matte.
How was it trained?
A knowledge-graph-driven data engine annotated over 100M images with about 3.6B boxes. SAM3, Rex-Omni and an in-house detector proposed regions, and Qwen3-VL verified them.
Training used 3 SFT stages. Stage 1 aligned output formats on 90M examples. Stage 2 raised input to 1,960 visual tokens and up to 500 boxes per response. Stage 3 trained the pixel decoder, then specialized it for matting on 130M samples. A final GRPO stage used 178K uncertainty-filtered examples, as described in the research paper.
How does WeVisOne-VL perform on benchmarks?
All numbers below are self-reported, using greedy decoding and at most 1,960 visual tokens per image. Detection, referring and OCR use F1 averaged over IoU 0.5 to 0.95.
- Detection: 55.3 F1 on LVIS and 71.3 on Dense200 for the 4B model.
- Referring: 81.8 F1 on RefCOCOg test, versus 77.6 for LocateAnything-3B.
- Rejection: 96.4% on Tencent’s in-house WeRef-Bench (2,880 queries), versus 73.0%.
- Counting: 89.6% exact-match on PixMo-Count test, versus 82.8%.
- OCR: 52.0 F1 on TotalText, ahead of the PP-OCRv5 specialist at 25.7.
- Segmentation: 71.2 gIoU on ReasonSeg val, versus 65.0 for LISA-13B.
- Matting: 7.3 MAD on P3M-500-NP box prompts, versus 15.4 for SAM2-Matting (SAM3). Lower is better.
The 2B model trails the 4B model on most tasks. It still scores better box-prompted matting on P3M-500-NP, at 6.6 MAD.
WeVisOne-VL vs Qwen3-VL, Rex-Omni and LocateAnything
| Feature | WeVisOne-VL-4B | Qwen3-VL-4B-Instruct | Rex-Omni | LocateAnything-3B |
|---|---|---|---|---|
| Developer | Tencent WeChat Vision | Alibaba Qwen | IDEA Research | NVIDIA |
| Size | 4.83B (HF listing) | 4B | 3B (card) | 3B (card) |
| Base model | Qwen3-VL | Not applicable | Qwen2.5-VL-3B-Instruct | Qwen2.5-3B-Instruct + MoonViT-SO-400M |
| License | Apache 2.0 | Apache 2.0 | IDEA License 1.0 | NVIDIA non-commercial |
| Native masks / mattes | Yes (9.6M decoder) | Not in model card | Not in model card | Not in model card |
| Context | Not disclosed (8,192 in training) | 256K native | Not disclosed | Not disclosed |
| LVIS F1 Mean | 55.3 | 44.0 | 46.9 | 50.7 |
| Dense200 F1 Mean | 71.3 | 15.2 | 58.3 | 58.7 |
| RefCOCOg test F1 Mean | 81.8 | 74.3 | 74.3 | 77.6 |
| RefCOCOg-Rej rejection | 67.4% | 18.8% | 0.5% | 0.6% |
| PointBench | 71.9 | 57.1 | 69.9 | 56.9 |
| TotalText F1 Mean | 52.0 | 33.0 | 40.6 | 43.3 |
How do you run WeVisOne-VL?
Weights for the 2B and 4B models are on Hugging Face. The wevisone_vl package wraps Transformers and vLLM backends. Tasks include detection, referring, point, count_box, ocr, segmentation and matting.
from wevisone_vl import WeVisOneVL
model = WeVisOneVL("tencent/WeVisOne-VL-4B", backend="transformers") # or "vllm"
result = model.predict(image, task="matting", query="person")Hosted demos are live for 2B and 4B. Tencent also open-sourced Mask2Alpha, the matting annotator from its data engine. Fine-tuning code is still on the TODO list. GGUF builds cannot produce masks, since the decoder needs native visual features.
Key Takeaways
- 1 model covers detection, OCR, counting, pointing, segmentation and matting.
- A 9.6M-parameter decoder replaces SAM for masks and alpha mattes.
- 4B scores 71.3 F1 on Dense200, 12.6 points above LocateAnything-3B.
- Apache 2.0 weights, unlike non-commercial LocateAnything-3B.
Check out the Paper, GitHub repo, Model weights and Project page. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
The post Tencent WeVisOne-VL Unifies Detection, OCR, Segmentation and Matting appeared first on MarkTechPost.
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 MarkTechPost(Tencent WeVisOne-VL Unifies Detection, OCR, Segmentation and Matting),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。