谷歌发布Agile Deliberation灰色地带图像分类框架
Google Research 发布了 Agile Deliberation,这是一个人在回路(human-in-the-loop)框架,用于围绕主观概念构建图像分类器。比如「健康食品」「鼓舞人心的艺术作品」,或内容审核中的标题党。大多数工具都假定用户已经知道概念边界,而 Agile Deliberation 假定他们并不知道,并帮助他们找到边界。
Agile Deliberation 是 Google Research 的一个框架,它把模糊的概念转化为结构化的书面定义。这个定义有两大作用:它既是人类可读的策略,也是 VLM 分类器的文本提示词。每一轮都会保留在用户标注上 F1 得分最佳的候选定义。
这一设计来自对一组内容审核专家的访谈。专家们会先界定概念范围,再在边界图像上对其进行细化。而找到这些图像正是最耗时的环节。
第 1 阶段使用 Gemini 2.5 Pro 进行提示链(prompt chaining),把概念拆解为最多 3 个单元概念。每个单元都会得到候选的正向和负向子概念,并配以检索到的图像展示。对于健康食品,正向子概念可能是沙拉或谷物碗,负向子概念可能是加工食品或未经烹调的原材料。用户可以保留、翻转或删除每一个子概念。
第 2 阶段跳过了经典的不确定性采样。该分类器是一个以提示词驱动的生成式 VLM,因此其置信度分数往往未经过校准。相反,系统会主动寻找在含义上处于边界的图像。
该流水线包含 4 个步骤:
用户为每张图像打标签,并就不一致之处添加简短备注。LLM 将这些备注转化为候选定义。贪心更新会保留得分最高的那一个。
主观概念没有固定的真值,因此 Google 团队跳过了静态基准测试。它开展了 18 场实时会话,每场约 90 分钟。9 名没有专业主持经验的参与者每人进行了 2 场会话。一场会话使用了敏捷审议(Agile Deliberation),另一场使用了人工审议。每位参与者花约 45 分钟定义一个概念,然后标注了 200 张留出的测试图像。
这两个概念分别是“付费参与”点击诱饵和“健康食品”。检索分别在一个专有的 1 亿图像数据集和 WebLI 上进行。
来源:CVPR 2026 论文的表 1。F1 为对参与者取平均的值。
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Google AI Research Releases Agile Deliberation Framework for Gray-Area Image Classifiers)
Google Research has released Agile Deliberation, a human-in-the-loop framework for building image classifiers around subjective concepts. Think ‘healthy food’, ‘inspiring artwork’ or clickbait in content moderation. Most tooling assumes users already know the concept boundary. Agile Deliberation assumes they do not, and helps them find it.
TL;DR
- Framework: It prompts Gemini 2.5 Pro and Gemini 2.5 Flash through the API, without fine-tuning.
- Runs on: A Jupyter or Colab notebook. Retrieval uses CLIP embeddings and a FAISS index on CPU or GPU. The demo index starts at about 260 MB.
- Performance: 10.5 F1 points above zero-shot Gemini 2.5 Flash on average, across 2 concepts and 18 live sessions.
- Best: Paid to play F1 of 0.59 vs 0.48 zero-shot, with precision up 22 points (0.35 to 0.57).
- Worst: Recall drops 17 points on paid to play (0.80 to 0.63) and 15 points on healthy food.
What is Agile Deliberation?
Agile Deliberation is a Google Research’s framework that turns a vague concept into a structured written definition. That definition does two jobs. It is a human-readable policy, and it is the text prompt for a VLM classifier. Each round keeps the candidate definition with the best F1 on the user’s labels.
The design came from interviews with a group of content moderation experts. Experts scoped a concept first, then refined it on borderline images. Finding those images was the slow part.
How does concept scoping work?
Stage 1 breaks the concept into at most 3 unit concepts, using prompt chaining with Gemini 2.5 Pro. Each unit gets candidate positive and negative subconcepts, shown with retrieved images. For healthy food, positives might be salads or grain bowls. Negatives might be processed food or raw, unprepared ingredients. The user keeps, flips or drops each one.
How does concept iteration find borderline images?
Stage 2 skips classic uncertainty sampling. The classifier is a prompted generative VLM, so its confidence scores are often uncalibrated. Instead, the system hunts for images that are borderline in meaning.
The pipeline runs 4 steps:
- Gemini 2.5 Flash writes borderline queries, such as ‘salads with heavy mayo dressings’. Each query retrieves 50 to 100 images.
- Images are de-duplicated, encoded, and grouped into clusters with dictionary learning.
- A UCB multi-armed bandit picks the next cluster. It favors clusters where users corrected the model or left rich feedback.
- A VLM summarizes each image in 1 sentence. The user sees at least 5 images that share 1 ambiguity.
Users label each image and add short notes on disagreements. An LLM turns those notes into candidate definitions. A greedy update keeps the best-scoring one.
How was it evaluated?
Subjective concepts have no fixed ground truth, so Google team skipped static benchmarks. It ran 18 live sessions of about 90 minutes each. 9 participants without professional moderation experience did 2 sessions each. One session used Agile Deliberation, the other used manual deliberation. Each participant spent about 45 minutes defining a concept, then labeled 200 held-out test images.
The two concepts were ‘paid to play’ clickbait and ‘healthy food’. Retrieval ran over a proprietary 100M-image dataset and WebLI, respectively.
What do the benchmarks show?
| F1 (Agile Deliberation group) | Zero-shot | Modeling Collaborator | Agile Deliberation | Gain vs zero-shot |
|---|---|---|---|---|
| Paid to play | 0.48 | 0.53 | 0.59 | +11 points |
| Healthy food | 0.48 | 0.50 | 0.58 | +10 points |
Source: Table 1 of the CVPR 2026 paper. F1 averaged over participants.
How does it compare with other subjective classifier methods?
| Feature | Agile Deliberation | Modeling Collaborator | Agile Modeling | Zero-shot VLM |
|---|---|---|---|---|
| Venue | CVPR 2026 | CVPR 2024 | ICCV 2023 | Baseline in paper |
| Core idea | Deliberation on semantic borderline images, then automatic prompt optimization | LLM agent decomposes the concept and auto-labels data | User labels images in a real-time loop | Classify from the initial description |
| Human effort | About 45 min of labeling and feedback | About 100 labels plus chat (vs about 2,000) | Under 30 min in a 14-person study | 1 concept description |
| Handles evolving definitions | Yes | No (static) | No (static) | No |
| Classifier | Prompted Gemini 2.5 Flash | Lightweight distilled classifier | Not disclosed | Prompted Gemini 2.5 Flash |
| F1, paid to play | 0.59 | 0.53 | Not tested | 0.48 |
| F1, healthy food | 0.58 | 0.50 | Not tested | 0.48 |
| Code | Apache 2.0, GitHub | Not disclosed | Not disclosed | Gemini API |
How can developers run it?
The repo ships a Demo.ipynb that walks through the healthy food example. It needs a Gemini API key and defaults to gemini-2.5-pro and gemini-2.5-flash. A setup script builds a FAISS index over LAION-400M CLIP embeddings. The default 2 shards take about 260 MB. All 100 shards take about 13 GB.
Key Takeaways
- Agile Deliberation helps users discover a subjective concept’s boundary, not just label it.
- It finds borderline images by meaning, not by classifier confidence.
- F1 rose 10.5 points over zero-shot Gemini 2.5 Flash on average.
- Users reported significantly lower effort and stress than with manual prompt editing.
- The code is open on GitHub, with a Gemini-based demo notebook.
Check out the paper, arXiv version, GitHub repo and Google Research blog. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
The post Google AI Research Releases Agile Deliberation Framework for Gray-Area Image Classifiers appeared first on MarkTechPost.
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 MarkTechPost(Google AI Research Releases Agile Deliberation Framework for Gray-Area Image Classifiers),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。