Abacus.AI 的 Smaug 模型哪里提升最多

Abacus.AI 将同一套微调方案应用于三个差异极大的开放权重模型。收益却落在完全不同的地方,这比任何单一基准测试分数都更能说明问题。
作者:KDnuggets,2026 年 11 月 11 日,发布于 Partners
Smaug Flash 创下该产品线的最大涨幅:智能体式编程 +14.3 分,AutomationBench +13.7 分,NL2Repo-Bench +19 分——直接针对 DeepSeek V4 Flash 在长工具调用循环中的失效模式。
Smaug Mini(28B)在 IFBench、AutomationBench 和 JobBench 上击败 Claude Sonnet 5、Claude Opus 4.6 和 GPT-5.6 Luna——尽管在 LiveBench 智能体式编程上略有下滑。
Smaug Agentic 的分数变化最小(整体 +0.3),但在 7 个多小时内运行了 113 项任务,智能体步骤中位数为 78 步,且零基础设施错误——真正的看点在于行为稳定性,而不是数字。
在三种情况下,收益都集中在基础模型短板最大的地方。同一套方案,不同的基线水平。
AI 智能体性能的意义
AI 智能体可能在未完成任务的情况下就给出输出。在真实场景中,智能体需要理解用户查询、识别合适的工具、分析输出并采取后续行动。经过若干轮循环后,模型会偏离最初的目标,从而损害结果的可信度。
这对开发可靠智能体的团队来说是个难题。聊天机器人只需对单个请求给出响应,而智能体必须保留上下文,并基于相关动作做出决策。随着工作流的延长,不仅会带来额外的成本影响,出错的可能性也会增加。
Abacus.AI 在自身运营中也遭遇了类似的短板:智能体式任务需要数小时才能完成,在各个阶段之间要维护大量上下文,并频繁调用来自其他文档、API 和数据库的工具。这些操作可能中途力竭,甚至根本无法成功完成。Abacus.AI 在其研究页面上这样描述其发现:“让最先进的模型像优秀智能体一样工作,是一个训练难题,而非提示词难题。”通用语言模型尽管能力不俗,却并不总能胜任长时间的使用。提示词缓存最终变得不再有效,而上下文则越来越大。
基本做法是直接改进模型本身,而不是试图通过提示词或权重调整来弥补其短板、修复已发现的问题。这种工作方式最终催生了 Smaug 系列模型。关键不仅在于追问这些模型的效果如何,还在于弄清微调在哪些场景下效果最佳。模型的性能在很大程度上取决于模型本身和任务类型。
Smaug 是 Abacus.AI 针对智能体式工作负载微调的三个开放权重模型组成的系列。每个模型都构建在不同的开放权重基座之上,分别对应能力—效率曲线上的不同位置。三个模型均已在 Hugging Face 上公开提供。
三个基础模型都采用了同一套方法:由人工整理的真实智能体轨迹,加上基于困难案例构建的人工数据。正因如此,各模型的提升模式才具有可比性,其间的差异也才有意义。
常驻在线的企业智能体,速度 + 成本
多模态、紧凑、一次性任务
前沿规模、长周期编码
把同一套方案应用到三个差异极大的基础模型上,本身就是一个研究主张:该方法足够通用,无论架构或规模如何,都能带来一致且可衡量的提升。下文将逐一考察各模型的结果,判断这一主张在哪里最为成立——又在哪里出现了取舍。
Smaug Flash:该系列中提升幅度最大
Smaug Flash 是 Smaug 方法论带来最大基准测试跃升的模型。它是在 DeepSeek V4 Flash 上做的智能体微调,后者常被视为该系列的主力,Smaug Flash 面向持续运行的企业级自我改进智能体而打造。其基础模型 DeepSeek V4 Flash 速度快、成本低,但在长上下文工具使用中容易出现打转和混乱。此次微调正是针对这一特定的失效模式。
LiveBench 上的结果体现了这种精准性。智能体编码得分从 46.8 提升至 61.1,提升 14.3 分,是整个 Smaug 系列中单项类别里最大的增幅。在 AutomationBench 上,得分提升 13.7 分(从 25.1 升至 38.83);在 NL2Repo-Bench 上提升 19 分(从 54.2 升至 73.3)。这些并非边际改善;它们意味着模型恰恰在其所设计的工作流中补齐了一项真实的能力缺口。
Smaug Flash、DeepSeek V4 Flash 与 Claude Sonnet 5 对比:
¹ = 厂商报告的分数,并非在 Abacus.AI 的测试框架下运行得出。
LiveBench 逐类别细分:
数据分析(-0.2)和推理(-0.4)上的这些小幅下降可以视为一种权衡,因为要让模型对生产环境中的智能体真正有用,打磨这些能力——例如正确使用工具、处理更长的上下文——比在推理上保持满血表现更为重要。就评估这些能力的基准测试而言,Smaug Flash 的表现优于 Claude Sonnet 5。
Smaug Mini:一个出人意料的故事
Smaug Mini 是 Qwen3.8-27B 模型的进阶迭代版本,也是 Smaug 系列中拥有 270 亿参数的小规模版本。该模型面向多模态任务,如图像/视频解读、指令遵循和工具调用。
凭借相对更小的体量,在特定的硬件与推理配置条件下,其部署可能会更容易。值得注意的是,在某些通常是先进闭源模型表现更优的基准测试中,Smaug Mini 取得了高于这些模型的性能。
Smaug Mini 与基础模型及前沿闭源模型对比:
¹ = 厂商自报分数,并非在 Abacus.AI 的评测框架下运行得出。
Smaug Mini 的 LiveBench 分类概况:
基本上,Smaug Flash 在 LiveBench 上的最大增益出现在智能体式编码,但由于 Smaug Mini 的基座 Qwen3.8-27B 在编码类任务上本已很强,其最大增益出现在数学(+3.5)和推理(+2.8),这也就没有留下改进空间。微调找到的是那些真正仍然敞开的缺口。在 NL2Repo-Bench 上,Mini 相对其基座也提升了 13.5 分(42.3 到 55.8),这一增益对于代码相关工作流值得注意。
这个反转很重要:Smaug Mini 的智能体式编码得分(-0.6)实际上比其基座略低,但它在 IFBench(指令遵循)、AutomationBench 和 JobBench 上领先于 Claude Sonnet 5、Claude Opus 4.6 和 GPT-5.6 Luna。尤其是 JobBench,相对基座提升了 17.1 分(33.4 到 50.5)。这正是基准测试类别所能衡量的内容与模型在现实中实际能达到的能力之间的区别所在——因为不同的基准测试衡量的是不同的能力,某一领域的下降并不意味着其他领域存在欠缺。
Smaug Agentic:当行为比数字更重要
Smaug Agentic 是这一系列中体量最大的一个。它是对 Kimi K3 的微调,Kimi K3 是 Moonshot AI 的当前最先进混合专家模型,拥有 2.8 万亿参数。它相对基座的基准测试差值(delta)是三个 Smaug 模型中最小的。研究页面直言:“行为上的变化比这些差值更大。”
Smaug Agentic 与 Kimi K3 基座对比:
Smaug Agentic 的 LiveBench 分类概况:
基础模型胜出的两项基准测试应当明确说明:Terminal-Bench 2.1(86.5 对 88.3)和 MMMU-Pro(81.0 对 81.6)。这些数据在官方数据中并未被藏进脚注,任何诚实的分析都应当把它们列出来。
这些数字无法体现的,是 Abacus.AI 所描述的该模型最主要的行为改进:抑制推理失控,同时保留正常的思考过程。在 113 项智能体式编程任务、超过七小时的连续运行中,Smaug Agentic 每项任务的中位智能体步骤数为 78 步,没有出现基础设施错误,也没有超时。在前沿规模上,这次微调并未带来标题级的分数跃升,而是产出了一个在前沿规模模型真正被部署的长循环中不会崩溃的模型。
这些提升背后的训练方法
三款 Smaug 模型都采用了同一套方法。它把来自真实智能体工作的人工筛选轨迹,与基于高难度挑战性样本构建的合成数据相结合。训练数据经过过滤,剔除那些并非真正解决问题、而是钻任务评分标准空子的样本。目标是奖励真正的解题能力,而不是格式上的合规。
三款模型的差异在于具体的微调方式:
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Where Does Abacus.AI’s Smaug Model Improve Performance the Most?)
Abacus.AI applied the same fine-tuning recipe to three very different open-weight models. The gains landed in completely different places & that tells you more than any single benchmark score.
By KDnuggets on November 11, 2026 in Partners

Smaug Flash posts the line's largest jumps: +14.3 pts in agentic coding, +13.7 on AutomationBench, +19 on NL2Repo-Bench - directly targeting DeepSeek V4 Flash's failure mode in long tool-use loops.
Smaug Mini (28B) beats Claude Sonnet 5, Claude Opus 4.6, and GPT-5.6 Luna on IFBench, AutomationBench, and JobBench - despite a slight dip in LiveBench agentic coding.
Smaug Agentic shows the smallest score deltas (+0.3 overall) but ran 113 tasks over 7+ hours with a median of 78 agent steps and zero infrastructure errors - behavioral stability is the story, not the numbers.
In all three cases, the gains concentrate where the base model's gaps were widest. Same recipe, different floors.
Significance of AI Agents' Performance
An AI agent could provide an output without completing the task. In real-world scenarios, there is a need for the agent to comprehend the user query, identify an appropriate tool, analyze output, and take subsequent actions. After some number of cycles, the model would veer away from the initial goal, thus compromising on the credibility of the result.
It presents a problem for teams developing reliable agents. While a chatbot provides a response to a single request, an agent has to retain its context and make decisions based on related actions. With the extension of workflow, there would be additional cost implications as well as chances of mistakes.
Abacus.AI experienced similar shortcomings in its own operations, with agentic tasks taking several hours to complete, maintaining a large amount of context from stage to stage, and frequently invoking tools from other documents, APIs, and databases. These operations could run out of steam or not be completed successfully at all. Abacus.AI describes its findings on its research page in the following way: “Making a state-of-the-art model work like a good agent is a training challenge, not a prompting challenge.” General language models, despite their abilities, were not always ready for extended usage. Prompt caching eventually became less effective, and context grew bigger.
The basic approach was to improve the models unaided instead of trying to compensate for their shortcomings by using prompting or weight adjustments that would fix the identified problems. This way of working has resulted in creating the family of Smaug models. It is essential not only to ask how effective these models are but also where the fine-tuning works the best. The performance of the models depends greatly on the model and task type.
Smaug is a family of three open-weight models fine-tuned by Abacus.AI for agentic workloads. Each model is built on a different open-weight base, chosen for a distinct point on the capability–efficiency curve. All three are publicly available on Hugging Face.
A common methodology involving human-curated, actual agentic trails and artificial data based on hard cases was used for all the three bases, which is why the gain patterns are comparable and the differences are meaningful.
Always-on enterprise agents, speed + cost
Multimodal, compact, one-off tasks
Frontier-scale, long-horizon coding
Applying the same recipe to three very different bases is itself a research claim: that the methodology is general enough to produce consistent, measurable improvements regardless of architecture or scale. What follows examines each model's results to determine where that claim holds strongest — and where trade-offs emerge.
Smaug Flash: The Biggest Gains in the Line
Smaug Flash is where the Smaug methodology delivers its largest benchmark jumps. It is an agentic fine-tune of DeepSeek V4 Flash, often known as the workhorse of the line, built for enterprise self-improving agents that run continuously. Its base, DeepSeek V4 Flash, is fast and cost-effective but susceptible to spins and confusion in long-context tool use. The fine-tuning targeted that specific failure mode.
The results on LiveBench reflect this precision. The +14.3 point gain in agentic coding ranging from 46.8 to 61.1 is the largest single-category delta in the entire Smaug line. On AutomationBench, the score improved by 13.7 points (25.1 to 38.83), and on NL2Repo-Bench the gain was 19 points (54.2 to 73.3). These are not marginal improvements; they represent the model closing a genuine capability gap in exactly the workflows it was designed for.
Smaug Flash vs DeepSeek V4 Flash vs Claude Sonnet 5:
¹ = vendor-reported score, not run under Abacus.AI's harness.
LiveBench Category-by-Category Breakdown:
These modest reductions in Data Analysis (-0.2) and Reasoning (-0.4) could be seen as a trade-off since it is more important to hone these skills to be useful for production agents, for example, the ability to use tools correctly and process longer contexts, than to retain full performance in reasoning. In terms of the benchmarks evaluating these skills, Smaug Flash outperforms Claude Sonnet 5.
Smaug Mini: The Surprising Story
Smaug Mini is an advanced iteration of the Qwen3.8-27B model and the small-scale version of the Smaug series with 27 billion parameters. The model is intended for multimodal tasks such as image/video interpretation, instruction following, and tool use.
With its comparatively smaller size, deployment could potentially become easier under certain conditions of hardware and inference configuration. It is important to note that Smaug Mini has achieved higher performance than several advanced closed models in certain benchmarks where the latter usually excel.
Smaug Mini vs Base and Frontier Closed Models:
¹ = vendor-reported score, not run under Abacus.AI's harness.
LiveBench Category Profile for Smaug Mini:
Basically, Smaug Flash has the biggest LiveBench gain in agentic coding, but because Qwen3.8-27B, the base of Smaug Mini is already strong at coding-style tasks, it’s largest gains are in Mathematics (+3.5) and Reasoning (+2.8) which leaves no room for improvement. The fine-tuning found the gaps that were actually open. On NL2Repo-Bench, Mini also gained 13.5 points over its base (42.3 to 55.8), a gain worth noting for code-related workflows.
The twist is important: Smaug Mini's agentic coding score (-0.6) is actually slightly lower than its base, yet it leads Claude Sonnet 5, Claude Opus 4.6, and GPT-5.6 Luna on IFBench (instruction following), AutomationBench, and JobBench. JobBench in particular shows a 17.1 point gain over the base (33.4 to 50.5). Herein lies the distinction between what a benchmark category is able to measure and what the model is capable of achieving in reality — since different benchmarks measure different abilities, and a drop in one area does not mean a lack in others.
Smaug Agentic: When Behavior Matters More Than Numbers
Smaug Agentic is the largest one in this line-up. It's a fine-tuning of Kimi K3, which is the state-of-the-art Mixture of Experts model from Moonshot AI, with 2.8 trillion parameters. Its benchmark deltas over the base are the smallest of the three Smaug models. The research page states this directly: "the behavioral change is bigger than the deltas."
Smaug Agentic vs Kimi K3 Base:
LiveBench Category Profile for Smaug Agentic:
Two benchmarks where the base model wins should be stated clearly: Terminal-Bench 2.1 (86.5 vs 88.3) and MMMU-Pro (81.0 vs 81.6). These are not hidden in footnotes in the official data, and they belong in any honest analysis.
What the numbers do not capture is what Abacus.AI describes as the model's primary behavioral improvement: suppressing runaway reasoning while preserving normal deliberation. Across 113 agentic coding tasks and more than seven hours of continuous operation, Smaug Agentic ran a median of 78 agent steps per task with no infrastructure errors and no timeouts. At frontier scale, the fine-tuning did not produce headline score jumps, rather it produced a model that does not break down in the long loops where frontier-scale models are actually deployed.
The Training Methodology Behind the Gains
The same methodology was applied to all three Smaug models. It combines human-curated traces from real agentic work with synthetic data grounded in hard, challenging examples. Training data is filtered to remove instances that exploit the scoring criteria of a task rather than actually solving it. The goal is to reward genuine problem-solving, not format compliance.
What differs across the three models is the specific fine-tuning approach:
Smaug Flash: The attention factor matrices are alone modified with the help of three LoRA adapters, two of which have been trained by supervised learning and one with preference training. The rest of the components stay unchanged and byte identical to that of the official DeepSeek V4 Flash release. This ensures that the model is fully compatible with the base model on any inference stack, with the ability to have full 1-million-token context window.
Smaug Agentic: Supervised fine-tune of Kimi K3 with reasoning tokens masked from the loss during training. This preserves the model's native thinking process while teaching better tool-use habits. Every architectural parameter is unchanged from the base, making it a drop-in replacement for any inference stack that serves Kimi K3.
Smaug Mini: The same general methodology applied to a 27B dense base, producing consistent patterns: gains in the categories where the base had the most headroom, and trade-offs in areas already well-covered.
"The core philosophy of this methodology is captured in one quote on the Abacus.AI's research page: Most fine-tuning silently overwrites the reasoning of a model for the sake of task compliance – you get the format, but you lose the mind.” Smaug training recipes try to get around this pitfall by using the training budget on data which improves the model, not data which leverages benchmark scores. This might be a factor explaining the significant increase in agentic coding (+14.3 points) seen in Smaug Flash – its fine-tuning addressed the weakness of the base model."
What the Benchmarks Are Actually Measuring
Understanding where Smaug improves most requires understanding what each benchmark is designed to measure.
LiveBench is an ICLR Spotlight paper from Abacus.AI. It aims to prevent data contamination by generating questions based on the most up-to-date sources such as math contests, LeetCode questions, arXiv abstracts, news articles, and Kaggle datasets. Questions are refreshed every month to make it harder for models to have memorized the answers before training. Scoring is done based on ground truth, not LLM as a judge scoring method, which can be erroneous on challenging prompts. There are seven categories in LiveBench: Reasoning, Coding, Agentic Coding, Instruction Following, Mathematics, Language, and Data Analysis.
The other benchmarks each measure narrower skills:
AutomationBench — real-world tool-use automation (600 public tasks, strict pass)
DeepSWE 1.1 — software engineering executed by coding agents
NL2Repo-Bench — translating natural language to repository-level code
IFBench — instruction following precision (300 tasks, prompt-level loose)
JobBench — multi-step job-level task completion (65 tasks, official protocol)
GPQA Diamond — graduate-level science reasoning
SciCode — scientific coding
AA-LCR — long-context reasoning
MMMU-Pro — multimodal understanding
Because these benchmarks measure different, though related, skills, a high score on one does not guarantee a high score on another. That is why Smaug Mini leads all compared closed models on AutomationBench and JobBench while showing a -0.6 delta in LiveBench's Agentic Coding category. They are not the same measure. It is also why some competitor scores marked ¹ in the tables above which are the vendor-reported figures must be interpreted carefully, not run under Abacus.AI's own harness.
Where Gains Are Biggest: Comparison Across All Three Models
LiveBench Agentic Coding (delta vs base)
LiveBench Instruction Following
AutomationBench (delta vs base)
NL2Repo-Bench (delta vs base)
Behavioral endurance at scale
78 steps/task, 7+ hours, zero timeouts ★
The pattern across all three is consistent with one explanation: the gains concentrate where the base model's gaps were widest and where the training data was designed to push. Flash's base had the largest failure mode in agentic coding and hence it was prone to spinning in long tool-use loops in return, the fine-tuning produced the largest gain there (+14.3). Mini's base was already strong at coding, so the gains spread across math, reasoning, and language instead. Agentic's base was already at frontier capability, leaving little room for benchmark score improvement; the fine-tuning focused on behavioral reliability rather than raw capability.
Implications for Practical Use
It is more helpful when evaluating the use of Smaug models to start with the workload that the model is designed to handle.
For applications that demand sustained, long-context tool use, document processing, or repeated API calls across hours-long workflows, Smaug Flash's +14.3 agentic coding and +13.7 AutomationBench gains over its base are the most directly relevant data. Where multimodality and instruction-following tasks are concerned, and where a model that fits on a single GPU at 28B model size is desired, the relevant comparisons come from Smaug Mini’s IFBench score (82.0), AutomationBench (41.8), and JobBench (50.5), all beating Claude Sonnet 5 and GPT-5.6 Luna. Where longevity in hundreds of steps per agent is required without infrastructural failure, the behavioral stability information from Smaug Agentic (median: 78 steps, 7+ hours, zero timeouts from 113 tasks) carries far more weight than any benchmark delta.
Other useful metrics besides the benchmarks are: task completion ratio, latency at context length as in production, cost per thousand agent steps, and reproducibility.
Open weights provide access and deployment flexibility, but do not guarantee efficiency or ease of integration without testing against the actual workload. Hardware configuration, context length, and task difficulty all affect inference behavior in ways that benchmark scores cannot predict.
The benchmark results are a starting point for identifying which models are worth evaluating — not a substitute for testing on the actual task.
Each improvement seen in the Smaug series depends on what particular weakness this version’s fine-tuning aimed to solve. For example, Smaug Flash exhibits the biggest improvements in its own class, increasing scores by 14.3 points for LiveBench agentic coding, 13.7 points for AutomationBench, and 19 points for NL2Repo-Bench. This is because of the weaknesses that the original model had with the extended multi-step agent workflow. Smaug Mini, on the other hand, demonstrates improvements in mathematics (+3.5), reasoning (+2.8), and language (+2.7). It beats the mentioned frontier closed models in instruction following and job-level automation in spite of being a 28 billion parameter model. On the other hand, Smaug Mini shows decline in agentic coding score (by 0.6 points) in comparison with its original model. There is also a unique case of Smaug Agentic, which shows a modest improvement of +0.3 points but has high operational performance that is defined by 78 agent steps per task, seven plus hours of run time, and zero infrastructure errors.
The results stand out by virtue of their divergence from one model to another. The same training regimen is reflected differently based on the specific model that is used and the constraints of that particular model. The results also include information about those parts of the testing procedure where there is no improvement in performance. For instance, the Kimi K3 base model outperforms Smaug Agentic in Terminal-Bench 2.1 and MMMU-Pro while Smaug Mini shows some reduction in performance in LiveBench agentic coding. Including such limitations makes the results of this evaluation exercise more realistic.
Developers seeking a performance evaluation of various open-weight models for use in agent-based applications would find the results of Smaug family very useful. Instead of identifying the best model for any application, the performance results would help understand which models might be better under certain circumstances.
原文为英文,正文由自动抓取并保留原文。
原文发布于 KDnuggets(Where Does Abacus.AI’s Smaug Model Improve Performance the Most?),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。