百度 ERNIE Evolve 自动化研究系统刷新 NanoChat 与 NanoGPT 纪录
百度刚刚发布了 ERNIE Evolve,这是一个自动化研究系统,可改进 AI 训练和推理代码。给定一个代码仓库、一个目标和评估标准,它会选择研究方向、编辑代码并运行实验。只有通过隐藏评估器和独立审查者的更改才会被保留。百度团队在三个公开基准测试上测试了 ERNIE Evolve 自动化研究系统。随后,它针对仍在开发中的 ERNIE-5.5-Flash 优化了推理。
ERNIE Evolve 是百度推出的自动化研究系统,它围绕固定且可衡量的目标优化代码。百度在 X 上宣布它,称其是迈向递归自我改进(RSI)的务实一步。在 RSI 中,AI 系统会改进用于构建和运行 AI 的软件。
该系统运行 2 个嵌套循环。外层循环决定要研究什么。内层循环在每个方向内开发和测试改进。
多个研究路线并行运行,每条路线都有自己的代码版本树。失败的实验会被记录下来,以便后续轮次避免重复。
百度在 NanoChat AutoResearch、NanoGPT Speedrun 和 SOL-ExecBench 上运行了同样的循环。
Recursive 和 Hyra-1.0 的 Speedrun 用时是相对于较早约 79.7 秒的纪录来设定的,因此并非同类可比。Hyra SOL 为自我报告,经由腾讯报道。
随后,百度团队让该系统处理 ERNIE-5.5-Flash 上的 3 个实际工程任务。基线已经由工程师调优过。
需要通过我们推广你的 GitHub 仓库、Hugging Face 页面、产品发布或网络研讨会等?请联系我们。
文章《Baidu 的 ERNIE Evolve 自动化研究系统超越 NanoChat 和 NanoGPT 的纪录》最先出现在 MarkTechPost 上。
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Baidu’s ERNIE Evolve Automated Research System Beats NanoChat and NanoGPT Records)
Baidu has just released ERNIE Evolve, an automated research system that improves AI training and inference code. Given a repository, an objective and evaluation criteria, it picks research directions, edits code and runs experiments. Only changes that pass a hidden evaluator and a separate reviewer are kept. Baidu team tested the ERNIE Evolve automated research system on three public benchmarks. It then optimized inference for ERNIE-5.5-Flash, still in development.
TL;DR
- Size: It is an agent system; Baidu has not disclosed which LLMs power its agents.
- Runs on: Benchmarks used 1 NVIDIA B200 (NanoChat, SOL-ExecBench) and 8×H100 (Speedrun Track 1). Production tests ran on Kunlunxin P800 and GPUs (GPU model not disclosed).
- Performance: Beat the best public result on 3 of 4 public metrics, and cleared production checks on ERNIE-5.5-Flash.
- Best: Generation speed on Kunlunxin P800 reached 623.6% of baseline after 4 research rounds.
- Worst: SOL-ExecBench mean SOL of 0.819, behind the 0.827 leaderboard best.
- Bottom line (best): Real production inference gains: +33.7% deterministic throughput, +14.9% decode speed.
- Bottom line (worst): Self-reported, system not released, and public-benchmark margins are small.
What is ERNIE Evolve?
ERNIE Evolve is an automated research system from Baidu that optimizes code against a fixed, measurable objective. Baidu announced it on X as a practical step toward recursive self-improvement (RSI). In RSI, AI systems improve the software that builds and serves AI.
How does ERNIE Evolve work?
The system runs 2 nested loops. The outer loop decides what to investigate. The inner loop develops and tests improvements inside each direction.
- Director: picks the next research direction and a starting code version. It reads an Experience Bank of past results and a Knowledge Base of papers and code.
- Selector: chooses a candidate from a lane’s tree and names the next hypothesis to test.
- Executor: implements that plan in an isolated environment.
- Evaluator: a fixed program kept outside the repository. No agent can see it. Gains must exceed the noise threshold.
- Reviewer: checks whether a change follows the intended direction or exploits an evaluation loophole.
Several research lanes run in parallel, each with its own tree of code versions. Failed experiments are recorded, so later rounds can avoid repeating them.
How did ERNIE Evolve score on public benchmarks?
Baidu ran the same loop on NanoChat AutoResearch, NanoGPT Speedrun and SOL-ExecBench.
- NanoChat AutoResearch: 0.8763 validation BPB within a 5-minute, single-GPU budget. It added an n-gram branch and a document-retrieval branch beside the network, mixed by a learned gate.
- NanoGPT Speedrun Track 1: 38.9 s to target loss on 8×H100, versus the 39.9 s record. The results repo reports 38.92 s over 18 runs.
- NanoGPT Speedrun Track 3: 2,590 steps, versus 2,690. It decomposed weights into direction and magnitude, then added EMA-Nesterov extrapolation.
- SOL-ExecBench: 0.819 mean SOL across 235 kernel problems on B200, short of the 0.827 leader.
ERNIE Evolve vs other automated research systems
| Metrics | ERNIE Evolve (Baidu) | Recursive | Hyra-1.0 (Tencent Hunyuan) | (auto)²-research (rekursiv.ai) |
|---|---|---|---|---|
| NanoChat val BPB ↓ | 0.8763 | 0.9109 | 0.9015 | 0.8878 |
| Speedrun Track 1 ↓ | 38.9 s | 77.5 s | 76.4 s | Not reported |
| Speedrun Track 3 ↓ | 2,590 steps | Not reported | Not reported | Not reported |
| SOL-ExecBench, self-reported ↑ | 0.819 | 0.754 | 0.771 | Not reported |
| SOL-ExecBench, per Baidu (Eval Stack v1.1) ↑ | 0.819 | 0.676 | 0.736 | Not reported |
| Production use reported | ERNIE-5.5-Flash inference | Not disclosed | Science tasks (math, molecules) | Not disclosed |
| What is released | Final code + logs | Run artifacts | Results repo | Recipe + harness tools |
Recursive and Hyra-1.0 Speedrun times were set against the older ~79.7 s record, so they are not like-for-like. Hyra SOL self-report via Tencent coverage.
What did ERNIE Evolve do for ERNIE-5.5-Flash inference?
Baidu team then pointed the system at 3 live engineering tasks on ERNIE-5.5-Flash. Baselines had already been tuned by engineers.
- Low latency on Kunlunxin P800: generation speed rose to 623.6% of baseline over 4 rounds. Round 1 enabled fused mHC and speculative decoding via the model’s multi-token prediction layer (282.9%). Round 2 cut host-side overhead (588.8%). Round 4 fixed dispatch so DeepEP’s low-latency path engaged.
- Deterministic inference for RL: under SGLang’s deterministic mode, throughput rose 33.7% in about 3 days. Time per output token fell 26% and time to first token fell 12%. A Triton kernel for deterministic sampling, which used about 14% of GPU time, gave the largest single gain.
- Single-request decoding on GPUs: speed rose 14.9% with accuracy unchanged. 3 of 7 research directions produced gains. Key changes: FP16 instructions for tiny FP8 matmuls and a 2-level MoE expert scheduler.
Key Takeaways
- Two nested loops: a Director picks directions, lanes evolve code trees.
- A hidden Evaluator plus a Reviewer guards against reward hacking.
- Public wins on NanoChat (0.8763 BPB) and Speedrun (38.9 s, 2,590 steps).
- ERNIE-5.5-Flash: 623.6% speed on Kunlunxin P800, +33.7% deterministic throughput.
- System stays closed; only results and logs are on GitHub.
Check out the ERNIE Evolve research post and the results repo on GitHub. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
The post Baidu’s ERNIE Evolve Automated Research System Beats NanoChat and NanoGPT Records appeared first on MarkTechPost.
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 MarkTechPost(Baidu’s ERNIE Evolve Automated Research System Beats NanoChat and NanoGPT Records),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。