11-20 更新 实时
首页/ 技术栈/ 正文

推出面向 Serverless 的 Genie Migrate

技术栈 来源:Databricks Blog 2026-11-20 00:00 0 阅读 机器翻译
推出面向 Serverless 的 Genie Migrate

你今天就可以开始使用。该智能体会保持你的工作负载代码不变,并将不确定的 Spark 配置标记出来交由你决定,而不是悄悄移除它们,让你始终掌控生产环境中运行的内容。

Databricks 中的 Serverless 计算提供了一种更简单、更快捷的方式来运行你的数据和 AI 工作负载。它消除了管理基础设施的运维负担,自动处理预置、扩缩容和优化,让团队可以专注于业务成果。借助 Serverless,组织可以更快行动、提高效率,并释放云原生计算的优势,而无需管理和维护复杂的集群。

仅仅在过去一年,我们的 Serverless 使用量就增长了好几倍,如今已有数千客户在其上运行笔记本和作业。大多数团队仍拥有经典集群资源,他们希望迁移到 Serverless 世界,以便在不自行管理基础设施的情况下获得成本和运维收益。逐个迁移每个工作负载的工作,常常使我们的客户无法为现有工作负载释放 Serverless 的收益。

你可能已经见过这种情况。你选择一个作业,把它切换到 Serverless,然后就会遇到错误、访问模式不兼容、不适用的 Spark 配置、在经典环境上安装了但在 Serverless 上没有安装的库。原本应该花一个下午的事情会拖成几周,而大部分时间都花在你以前已经做过很多次的转换上。

不再如此!今天,我们很高兴地宣布推出 Genie Migrate for Serverless(Beta 版),这是 Databricks 的智能体,可将你的作业和笔记本从经典计算迁移到 Serverless 计算——轻松、毫不费力。工作负载迁移变成了一个审核并批准步骤,而不是一个研究项目。

迁移过去掌握在从业者手中

直到现在,把工作负载迁移到无服务器一直是手工完成的。经典集群配置如何映射到无服务器、哪些库能够正常解析、哪些 Spark conf 仍然适用、某个工作负载需要哪种访问模式——这些知识既存在于从业者的头脑中,也存在于反复运行 notebook 或作业、阅读报错信息的试错过程里。

Genie Migrate for Serverless 把这层转换从你身上转移到平台上,就像无服务器把资源供给和扩缩容转移到平台上一样。留给你的仍然是判断:是否现在就迁移某个工作负载、提议的改动是否合理、迁移后的运行是否健康。智能体自行完成这层转换,并以 diff 的形式提出建议,在你确认之前不会改动任何东西。

为了构建成功的无服务器迁移工具,我们牢记以下关键原则:

不破坏生产环境:关键业务工作负载上出现无声的退化,比完全不迁移还要糟糕,因此智能体会先评估每个工作负载,并标示出哪些可以安全迁移。

对每一处改动都可掌控:能够看清哪些会变、哪些不会变,能够接受或拒绝每一处提议的修改,并能确信它不会碰你没有批准的代码。

干净的回滚:如果无服务器运行出了问题,能在几次点击内回到迁移前的状态。

这三项需求决定了迁移智能体的工作方式,下面的工作流程则定义了它如何执行迁移:评估哪些可以迁移、进行转换,并以可逆的方式应用改动。

Genie Migrate for Serverless 能做什么

评估(Assess):扫描你的作业并对整个工作负载群进行分类,让你清楚哪些现在就能迁移、哪些需要管理员介入、哪些尚不具备迁移条件。

转换(Translate):为工作负载提出迁移到 serverless 所需的基础设施与代码变更,每项变更都以带标注的 diff(并排展示哪些内容会变、哪些保持不变)的形式呈现,供你审阅。

应用(Apply,可逆):验证转换结果并加以应用,且不会抢占正在运行的工作负载,同时保持原始工作负载完整,这样一旦 serverless 运行不健康,你就能回滚。

该智能体可通过 Genie 内的 /compute migrate 斜杠命令,对任意作业或 notebook 运行。它完全托管,你无需任何设置即可开始扫描。由于由 Databricks 托管,智能体会在你自己的工作区内读取你的作业配置和工作负载代码。

下面我们逐一深入了解上面列出的各个部分。

在你动手改动任何东西之前,该智能体会扫描你的 Jobs 和 Notebooks,并为每个工作负载标注其 serverless 就绪程度。你可以直接看到哪些工作负载是理想的迁移候选,而不必手动逐一审计——省去了繁琐的人工排查与分类工作。

关于外部来源的一点说明:如果某个工作负载的真实来源位于工作区之外,例如由 Databricks Asset Bundles 管理的作业,或 Git 文件夹中的 notebook,那么该智能体目前还无法迁移它。对这类来源的支持即将推出。

一旦你选定了一个符合条件的工作负载,迁移智能体会提出迁移到 serverless 所需的变更,并在写入任何内容之前提交供审查。在你批准之前,不会应用任何变更。

迁移智能体会有效地转换经典模式与 serverless 之间存在差异的各个部分:

计算配置:经典集群配置会映射为 serverless 的计算选择。

环境与库:库列表会针对 serverless 运行时进行解析,环境定义也会为 serverless 生成。

Spark 配置:仍然适用的配置会被保留,不再适用的配置会被提示出来交由你决定,而不是被静默移除。

工作负载代码:如果代码使用了无法在 serverless 上运行的写法,例如与 Spark Connect 不兼容的 API,智能体会重写这些代码,并以 diff 形式展示变更。

每一项变更,包括代码重写,都以可审查的 diff 形式呈现。在应用之前,智能体会展示一份并排对比,列出哪些会变更、哪些不会,这是一份明确的契约,你可以交给管理者或值班轮值团队。

欢迎观看最新演示,了解如何使用 Genie Migrate for Serverless 轻松迁移经典工作负载。

如果智能体对某件事不确定,比如某个 Spark 配置项是否仍然需要,它会报告该配置项,而不是将其移除。对静态扫描而言,某个配置项可能看起来未被使用,但运行时加载的监听器却依赖它。静默移除会丢掉团队赖以为继的遥测数据。智能体把这一决定权留给你。

应用迁移是你风险敞口最大的环节,因此 Genie Migrate for Serverless 在这里最为保守。它目前会执行确定性的正确性检查,更深入的运行时验证即将推出。

Apply 之前先验证:智能体检查转换结果是否自洽、库依赖能否在无服务器运行时中解析、配置能否干净地应用,从而让故障在工作负载运行之前就暴露出来,而不是等到明天才触及 SLA。

有人盯守的首次运行:由于改动是应用到一个克隆作业上的,你可以按需运行该克隆一次,确认它在无服务器上运行正常,然后再把它提升到你的原始作业上。这样首次无服务器运行就在你的眼皮底下发生,而非无人值守。

如果你选择不迁移,克隆的作业或对 notebook 的建议修改将不会被应用。

有一点需要注意:如果你用改动后的版本做了测试,这并不会撤销在此期间无服务器运行写入生产表的数据。

目前验证是静态的。它无需运行工作负载即可确认转换是可靠的,并且会如实说明它已经证明了什么、尚未证明什么。更深入的检查即将通过 Sandbox validation 推出。有了它,智能体将能够在隔离沙箱中运行迁移后的工作负载,而不影响你的生产表,并将其输出与经典基线进行对比——这样你就能在一个关键作业上、在 Apply 之前而非之后,看到运行时与正确性信号。

在 Genie 中一次迁移一个工作负载,对少量作业来说效果不错。但如果面对更大规模的作业群,你可以借助 Databricks CLI 在后台运行 Genie Migrate for Serverless。通过 compute-migrations 命令组从脚本中为多个作业启动迁移,智能体无需打开 Genie 会话,就能为每个作业构建一个无服务器的可用副本。在你审查并批准迁移之前,原始作业不会发生任何变化,而批准后会保留该作业的 ID、调度和权限。批量迁移目前支持作业(对 notebook 的支持即将推出),并要求 Databricks CLI 版本为 1.24.0 或以上。

让我们把 Genie Migrate for Serverless 的三大关键能力再串起来看一遍:

评估(Assess):识别并分类整个作业群中哪些作业符合无服务器迁移的条件。

翻译(Translate):以带标注、可审查的 diff 形式提出基础设施和代码变更建议。

可逆地应用(Apply reversibly):目前执行确定性验证,沙箱验证即将推出。

下面简要总结一下:目前哪些工作负载适合使用该智能体进行迁移,以及未来会推出哪些新能力。

你现在就可以使用 Genie Migrate for Serverless 把工作负载迁移到无服务器。以下是上手方法:

原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Introducing Genie Migrate for Serverless)

You can get started today. The agent leaves your workload code untouched and flags uncertain Spark configs for your decision rather than silently removing them, keeping you in control of what runs in production.

Serverless compute in Databricks offers a simpler, faster way to run your data and AI workloads. It removes the operational burden of managing infrastructure, automatically handling provisioning, scaling, and optimization so teams can focus on their business outcomes. With serverless, organizations can move faster, improve efficiency, and unlock the benefits of cloud-native compute without managing and maintaining complex clusters.

Our serverless usage has grown multifold just in the past year, and today thousands of customers run notebooks and jobs on it. Most teams still have a classic fleet they want to migrate over to the serverless world in order to get the cost and operational benefits without managing infrastructure themselves. The work of moving each workload one at a time often prevents our customers from unlocking serverless benefits for their existing workloads.

You have probably watched this happen. You pick a job, flip it to serverless, and then get hit with an error, an access-mode incompatibility, a Spark config that does not apply, a library that is installed on classic but not on serverless. What should take an afternoon stretches into weeks, and most of the time goes to translation you have done many times before.

Not any more! Today we are excited to announce the launch of Genie Migrate for Serverless (in Beta), the Databricks agent that migrates your jobs and notebooks from classic to serverless compute - easily and effortlessly. A workload migration becomes a review-and-approve step rather than a research project.

Migration used to live in the practitioner's hands

Until now, migrating a workload to serverless was manual. The knowledge of how a classic cluster config maps to serverless, which libraries resolve, which Spark confs still apply, and which access mode a workload needs lived in the practitioner's head and in the trial-and-error of running the notebook or job and reading the failure.

Genie Migrate for Serverless moves that translation off you and onto the platform, the same way serverless moved provisioning and scaling onto the platform. What stays with you is judgment: whether to migrate a workload now, whether the proposed changes look right, and whether the migrated run is healthy. The agent handles the translation itself, proposing as a diff you review before anything changes.

In order to build successful serverless migration tooling, we kept these critical principles in mind:

No broken production: a silent regression on a business-critical workload is worse than no migration at all, so the agent first assesses each workload and flags what is safe to move.

Control over every change: the ability to see what will change and what will not, to accept or reject each proposed edit, and to trust that nothing touches code you did not approve.

A clean revert: if the serverless run goes wrong, a way back to the pre-migration state in a few clicks.

Those three needs shape how the migration agent works, and the workflow below defines how it executes the migration: assess what can move, translate it, and apply the change reversibly.

What Genie Migrate for Serverless does

Assess: scan your jobs and classify a fleet so you know what can move today, what needs an admin, and what is not ready for migration yet.

Translate: propose the infrastructure and code changes a workload needs for serverless, each as a labeled diff (a side-by-side view of what changes and what stays the same) you review.

Apply (reversibly): validate the translation and apply it without preempting a running workload, keeping your original workload intact so you can revert if the serverless run is not healthy.

image2.png

The agent runs from the /compute migrate slash command inside Genie on any job or notebook. It is fully managed, and you need no setup to start scanning. Because Databricks manages it, the agent reads your job config and workload code inside your own workspace.

Let’s dive a bit deeper into each of these areas listed above.

Before you touch anything, the agent scans your Jobs and Notebooks and marks each workload with its serverless readiness. You see which workloads are ideal candidates for migration instead of auditing the fleet by hand - eliminating painstaking manual discovery and triage work

A quick note on external sources: if a workload's source of truth lives outside the workspace, for example a job managed by Databricks Asset Bundles or a notebook in a Git folder, the agent cannot migrate it yet. Support for those is coming soon.

Once you pick an eligible workload, the migration agent proposes the changes it needs for serverless and presents them for review before anything is written. Nothing is applied until you approve it.

The migration agent effectively translates the pieces that differ between classic and serverless:

Compute configuration: the classic cluster config maps to a serverless compute selection.

Environment and libraries: the library list is resolved against the serverless runtime, and the environment definition is synthesized for serverless.

Spark configurations: confs that still apply are carried over, and confs that do not are surfaced for you to decide on rather than removed silently.

Workload code: where the code uses patterns that do not run on serverless, such as APIs incompatible with Spark Connect, the agent rewrites them and shows the change as a diff.

Every change, including the code rewrites, is presented as a reviewable diff. Before applying, the agent shows a side-by-side comparison of what will change and what will not, an explicit contract you can hand to a manager or an on-call rotation.

Check out the latest demo on how to migrate classic workloads easily using Genie Migrate for Serverless.

If the agent is unsure about something, for example whether a Spark conf is still needed, it reports the conf rather than removing it. A conf can look unused to a static scan while a listener loaded at runtime depends on it. Silent removal would drop the telemetry a team relies on. The agent leaves that decision with you.

Applying a migration is where you are most exposed, so Genie Migrate for Serverless is most conservative here. It performs deterministic checks for correctness today, and deeper runtime validation is coming next.

Validation before Apply: the agent checks that the translation is coherent, that library dependencies resolve against the serverless runtime, and that the configuration applies cleanly, so that failures surface before the workload runs rather than during tomorrow's SLA.

A watched first run: because changes are applied to a cloned job, you can run the clone once on-demand to confirm it is healthy on serverless before you promote it onto your original job. The first serverless run then happens under your eye rather than unattended.

If you opt not to migrate, cloned jobs or suggested changes to notebooks will not be applied.

One thing to be cautious about: this does not undo data that serverless runs wrote to production tables in the meantime if you tested with the changed versions.

Today the validation is static. It confirms the translation is sound without running the workload, and it is honest about what it has and has not proven. The deeper check is coming next via Sandbox validation. With this, the agent will be able to run the migrated workload in an isolated sandbox without affecting your production tables and compare its output against the classic baseline – so you see a runtime and correctness signal on a critical job before Apply, not after.

Migrating one workload at a time in Genie works well for a handful of jobs. For a larger fleet, you can run Genie Migrate for Serverless in the background with the Databricks CLI. Start a migration for multiple jobs from a script using the compute-migrations command group, and the agent builds a serverless working copy of each job without an open Genie session. Your original job does not change until you review and approve its migration, and approving keeps the job's ID, schedule, and permissions. Bulk migration supports jobs today (support for notebooks coming soon) and requires Databricks CLI version 1.24.0 or above.

Let’s put the three key capabilities of the Genie Migrate for Serverless back together:

Assess: identifies and classifies the eligibility of jobs for serverless migration across the fleet.

Translate: proposes infrastructure and code changes as labeled, reviewable diffs.

Apply reversibly: performs deterministic validation today with sandbox validation coming next.

Here is a quick summary of what workloads are good fit today to migrate using the agent, and what’s coming in the future.

You can start moving workloads to serverless today using Genie Migrate for Serverless. Here is how to get going:

Turn on the beta: enable Genie Migrate for Serverless in your workspace preview settings, then run /compute migrate in Genie on a job or notebook to scan what is eligible.

Pick a low-risk workload first: migrate a non-critical job or notebook to build confidence in the flow before you touch anything business-critical.

Review before you apply: read the what-will-change and what-will-not panels, and accept or reject each proposed change.

Watch the first serverless run: confirm the migrated workload runs healthy; your original stays intact so you can revert if it does not.

Work outward from there: migrate the rest of the eligible fleet, and use the admin-action-needed banners to unblock the workloads that need setup.

The Genie Migrate for Serverless is available today via the /compute migrate command in Genie, on any job or notebook. We are actively shaping the sandbox-validation capability with preview customers, and we continue to evolve the agent to make serverless migrations faster, simpler and easier.

Serverless migration documentation - AWS, Azure, GCP

Connect to serverless compute | AWS | Azure | GCP

Get the latest posts in your inbox

原文为英文,正文由自动抓取并保留原文。

本文由采集器自动同步

原文发布于 Databricks Blog(Introducing Genie Migrate for Serverless),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗