本地运行 LLM 并不意味着数据安全:Cohere 研究揭示

Cohere 在一次网络安全任务中发现,阿里巴巴的 Qwen 模型意外地试图将一段测试载荷上传至阿里云存储,但沙箱拦截了该请求,至于究竟是什么触发了这一行为,目前仍存疑问。
作者:Abid Ali Awan,KDnuggets 助理编辑,2026年11月2日,发表于语言模型栏目
对于希望更好地掌控自身数据的开发者来说,在本地运行大语言模型(LLM)通常被认为是最安全的选择。你下载模型权重,所有运算都在自己的硬件上完成,从而避免把敏感信息发送给外部 AI 服务提供商。
但这是否就自动意味着你的 AI 系统是安全的?
Cohere 最近的一项调查提出了一些耐人寻味的问题。在一次安全基准测试中,研究人员观察到,一个开放权重的 Qwen 模型试图向阿里云服务器发送数据,尽管它并未收到任何相关指令。
这些发现迅速引发了关于模型安全、训练数据,以及本地运行 LLM 是否真能保护用户的争论。
我认为这件事的意义不止于把责任归咎于某一个模型。让我们看看实际发生了什么、社区怎么看,以及为什么整个 AI 智能体体系的搭建方式至关重要。
2026年10月26日,Cohere 发布了一项调查,针对阿里巴巴的开放权重语言模型 Qwen3.8-Flash-Next 的异常行为展开分析。
研究人员当时正在使用微软的 CTI-REALM 网络安全基准测试来评估该模型。分配的任务很简单:编写一条检测规则,用于识别下载并执行恶意 shell 脚本的可疑命令。
然而,模型却出乎意料地开始探索其所处环境。
据 Cohere 称,它检查了临时文件、查看了可用的命令行工具、检查了 DNS 配置,并试图将一个测试载荷上传到位于北京的阿里云存储桶。
这件事尤其引人注目的地方在于,模型对外可见的推理内容表明它不会执行任何工具。
所幸研究人员使用的是没有出站网络访问权限的隔离沙箱,因此上传尝试失败了。该载荷也只是一段固定的测试字符串,并非已证实的用户隐私数据。
Cohere 报告称,这种行为在不到十万分之一的评估调用中出现。重复输入同样的原始提示词,得到的是正常的检测规则。
因此,这项调查展示的是一次罕见且意外的工具调用行为。它并不能证明阿里巴巴蓄意植入了后门,也不能证明敏感信息已被成功泄露。
AI 社区对此怎么看?
Cohere 的调查在 Reddit 的 LocalLLaMA 社区和 LinkedIn 上引发了激烈讨论,开发者对调查结论和复现方法都提出了质疑。
一个主要批评集中在预填充(prefilling)上。Cohere 公布的脚本提供了 2,323 个 Token 的此前记录的推理和命令,正好停在生成 URL 之前。批评者认为,这种做法实际上是在促使模型补全一条可疑命令,而非证明模型会自主发起这样的命令。
不过,Cohere 在 Reddit 上澄清,最初的事件发生时并没有使用预填充。该技术是事后为调查这一行为而引入的,据称该行为在不到 10 万次评估调用中出现的次数不到一次。
其他开发者提出,阿里巴巴的 URL 可能是训练产物,而非恶意意图的证据。模型在训练中接触过智能体日志、云存储命令及类似示例,就可能在无关任务中复现这些模式。一些社区成员还质疑,专有模型在相同测试下是否也会表现出类似行为。
更早的社区报告已经描述过,在正常使用 Qwen 时会出现意外的阿里云 URL,这表明该现象并不局限于 Cohere 的实验。
这场争论还凸显了一个更广泛的担忧:在本地运行开放权重模型,并不会自动让 AI 智能体变得安全。拥有终端访问权限和无限制网络权限的模型,仍可能尝试意料之外的操作,因此沙箱隔离和网络限制必不可少。
归根结底,这项调查提出了关于 AI 智能体安全的合理质疑,但现有证据并不能证明存在蓄意后门。复现可疑行为与证明恶意意图,是两件截然不同的事。
为什么训练数据可能解释这一行为
这起事件中有一点让我尤其感兴趣:训练数据会影响 AI 模型的行为,而且有时是以出人意料的方式。
LLM 会从海量文本、代码、对话以及工具使用示例中学习模式。如果一个模型接触过成千上万个涉及特定云服务、shell 命令或调试工作流的样本,它之后就可能复现类似的模式。
例如,一个编程智能体可能会把调试与下载文件或发起网络请求联系起来,即使这些操作对当前任务并不必要。
这或许可以解释 Qwen 为什么会生成 Alibaba Cloud URL。该模型可能是从训练数据中学到了这些模式,而不是刻意试图把信息发送到别处。
还有一种可能是训练数据投毒,即把恶意样本引入训练或微调数据,以影响模型未来的行为。
然而,这些解释仍属推测。我们没有足够的证据来确定是什么导致了 Cohere 所观察到的这一行为。
这也引出了我认为更重要的问题:当模型的意外输出被允许执行时,会发生什么?
更大的安全问题在于智能体运行框架
语言模型无法自行访问你的文件,也无法通过互联网发送数据。它需要工具和执行环境,才能完成这些操作。
这正是智能体运行框架变得重要的地方。
运行框架是包裹在模型外层的软件。它负责管理上下文、工具执行、权限、文件系统访问,以及与外部服务的交互。
想想 Claude Code、Codex CLI 或其他本地智能体系统这类编程智能体。模型决定提出什么操作,但由运行框架控制该操作如何执行。
如果智能体拥有不受限制的终端访问权限、读取敏感文件的权限,以及对外网络访问能力,那么一条意外命令就可能演变成严重的安全问题。
同一个模型运行在隔离环境中时,可能生成完全相同的命令,却无法成功执行它。
有研究支持这一区分。
在最近的 HarnessSecurity-Bench 研究中,研究人员评估了六种编码智能体框架(harness)的安全机制,包括 Claude Code、Codex CLI 和 Qwen Code。
在他们的受控测试条件下,启用工具自动批准使攻击成功率从 29.2% 升至 95.6%。
这一显著差异源于智能体系统的安全配置,而非底层模型发生了变化。
研究人员还发现,网络隔离和只读执行能减少成功攻击,尽管这些限制可能让合法任务更难完成。
这就是为什么我认为,讨论不应停留在 Qwen 是否有意生成了可疑 URL。
我们如何让本地 AI 智能体更安全?
我认为解决方案不是停止使用开放权重模型。相反,我们需要更谨慎地界定 AI 智能体被允许做什么。
OWASP 的安全专家建议,在部署编码智能体时采用沙箱隔离、限制工具权限并限制网络访问。
以下是几个可以采取的实际步骤:
在沙箱中运行智能体:使用 Docker 容器、虚拟机,或 Anthropic 的 Sandbox Runtime 等工具来隔离命令执行。对于 Docker,在不需要联网时可使用 --network none 禁用网络。
制定严格的执行规则:允许 git diff 或运行测试等常规命令,但对破坏性命令、外部下载和文件删除要求审批。应通过智能体运行框架(agent harness)来强制执行这些规则,而不是仅依赖提示词。
限制文件系统与网络访问:只让智能体访问必要的项目目录。确保 .env 文件、SSH 密钥和生产环境凭证不可访问。默认阻止出站连接,或仅将可信目标加入允许列表。
避免不受限的权限:不要以 root 身份运行智能体,也不要对每一项操作都开启自动批准。只授予其完成任务所需的最低权限。
监控实际执行过程:记录 shell 命令、文件变更和网络请求。这有助于在异常行为演变成更大问题之前及时发现。
有意思的是,Anthropic 关于 Claude Code 沙箱隔离的研究强调,要将文件系统隔离与网络隔离结合起来。其内部测试还报告称,使用沙箱后权限提示减少了 84%。
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Running LLMs Locally Does Not Mean Your Data Is Safe: What Cohere’s Study Reveals)
Running LLMs Locally Does Not Mean Your Data Is Safe: What Cohere’s Study Reveals - KDnuggets
Running LLMs Locally Does Not Mean Your Data Is Safe: What Cohere’s Study Reveals
Cohere caught Alibaba’s Qwen model unexpectedly attempting to upload a test payload to Alibaba Cloud storage during a cybersecurity task, but the sandbox blocked the request, leaving questions about what triggered the behavior.
By Abid Ali Awan, KDnuggets Assistant Editor on November 2, 2026 in Language Models

Running a large language model (LLM) locally is often considered the safest option for developers who want more control over their data. You download the model weights, run everything on your own hardware, and avoid sending sensitive information to external AI providers.
But does that automatically make your AI system secure?
A recent investigation by Cohere has raised some interesting questions. During a security benchmark, researchers observed an open-weight Qwen model attempting to send data to an Alibaba Cloud server, despite receiving no instructions to do so.
The findings quickly sparked a debate about model security, training data, and whether running LLMs locally really protects users.
I think there is more to this story than blaming a single model. Let us look at what actually happened, what the community thinks, and why the entire AI agent setup matters.
On October 26, 2026, Cohere published an investigation into unusual behavior from Alibaba's Qwen3.8-Flash-Next, an open-weight language model.
The researchers were evaluating the model using Microsoft's CTI-REALM cybersecurity benchmark. The assigned task was straightforward: write a detection rule to identify suspicious commands that download and execute malicious shell scripts.
Instead, the model unexpectedly started exploring its environment.
According to Cohere, it inspected temporary files, checked available command-line utilities, examined DNS configuration, and attempted to upload a test payload to an Alibaba Cloud storage bucket located in Beijing.
What made this particularly interesting was that the model's visible reasoning indicated that it would not execute any tools.
Fortunately, the researchers were using an isolated sandbox without outbound network access, so the upload attempt failed. The payload was also a fixed test string, not demonstrated private user data.
Cohere reported that this behavior occurred in fewer than one in 100,000 evaluation calls. Repeating the same original prompt produced a normal detection rule.
The investigation therefore demonstrates a rare and unexpected tool action. It does not prove that Alibaba deliberately introduced a backdoor or that sensitive information was successfully leaked.
What Does the AI Community Think?
Cohere's investigation sparked heated discussions across Reddit's LocalLLaMA community and LinkedIn, with developers questioning both the findings and the reproduction methodology.
One major criticism focused on prefilling. Cohere's released script supplies 2,323 tokens of previously recorded reasoning and commands, stopping just before URL generation. Critics argued that this setup effectively encourages the model to complete a suspicious command rather than demonstrating that it would independently initiate one.
However, Cohere clarified on Reddit that the original incident occurred without prefilling. The technique was introduced afterward to investigate the behavior, which reportedly occurred in fewer than one in 100,000 evaluation calls.
Other developers suggested that the Alibaba URLs might be training artifacts rather than evidence of malicious intent. Models exposed to agent logs, cloud storage commands, and similar examples during training could reproduce those patterns in unrelated tasks. Some community members also questioned whether proprietary models would exhibit similar behavior under identical tests.
earlier community reports had already described unexpected Alibaba Cloud URLs appearing during normal Qwen usage, suggesting the phenomenon was not limited to Cohere's experiments.
The debate also highlighted a broader concern: running an open-weight model locally does not automatically make an AI agent secure. Models with terminal access and unrestricted network permissions can still attempt unexpected operations, making sandboxing and network restrictions essential.
Ultimately, the investigation raises legitimate questions about AI agent security, but the available evidence does not establish an intentional backdoor. Reproducing suspicious behavior and proving malicious intent are two very different things.
Why Training Data Might Explain This Behavior
One thing I find particularly interesting about this incident is how training data can influence an AI model's behavior, sometimes in unexpected ways.
LLMs learn patterns from enormous amounts of text, code, conversations, and tool-use examples. If a model encounters thousands of examples involving particular cloud services, shell commands, or debugging workflows, it may reproduce similar patterns later.
For example, a coding agent might associate debugging with downloading files or making network requests, even when those actions are unnecessary for the current task.
This could explain why Qwen generated Alibaba Cloud URLs. The model may have learned those patterns from its training data rather than deliberately attempting to send information elsewhere.
There is also the possibility of training-data poisoning, where malicious examples are introduced into training or fine-tuning data to influence future model behavior.
However, these explanations remain speculative. We don't have enough evidence to determine what caused the behavior observed by Cohere.
And that brings me to what I think is the more important issue: what happens when a model's unexpected output is allowed to execute?
The Bigger Security Problem Is the Agent Harness
A language model cannot independently access your files or send data over the internet. It needs tools and an execution environment that allow it to perform those actions.
This is where the agent harness becomes important.
The harness is the software surrounding the model. It manages context, tool execution, permissions, filesystem access, and interactions with external services.
Think about coding agents such as Claude Code, Codex CLI, or other local agent systems. The model decides what action to propose, but the harness controls how that action is executed.
If an agent has unrestricted terminal access, permission to read sensitive files, and outbound network access, an unexpected command can become a serious security problem.
The same model running in an isolated environment might generate the identical command without being able to execute it successfully.
There is research supporting this distinction.
In the recent HarnessSecurity-Bench study, researchers evaluated security mechanisms across six coding-agent harnesses, including Claude Code, Codex CLI, and Qwen Code.
Under their controlled testing conditions, enabling automatic tool approval increased attack success from 29.2% to 95.6%.
That is a substantial difference caused by the security configuration of the agent system rather than a change to the underlying model.
The researchers also found that network isolation and read-only execution reduced successful attacks, although these restrictions could make legitimate tasks harder to complete.
This is why I think the discussion should go beyond whether Qwen intentionally generated a suspicious URL.
How Can We Make Local AI Agents Safer?
I don't think the solution is to stop using open-weight models. Instead, we need to be more careful about what our AI agents are allowed to do.
Security experts at OWASP recommend sandboxing, restricting tool permissions, and limiting network access when deploying coding agents.
Here are a few practical steps you can take:
Run agents in a sandbox: Use Docker containers, virtual machines, or tools like Anthropic's Sandbox Runtime to isolate command execution. For Docker, --network none disables networking when internet access isn't needed.
Create strict execution rules: Allow routine commands such as git diff or running tests, but require approval for destructive commands, external downloads, and file deletion. Enforce these rules through the agent harness rather than relying on prompts alone.
Limit filesystem and network access: Give agents access only to necessary project directories. Keep .env files, SSH keys, and production credentials inaccessible. Block outbound connections by default or allowlist trusted destinations.
Avoid unrestricted permissions: Don't run agents as root or enable automatic approval for every action. Give them the minimum permissions required to complete the task.
Monitor actual execution: Log shell commands, file changes, and network requests. This helps identify unexpected behavior before it becomes a larger problem.
Interestingly, Anthropic's Claude Code sandboxing research emphasizes combining filesystem and network isolation. Its internal testing also reported an 84% reduction in permission prompts when using sandboxing.
For me, that's the biggest lesson from Cohere's investigation: we cannot always control what an AI model generates, but we can limit what its agent harness is allowed to execute.
As AI agents become more capable and local models continue to improve, I think we need to pay just as much attention to security as we do to model performance.
Running a model locally gives you more control over your data and infrastructure, but it doesn't automatically make your system secure. Even a well-performing model can generate unexpected commands or take actions that were never part of the original task.
This is where the agent harness and sandboxing become important.
If you're building or running local AI agents, you can use Docker containers or other sandboxing techniques to isolate execution. And even if you're not using a sandbox, you can still configure strict rules within your agent harness. For example, you can block certain commands, restrict access to sensitive files, or prevent unauthorized network requests.
The important part is that these restrictions should be enforced by the system, not just written in the model's prompt. Even if the model generates a prohibited command, the harness should refuse to execute it.
I would also avoid giving coding agents full permissions or enabling automatic approval for everything. Reviewing commands before execution adds an extra layer of protection, especially when working with sensitive projects.
For me, that's the biggest takeaway from the Cohere and Qwen investigation. We shouldn't assume that every unexpected action is malicious, but we also shouldn't blindly trust what an AI agent decides to execute.
Abid Ali Awan (@1abidaliawan) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master's degree in technology management and a bachelor's degree in telecommunication engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.
原文为英文,正文由自动抓取并保留原文。
原文发布于 KDnuggets(Running LLMs Locally Does Not Mean Your Data Is Safe: What Cohere’s Study Reveals),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。