Findings from the $1M Vercel Sandbox challenge

$1M Vercel Sandbox 挑战赛成果 - Vercel

今年8月，我们启动了针对 Vercel Sandbox 的100万美元黑客挑战赛，邀请安全研究人员尝试突破 Vercel Sandbox。我们还与网络安全专家 Trail of Bits 协调，对源代码和安全架构进行了审查。

以下是挑战赛的总结：

• 两周内收到1285份报告
• 支付242,000美元漏洞赏金
• 发现一个Linux内核零日漏洞

所有报告均未显示任何人访问了 Vercel Sandbox 中的客户数据。最严重的发现是上游Linux内核中的缺陷。

复制链接到标题 内核零日漏洞

该计划最重要的报告在第三天就出现了。

HackerOne研究员 Paulos Yibelo（pwn.ai）在没有特殊访问权限的标准沙箱环境中，发现Linux内核网络栈存在缺陷。Paulos发现，从微虚拟机内部发送精心构造的数据包可以实现三种不同攻击：

• 通过网络泄露主机内存
• 导致主机崩溃
• 在主机上执行代码

#### 复制链接到标题 泄露主机内存

在第一种攻击中，主机内存（与您共享沙箱的其他租户共用的机器内存）会出现在客户发送到互联网的数据包中。只要客户持续发送数据包，这种泄露就可以持续发生。

该攻击已在Vercel生产环境沙箱中成功演示。

#### 复制链接到标题 导致主机崩溃和在主机上执行代码

研究人员还发现，通过发送特定数据包可以入侵主机。

首先，发送两个数据包即可使主机崩溃，关闭运行在该主机上的所有沙箱。其次，利用相同漏洞，研究人员能够在主机上以管理员权限运行命令，突破沙箱的隔离边界。

与数据包相关的崩溃和代码执行攻击已在研究人员自己的KVM和Firecracker机器上成功演示，这些机器的配置与Vercel Sandbox完全一致。

复制链接到标题 负责任地披露内核漏洞

几乎所有主要云服务提供商都在Linux内核的这一层隔离客户工作负载，因此任何缺陷都会构成共享基础设施风险。

在收到通知后，我们立即与研究人员合作，理解并缓解该漏洞，与AWS及其他行业合作伙伴协作优先修复漏洞，确保Vercel客户和其他使用该内核的开发者安全。

研究人员还联系了内核维护者，并与他们合作创建候选补丁。修复方案正在接受私密审查，CVE编号尚未确定，因此在补丁公开之前我们不会发布更多细节。我们披露该漏洞的存在是因为它已被公开披露。

Not every finding came from the outside. Marcos Hernanz, a Vercel intern, found a cross-tenant data leak in the Linux device-mapper storage layer. This vulnerability let any sandbox read recycled disk blocks from previously deleted sandboxes.

Copy link to heading The white-box deep dive

Trail of Bits spent four engineer-weeks inside our source code, with full access to the repositories and two enterprise test teams. They delivered 20 findings, and each was tested, confirmed, and referenced to the exact code.

They also performed an architectural review and concluded that our virtualization boundary is sound. Each sandbox runs on its own Firecracker microVM with a dedicated guest kernel, and the device model is deliberately tiny. No guest shares a running kernel with the host, or with another tenant.

On top of that foundation, they gave us three structural recommendations, which we implemented:

• Find and eliminate all trust in the guest. Our control plane accepted values returned by software running inside the microVM, and every one of those values is tenant input. Anything that crosses the boundary should be derived server-side or signed with a key the guest cannot reach.

• One policy, one decider. Egress enforcement was split across subsystems that each parsed the same network policy independently, and they did not always agree. We now use one canonicalization routine per data type, one component per decision, and revalidate connections when policy changes.

• Assume the next zero day. We strengthened the protections around each sandbox to limit the damage if an attacker exploits the virtualization software. We keep host kernels up to date, have disabled nested virtualization, and use Firecracker’s jailer to run each microVM process with restricted permissions and isolate it from the rest of the host.

Copy link to heading Vercel Sandbox findings

We received 1,285 reports in two weeks. This volume is only possible because researchers now hunt at AI speed.

Here is a breakdown of reports by category:

• 877 duplicates

• 319 not applicable

• 57 valid

• 32 informative

In our report triage we have validated:

• 1 Critical finding

• 5 High findings

• 9 Medium findings

• 42 Low findings

• 32 Informative findings

We have paid out $242,000 in bug bounties.

The medium, Low, and Informative findings are small policy-boundary gaps, like a firewall rule that behaved differently than documented and a field that trusted the guest more than it should.

The most severe finding was the Linux kernel zero day defect, and no reports showed anyone reading a customer’s files, secrets, or application data.

We were most impressed with the heterogeneity of the reports. It was clear that researchers used different setups and have different obsessions. Some clearly know Linux namespaces and steered their agents straight at the kernel boundary. Others went after the network policies, and many went after areas we didn’t think to put on the menu.

That variety is what you're funding with a public challenge. Here's one of the more interesting findings it produced.

Copy link to heading Cross-tenant disk snapshot restore (with limits)

Filed on day one of the challenge, this report is our favorite example of why we run a public bug bounty, even after internal assessments and a white-box engagement.

当对沙箱进行快照时，控制平面需要记录快照数据的存储位置：{存储地址、大小、校验和}。研究人员发现的漏洞非常隐蔽：这个信息来源于运行在客户机内部的软件，而运行在宿主机上的服务却信任了这个信息。这构成了信任边界违规。客户机是运行客户（以及黑客）不可信代码的地方，因此宿主机永远不能将其视为权威来源。

研究人员将已知的、有文档记录的容器逃逸原语与这个信任漏洞结合利用。他们在自己的沙箱中拦截了快照响应并篡改了内容，将两个不同账户所属的两个快照的存储对象进行了交换。这意味着恢复"你的"快照时，会将另一个账户的整个文件系统还原出来。

研究人员证明了每个账户都可以读取和修改其他账户的文件，并通过哈希验证了结果，同时保留了测试资源以便我们独立复现该发现。

此次攻击的限制条件具有重要意义。要针对特定目标实施交换，需要知道该目标的内部快照存储密钥——这是一个平台为每个快照随机生成且从不通过API暴露的96位标识符。该密钥只会在受害者自己的沙箱中出现。研究人员同时拥有两个账户，这使得证明成为可能；目前尚未发现陌生人针对其他租户的攻击路径。我们仍然将其视为优先级修复项，因为依赖秘密信息保密的信任边界本质上就不是真正的边界。

Trail of Bits在他们的评估中独立指出了相同的根本原因，这证实了我们的发现是真实、重要且可以从两个完全不同的角度被发现的。

修复方案在一个星期内上线，现在控制平面会将每个快照对象绑定到创建它的项目，并拒绝其他任何来源。快照创建操作已从客户机可访问的环境转移到宿主机服务，存储路径现在以项目为作用域，并使用严格验证的URL。

复制链接到标题 感谢研究社区

我们感谢每一位提交报告的研究人员，无论他们的发现是零日漏洞还是重复报告。挑战中发现的每种技术都影响着我们如何保护在Vercel Sandbox上运行的工作负载。

同时感谢Trail of Bits团队提供的全面内部调查和指导。

阅读原始挑战公告，了解Vercel Sandbox的更多信息，请查阅文档。