AI 助力研究数千项医学研究的代码共享

支撑一项研究的数据和代码可以告诉我们实际做了什么,以及是如何做的。例如,在一项医学研究中,患者是如何被选择纳入的?缺失值是如何处理的?结果是如何评估的?
能够获取一项研究的数据和代码,有助于研究人员回答这些问题、审视其方法,并在此基础上继续开展工作。
“研究涉及许多关于如何准备数据、开发模型和评估性能的选择。这些有时被称为‘研究者的自由度’,”MIT 计算生理学实验室与计算机科学与人工智能实验室的研究科学家 Tom Pollard 说。“我们做出的选择会影响结果,而论文中的方法部分只能记录这么多细节。”
虽然共享代码可以是第一步,但要让代码能够被理解和复用,它必须得到充分的结构化组织和描述。问题在于,目前几乎没有指导研究人员何为组织良好、描述充分的代码集合。
这正是由 MIT 和 Dana-Farber Cancer Institute 的研究人员领导的一个国际团队试图提供帮助的地方:他们考察了临床预测研究中目前如何共享代码。这类研究开发或评估模型,以估计患者结局,例如患上某种疾病的风险。由于这些模型会为患者护理决策提供依据,研究人员需要能够仔细审查它们是如何构建和测试的。
研究人员使用大型语言模型(LLM),考察了近 4,000 篇论文,以识别关于共享代码的陈述。随后,他们顺着这些陈述评估了数百个相关代码仓库。
他们的综述揭示了两大关键缺口:代码共享仍不普遍,而将代码公开也并不一定意味着提供了复用所需的文档和配套信息。这项开放获取的研究成果最近发表在 Nature Medicine 上,正在为 TRIPOD-Code 提供参考——这是一项正在制定的报告指南,旨在明确研究人员应当共享哪些内容。
追踪代码
TRIPOD-Code 建立在 TRIPOD 和 TRIPOD+AI 之上,这两份清单帮助研究人员报告临床预测模型的开发与评估方式。为了解当前的代码共享实践,研究团队筛选出了引用这两份指南中任一份的研究。该综述纳入了近 4,000 项研究,这些研究开发、更新或验证了预测模型,且其全文可通过 PubMed Central 的开放获取合集获取。
“挑战在于这些信息高度不一致。首先,我们必须判断一篇论文是否共享了代码,”该研究的第一作者 Thomas Sounack 说。他还补充说,作者们“会以许多不同的方式讨论并链接论文中的代码”。
团队使用 LLM 帮助筛选论文、识别有关代码可用性的陈述,并提取链接。随后,软件抓取所链接的代码仓库——即代码及相关文件的在线集合——以便进一步评估。
该工作流程不止于检查链接是否存在。它还依据文档完整性、软件运行要求和复用条款等特征对代码仓库进行评估。
“LLM 真正的价值在于其处理文本等非结构化信息的能力。这些模型帮助我们处理那些散落在论文和软件仓库中、而非整齐排列在表格里的信息,”该研究的高级作者 Pollard 说。“这让我们能够更细致地考察大量研究中代码是如何被共享的。”
人工评审员对 LLM 的工作进行了评估。两名研究人员独立核查了 500 篇论文以及另一组仓库样本,先解决分歧,再将他们的评估与自动化结果进行比较。
在具体特征上——例如是否存在许可证文件——双方的一致性高于对文档是否充分的判断。由于分类错误仍有可能出现,作者强调总体规律,而不把每一项计数都视为精确无误。
代码共享情况不一致
只有约八分之一的研究报告了共享代码。在共享代码的情况下,仓库内容差异很大。
Sounack 说,一些仓库“组织得极为有序、文档也极为完善,而另一些则几乎没有任何说明”。他解释说,良好的代码共享意味着“提供足够的信息,让人了解代码的功能,并且能够运行它来复现论文中报告的结果”。
不同期刊和国家的做法也差异很大。在各期刊之间,共享代码的研究比例从 0 到超过 70% 不等。一些共享率较低的期刊其实也制定了书面的代码共享政策,这表明仅靠政策可能不足以推动代码共享。
这次综述还帮助识别出了一些有效实践的例子。例如,一些研究人员在原始患者记录无法公开共享的情况下,提供了合成数据集来帮助代码复用。
为代码共享设定标准
共享代码的研究人员已经在努力让自己的工作保持透明。但临床预测研究汇集了来自医学、统计学和计算机科学领域的人,他们对代码记录和共享的期望可能各不相同。
TRIPOD-Code 旨在建立共识。通过专家评审流程,团队正在制定指南,明确共享的分析代码中哪些要素应被视为必不可少。
“我们希望为研究人员提供如何共享代码的实用指南,也让期刊有更清晰的期望可以传达给作者,”Pollard 说。“这次综述为我们提供了证据,为这些讨论提供依据。”
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(AI helps researchers investigate how thousands of medical studies share code)
The data and code that underpin a research study can tell us what was actually done, and how. In a medical study, for example, how were patients selected for inclusion? How were missing values handled? How were the results evaluated?
Access to a study's data and code help researchers to answer these questions, to examine the methods, and to build on the work.
“Studies involve many choices about how to prepare data, develop models, and evaluate performance. These are sometimes called ‘researcher degrees of freedom,’” says Tom Pollard, a research scientist at MIT's Laboratory for Computational Physiology and Computer Science and Artificial Intelligence Laboratory. “The choices we make affect the results, and a methods section in a paper can only capture so much detail.”
While sharing code can be a first step, in order for code to be understood and reused, it must be sufficiently structured and described. The problem is that, right now, there is little guidance for researchers on what constitutes a well-organized, well-described collection of code.
This is where an international team led by researchers at MIT and Dana-Farber Cancer Institute sought to help, by examining how code is currently shared in clinical prediction studies. Such studies develop or evaluate models to estimate patient outcomes, such as the risk of developing a disease. Because these models inform decisions about patient care, researchers need to be able to scrutinize how they were built and tested.
Using large language models (LLMs), the researchers examined nearly 4,000 papers to identify statements about shared code. They then followed the statements to assess hundreds of associated code repositories.
Their review revealed two key gaps: Code sharing remains uncommon, and making code public does not consistently mean providing the documentation and supporting information needed for reuse. The open-access findings, recently published in Nature Medicine, are informing TRIPOD-Code, a reporting guideline being developed to clarify what researchers should share.
Following the code
TRIPOD-Code builds on TRIPOD and TRIPOD+AI, checklists that help researchers report how clinical prediction models were developed and evaluated. To understand current code-sharing practices, the team identified studies citing either guideline. The review included almost 4,000 studies that developed, updated, or validated prediction models, and whose full texts were available through PubMed Central’s open-access collection.
“The challenge was that the information was highly inconsistent. First, we had to identify whether or not a paper shared code” says Thomas Sounack, the study’s first author. He adds that authors “discuss and link to code from a paper in many different ways.”
The team used an LLM to help screen papers, identify statements about code availability, and extract links. Software then retrieved the linked repositories — online collections of code and related files — for further assessment.
The workflow went beyond checking whether a link existed. It also assessed repositories against characteristics such as completeness of documentation, software requirements, and terms of reuse.
“The real benefit of LLMs is their ability to process unstructured information such as text. These models helped us work with information that was scattered across papers and software repositories, rather than neatly organized in a table,” says Pollard, who is a senior author of the study. “That allowed us to examine how code was being shared in much more detail across a large collection of studies.”
Human reviewers evaluated the work of the LLMs. Two researchers independently checked 500 papers and a separate sample of repositories, resolving disagreements before comparing their assessments with the automated results.
Agreement was stronger for concrete characteristics, such as whether a license file was present, than for judgments about the adequacy of documentation. Because classification errors remain possible, the authors emphasize the overall patterns rather than treating every count as exact.
Inconsistent code sharing
Only about 1 in 8 studies reported sharing code. Where code was shared, the contents of repositories varied widely.
Some repositories were “extremely well-organized and -documented, while others gave almost no guidance at all,” Sounack says. Good code sharing, he explains, means “giving enough information to understand what the code does and be able to run it to reproduce the results reported in the paper.”
Practices also varied substantially across journals and countries. Among journals, the proportion of studies sharing code ranged from zero to more than 70 percent. Some journals with low sharing rates nevertheless had written code-sharing policies, suggesting that policies alone may not be enough to motivate code sharing.
The review also helped to identify examples of effective practice. Some researchers, for example, provided synthetic datasets to assist with code reuse where the original patient records could not be shared publicly.
Setting a standard for code sharing
Researchers who share code are already making an effort to make their work transparent. But clinical prediction research brings together people from medicine, statistics, and computer science, whose expectations for documenting and sharing code may differ.
TRIPOD-Code aims to establish a consensus. Through an expert review process, the team is developing guidance on which elements of shared analytical code should be considered essential.
“We want to give researchers practical guidance on how to share code, and give journals clearer expectations to communicate to authors,” Pollard says. “The review gives us evidence to inform those discussions."
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 MIT News AI(AI helps researchers investigate how thousands of medical studies share code),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。