今天将分享手术外科视觉理解挑战赛完整实现版本为了方便大家学习理解整个流程将整个流程步骤进行了整理并给出详细的步骤结果。感兴趣的朋友赶紧动手试一试吧。一、SurgVU2026介绍SurgVU26SurgicalVisualUnderstanding2026是由 Intuitive Surgical 组织、作为 MICCAI 2026 内镜视觉挑战赛 系列的一部分在法国斯特拉斯堡举行的手术视觉理解竞赛是 SurgToolLoc2022–2023与 SurgVU2024–2025系列挑战赛的延续。其核心动机在于能够从内镜视频中检测并跟踪手术上下文Phase、Step、Task、Action 等层级与使用器械的机器学习模型将带来变革性的临床价值——例如自动化的手术进度评估、手术表现与效率分析、器械编排调度乃至手术室资源规划等应用。然而训练此类模型所需的数据获取成本极高需要对临床视频进行逐帧标注分割手术类别、框定器械边界框与关键点且随着术式与器械创新标注员还需持续培训。幸运的是在机器人辅助手术中器械的装卸信息可以从机器人系统中程序化采集为视频中的器械存在提供弱标注代理标准手术本体论的建立也进一步减轻了人工标注负担。因此本次挑战赛有两类以弱标注驱动强感知的任务一类利用工具存在弱标注训练器械定位模型另一类针对短视频片段生成视频问答答案。获胜方案将显著降低未来训练模型的标注负荷并广泛服务于临床应用。二、SurgVU2026任务挑战赛分为两个类别Category 1手术器械分类与定位。要求参赛队伍训练半监督/弱监督模型训练阶段仅使用噪标签性质的器械存在标签外加小规模验证集上的边界框标签测试阶段模型需对视频每一帧中出现的器械进行边界框定位与分类。Category 2手术视觉问答。同样属于弱监督范式要求模型基于 30 秒视频片段生成开放式问题的答案训练标签包括器械存在标签、手术步骤标签以及手术步骤类别的详细文字描述。Category 1 采用标准的 COCO 边界框检测指标 mAP[0.5:0.05:0.95] 作为排名依据。Category 2 对每个问题提供 5 条参考答案算法生成一条预测答案后评测容器基于 roberta-large 计算BERTScore-F1取与 5 条参考答案中的最高值作为该问题得分最后对所有问答对求均值。三、SurgVU2026数据集训练与测试数据来自 da Vinci 机器人系统的手术训练练习视频是 2022/2023 年 SurgToolLoc 挑战赛数据的扩展版并新增了Category 2所需的手术任务标签。训练数据规模280 段长视频来自 155 个训练 session原始采集为 60 FPS、720p1280×720单通道内镜画面总计超过 840 小时、约 1800 万帧。内容实习医生在猪模型等仿真场景下执行解剖组织、缝合、血管封闭等标准操作每段视频通常有 3 个机械臂器械装入视野可涉及 12 种器械存在少量稀有工具但测试集不会涉及。训练标签弱标签形式tools.csv器械存在标签记录每件工具安装/卸载的起止时间。主要标签为 groundtruth_toolname按工具功能归类的真值commercial_toolname 为商业化名称补充测试标签仅基于前者。由于标签直接从机器人系统数据导出存在噪声——例如已安装但暂未入镜的工具也会被标记为存在。tasks.csv手术任务标签提供每项手术任务的起止时间共 8 类任务Suturing缝合、Uterine Horn子宫角、Suspensory Ligaments悬韧带、Rectal Artery/Vein直肠血管、Skills Application技能应用、Range of Motion活动范围、Retraction and Collision Avoidance牵拉与避碰以及 Other其他未标注部分默认归入此类。Category 2 专属扩展新增 matched_description 字段为每类手术任务提供详细的自然语言描述涵盖任务过程、解剖结构、术者行为与所涉器械为视觉-语言模型提供更细粒度的训练信号。测试数据同样来自 da Vinci 训练练习视频时长可变统一以 1 Hz1 FPS采样。Category 1 测试集附带人工精标器械边界框由经验丰富的众包标注者完成框通常包围器械clevis器械关节旋转轴附近但单极弯剪、尖头有孔抓钳、吻合器、牵开抓钳与冲洗吸引器等器械的框会延伸至器械尖端测试阶段视频中的机器人 UI 会被模糊处理禁止利用 UI 信息进行预测。Category 2 测试集按 30 秒视频片段组织每个片段配有一组问答对每题附带 5 条参考答案。数据下载https://surgvu26.grand-challenge.org/data-download/四、技术路线任务1手术器械分类与定位在前面的文章说分享过相关文章可以参考这篇文章Endoscapes2024——用于手术腹腔镜视频数据解剖和工具分割、检测和 CVS评估1、图像预处理图像缩放到640x640然后采用均值为0方差为1的方式进行归一化处理并将数据划分成训练集和验证集。2、搭建YoloV5检测网络使用SGD优化器学习率是0.001batchsize是64epoch是500损失函数采用交叉熵和iou损失。3、训练结果和验证结果4、测试集结果已关注关注重播 分享 赞关闭观看更多更多退出全屏切换到竖屏全屏**退出全屏最新医学影像技术已关注分享视频时长02:540/000:00/02:54切换到横屏模式继续播放进度条百分之0播放00:00/02:5402:54倍速全屏倍速播放中0.5倍 0.75倍 1.0倍 1.5倍 2.0倍超清 流畅您的浏览器不支持 video 标签继续观看SurgVU2026——手术外科视觉理解挑战赛观看更多转载,SurgVU2026——手术外科视觉理解挑战赛最新医学影像技术已关注分享点赞在看已同步到看一看写下你的评论视频详情已关注关注重播 分享 赞关闭观看更多更多退出全屏切换到竖屏全屏**退出全屏最新医学影像技术已关注分享视频时长01:150/000:00/01:15切换到横屏模式继续播放进度条百分之0播放00:00/01:1501:15倍速全屏倍速播放中0.5倍 0.75倍 1.0倍 1.5倍 2.0倍超清 流畅您的浏览器不支持 video 标签继续观看SurgVU2026——手术外科视觉理解挑战赛观看更多转载,SurgVU2026——手术外科视觉理解挑战赛最新医学影像技术已关注分享点赞在看已同步到看一看写下你的评论视频详情任务2手术视觉问答MedGRPO 的核心在于对 GRPO (Group Relative Policy Optimization) 算法的医疗化改造跨数据集奖励归一化 (Reward Normalization)通过 Logistic 函数将不同任务的中位表现映射到统一的 0.5 奖励值平衡了不同数据集的权重有效防止了多任务强化学习中的训练不稳定。医疗 LLM 裁判 (Medical LLM Judge)引入 GPT-4.1 作为“专家裁判”从医疗术语准确性、器械识别、特异性描述、临床背景理解、动作准确度五个维度评分。这种评估方式比传统的自动指标更能真实反映模型的临床价值。MedGRPO 证明了通过结合大规模高质量数据集MedVidBench和优化的强化学习策略开源模型如 Qwen2.5-VL-7B在经过针对性训练后其在医疗视频理解任务上的表现可以显著超越现有的顶尖闭源模型。这为未来开发“临床级别”的手术辅助 AI 奠定了坚实的基础。项目地址https://github.com/UII-AI/MedGRPO-Code。具体可以参考这一篇MedGRPO2026——多任务强化学习在异构医疗视频理解中的应用[{id:case122,question:Are there forceps being used here?,prediction:No, the forceps do not appear to be actively used in this sequence; the grasper maintains consistent retraction of the gut throughout.,gt:[No, No, forceps are not mentioned., No forceps are being used., \No, theres no indication of forceps.\, No forceps are listed.]},{id:case123,question:Is a large needle driver among the listed tools?,prediction:No,gt:[No, No, a large needle driver is not listed., No large needle driver is included., \No, theres no large needle driver.\, \No, it doesnt appear a large needle driver is listed.\]},{id:case124,question:What type of forceps is mentioned?,prediction:The bipolar forceps are specifically mentioned as the instrument used for coagulation and tissue manipulation.,gt:[Cadiere Forceps, The type of forceps mentioned is Cadiere Forceps., Cadiere Forceps are the type mentioned., The forceps type is Cadiere Forceps., Cadiere Forceps is the specific type referenced.]},{id:case125,question:Is a suture required in this surgical step?,prediction:No.,gt:[Yes, Yes, sutures are required., Yes, a suture is necessary., Yes, the procedure involves sutures., Yes, suturing is part of the procedure.]},{id:case126,question:Was a large needle driver used in this clip?,prediction:No.,gt:[Yes, Yes, a large needle driver was utilized., Yes, it appears a large needle driver was used., Yes, a large needle driver was part of the procedure., Yes, the procedure involved a large needle driver.]},{id:case127,question:What organ is being manipulated?,prediction:The small intestine, which appears pale pink and is coiled within the abdominal cavity, is being manipulated by the instruments.,gt:[Uterine horn, The organ being manipulated is the uterine horn., Uterine horn is being manipulated., The manipulation involves the uterine horn., The organ in focus is the uterine horn.]},{id:case128,question:Is a needle driver involved in the procedure?,prediction:No, the needle driver does not appear to be actively engaged in the surgical maneuvers during this sequence.,gt:[Yes, Yes, a needle driver is involved., Yes, the procedure includes a needle driver., Yes, a needle driver is being used., Yes, needle driver is part of the procedure.]},{id:case129,question:What procedure is this summary describing?,prediction:The camera provides a clear view of the surgical field, with the grasper retracting the small intestine superiorly and laterally to maintain exposure, while the hook dissects along the serosal surface of the gut, progressively separating it from the surrounding tissue.,gt:[Endoscopic surgery or a laparoscopic surgery, The summary is describing endoscopic or laparoscopic surgery., This appears to be endoscopic or laparoscopic surgery., The procedure is likely endoscopic or laparoscopic surgery., \Its endoscopic surgery or laparoscopic surgery based on the summary.\]},{id:case130,question:What is the purpose of using forceps in this procedure?,prediction:The forceps are used to retract and maintain exposure of the surgical field, providing continuous retraction of the stomach and omentum to facilitate access to the operative area.,gt:[To grasp and hold tissues or objects during the surgery., The forceps are used for grasping and holding tissues or objects., Forceps are utilized to grasp and hold tissues or objects., The purpose of forceps is to grasp and hold tissues or objects., Forceps are used to grasp, hold, and manipulate tissues or objects.]},{id:case131,question:Is tissue being cut during this clip?,prediction:Yes, the electrosurgical hook systematically dissects and separates tissue, with intermittent repositioning for continued access.,gt:[Yes, Yes, tissue is being cut., Yes, the clip shows tissue being cut., Yes, cutting of tissue is occurring in this clip., Yes, tissue cutting is taking place.]},{id:case132,question:Was a large needle driver used during the surgery?,prediction:Yes, a large needle driver is used to grasp and pull suture material through tissue, with the suture line becoming more visible as the sequence progresses.,gt:[No, No, a large needle driver was not used., A large needle driver was not used., No large needle driver was utilized., There is no indication a large needle driver was used.]}]点击阅读原文可以访问参考项目如果大家觉得这个项目还不错希望大家给个Star并Fork可以让更多的人学习。如果有任何问题随时给我留言我会及时回复的。