Researchers have introduced OpenTumorBoard, a benchmark designed to evaluate large language models on the real-world trajectories of multidisciplinary tumor board discussions. The dataset was transcribed from 219 publicly available tumor board recordings on YouTube, totaling 12,534 minutes of footage.
The benchmark comprises 611 patient cases, 19,157 discussion turns across ten specialist roles, and 16,215 questions posed to specialists during the meetings. It evaluates models in two settings: Specialist Turn, where a model answers a clinically significant question actually posed during a real discussion, and Board Simulation, where it generates an entire back-and-forth discussion and reaches a consensus on therapy recommendations, surgical plans, next actions, and clinical trial matching.
Evaluation of 14 general-purpose frontier and medical LLMs revealed substantial limitations. The best models scored 3.43 out of 5 in clinical equivalence to specialist answers and 2.78 out of 5 in alignment with recorded board conclusions. Supervised finetuning and reinforcement learning improved performance on a held-out test set, suggesting that real-world discussion trajectories can support model adaptation.
Three M.D. experts reviewed a subset of the benchmark, finding high information coverage and factuality of patient cases, along with strong fidelity of extracted consensus conclusions. The benchmark, its automated curation pipeline, dataset, leaderboard, and code are being released to support the development and evaluation of LLMs for multidisciplinary, personalized cancer decision-making.