Physix FrontierLLM Judges Fear What Wasn't Written Most
Just saw a paper update: LLM judges in clinical note evaluation are better at checking what is there, rather than what is missing. If you show it an AI-generated medical summary, it carefully checks if the symptoms, medications, and diagnoses written inside are wrong; but if a crucial allergy history, follow-up suggestion, or danger signal simply wasn't included, it easily treats it as if nothing happened.
Recently, my advisor pulled me in to try using LLMs for automated medical text evaluation. I casually built a small script with Claude Code and fed the original text into OpenSearch for retrieval-augmented generation. At first, I thought this job was like a reward model in reinforcement learning—letting the model score outputs to save manual labor. After running it, I found that the reward signal indeed rewards what is written, but barely penalizes what is not written. It rates a summary highly if it seems complete, even if it misses critical information.
I saw ComposoAI's accompanying paper and dataset, named OmissionBench, which specifically isolates omissions for testing. Clinical notes fear these omissions the most. If a sentence is wrong, a doctor can scan and correct it; if a sentence is missing, there's nothing in the record to correct. Especially regarding allergy histories, medication contraindications, and follow-up reminders, missing info is more dangerous than extra info.
So, don't just ask if this generated text has errors; ask what is missing compared to the original records. No matter how expert-like the judge is, it needs a set of questions capable of tracking omissions.
📌 This article is compiled from Hacker News. Original text: https://arxiv.org/abs/2608.31016
Copyright belongs to the original authors. This is a compilation and independent analysis based on public reports.