what did they actually measure?

14 Jun 2026 health, evaluation

a nature medicine paper has been showing up in my feed recently: the one claiming general-purpose llms beat specialized clinical tools (openevidence, uptodate) on medical benchmarks.¹ i am having a hard time letting it go, so i want to write down why, as plainly as i can. the short version: they say they are measuring how good these systems are for medicine, and what they are actually measuring is much narrower than that.

i will only talk about their own benchmark, the “real clinical queries” (RCQ) set. the other two stages do not interest me much.² RCQ is the part the authors call their “primary evidence” anyway, and the part that got picked up, so it is the one worth arguing with.

the headline is a head-to-head of six systems on 74 questions³, sampled from physicians’ queries to nyu langone’s internal hipaa-compliant gpt. so: things doctors (i hope!) typed into a chat box. the queries are de-identified, stripped of their context, and we never get to see them. the single example they do show is a request to “write a patient-friendly list of red flag symptoms and return precautions for back pain”, which is then scored on, among other things, “clarity for clinicians”.

that example does a lot of work (if you let it): it is a patient-handout task, really not the thing you use openevidence or uptodate to do. the job of a reference tool is to hand you the relevant guideline and tell you where it applies, especially when the question sits outside your own field, which is a good part of why clinicians reach for them at all. and citations, the actual surface of that job, were not evaluated. so the thing these tools exist to do is the thing the benchmark does not look at.

i wanted to show you one of the answers they marked as wrong, to make this concrete. there is not one in the paper. the only example response they print is a good one, there to illustrate the scoring screen. the errors live in a table of counts, and the data is sealed, so i cannot go and check. the primary evidence is judgments i cannot inspect, of answers i cannot read, to questions i cannot see: i imagine you see where this is going.

i guess the thing is, “general beat specialized” is not the interesting question. it assumes a single axis of competition. the question i actually care about is whether grounding, citations and auditability matter for these tasks, and whether anything here touches them. unfortunately, as far as i can tell, nothing does.

then there is the figure everyone shares (their 2c). it reads as a competence gap. but their own breakdown says the separation sits mostly in clarity, not correctness: they put the weakness down to communication rather than knowledge.⁴ on the axis that should matter most, whether the answer is right, the gap is small and the tools are fine. so the headline, read slowly, is closer to this:

clinicians from some field (we are not told which, and i am not even sure the raters are the authors, i am guessing) are handed a context-free question and answer from some other doctor in some other field, and find the chattier model easier to read.

i would argue the thing that looks like a deficiency for openevidence there, terse and citation-shaped instead of a skimmable essay, is closer to a feature, at least next to a generic google ai answer.

benchmarking ai for medicine is important: that is exactly why it is frustrating to watch it done this loosely and then amplified (in nature medicine, of all places). i do not have a clean benchmark to offer in return.⁵ i keep landing on a smaller question instead.⁶ what would a test that actually probed the thing these tools are for even look like? i do not think this one tried.