The medical AI you paid for lost to the chatbot you already have
Here's an uncomfortable result for a booming industry: the AI tools built, marketed, and sold specifically for doctors just lost to the general chatbots anyone can open in a browser. Not by a hair, and not on a single test — across the board.
In a study published in Nature Medicine on 12 June 2026, independent researchers pitted two of the best-known specialist products, OpenEvidence and Wolters Kluwer's UpToDate Expert AI, against three frontier general-purpose models: GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6. They ran three tests: 500 US medical-licensing-style exam questions, 500 items from a clinician-alignment benchmark, and — most importantly — 100 real, anonymized questions physicians had actually typed during patient care, scored blind by 12 doctors.
The specialist tools lost every round. On the exam questions, Gemini hit 97.4% accuracy versus OpenEvidence's 89.6% and UpToDate's 88.4%. On the clinician benchmark the gap was a chasm: GPT scored 88 out of 100, the clinical tools around 62. And on the real-world physician queries — the part hardest to game — the three general models formed a clear top tier, while the purpose-built products landed with, and sometimes below, Google's automatic AI search summary. UpToDate's AI also simply refused to answer 19% of the questions.
Why would a tool designed for medicine lose to a generalist? The researchers' best guess is that the specialist products lean on retrieval — pulling in reference text to ground their answers — and that this can backfire when the retrieved material is irrelevant or clumsily stitched in. Frontier models, meanwhile, ride enormous training budgets, faster iteration and heavier alignment work. Raw scale and reasoning, it seems, can beat curated domain knowledge.
But the study earns credit for arguing against itself. The clinical tools were queried through their websites, not a clean API, which may have handicapped them. Two of the benchmarks were built by AI companies — including OpenAI, whose GPT-5.2 topped that particular test — so the deck may be subtly stacked. Crucially, no model was more likely to produce outright harmful or hallucinated content than another; the specialist tools weren't dangerous, just less useful. And on narrow sub-specialty tasks, tuned models may still win. The authors call their result "a snapshot of a rapidly evolving landscape," not a verdict.
So what? This is really about money and rules. Hospitals pay premiums for clinical AI partly for the reassurance of a medical brand. This study suggests that reassurance may be worth less than assumed — a question with real teeth for procurement, insurance reimbursement and regulators. The FDA still hasn't settled how it classifies general chatbots that dispense medical guidance without ever being cleared as medical devices, even as they outperform tools that were.
The takeaway isn't that ChatGPT should replace your doctor. It's that the "specialist premium" — in medicine and soon in law, finance and every field racing to sell domain-specific AI — can no longer be assumed. When the generalist keeps winning, "but ours is built for your industry" stops being a selling point and starts being a claim that has to be proven.