A dermatologist tested two frontier LLMs on 200 biopsy-proven skin lesions. A 2015 CNN beat both
This is one bounded comparison, not a verdict on LLMs. But on a narrow, high-stakes task, a small model trained on the actual data beat the frontier defaults, and that is the case you want your own evals to catch.
The CNN was fine-tuned in twenty minutes on one H100, and it runs in a phone browser without uploading anything.
Dr Magnus Lynch compared GPT-6 Astra and Claude Fable 5.1 against a fine-tuned Inception v3. The CNN named the right diagnosis 62% of the time; the LLMs managed 42% each. Each had a favourite wrong answer: Astra over-called basal cell carcinoma, and Fable over-called actinic keratosis.
My read: this is one bounded comparison, not a verdict on LLMs. But on a narrow, high-stakes task, a small model trained on the actual data beat the frontier defaults, and that is the case you want your own evals to catch.
Your pick: before you wire a frontier API into a specialist workflow, try a small task model first. Benchmark both on your own data, then choose.
12 months: domain teams run specialist models beside one general LLM.
24 months: regulated fields default to auditable task models, with LLMs writing around them.
36 months: “which model” is a per-task routing call made on private evals.
Why I may be wrong: 200 lesions is one study in one specialty, and LLMs improve with better prompts and tools.
Where has a small model beaten an LLM on your task?