Training-Free Medical Image Segmentation vs Trained Networks
A multi-domain data-efficiency study — frozen LLM polygon traces + a numpy genome + a gated SAM decoder, with zero gradient training.
4 modalitiesmeasured, from scratch
GT-free at inferenceLLM-agnostic skills
updated 2026-07-02
Results — best trained NN across N vs the LLM pipeline
Strongest trained NN per cell — max of SegFormer (mit_b2), pretrained U-Net (resnet34), and nnU-Net, 2-seed means — at matched budgets N = 10 / 25 / 50 / 100, against the LLM pipeline (re-measured from scratch via eval_all_domains.py). The LLM's only label-fit module is a ~5–8-param genome; localization and SAM are GT-free/frozen — so the LLM number is ~flat in N while the NN climbs. Bold = LLM ahead of the best NN at that budget.
| Domain | best NN N=10 | N=25 | N=50 | N=100 | LLM pipeline |
|---|---|---|---|---|---|
| ISIC 2018 (derm) | 0.815 | 0.829 | 0.832 | 0.868 | 0.880 |
| MSD Spleen (CT) | 0.633 | 0.802 | 0.859 | 0.907 | 0.903 |
| Kvasir-SEG (endo) | 0.756 | 0.824 | 0.833 | 0.889 | 0.348 |
| ChestXray (X-ray) | 0.938 | 0.948 | 0.952 | 0.953 | 0.898 |
The LLM pipeline wins the scarce-label regime where it has a cue — ISIC through N=100 (best NN needs N=189 to reach 0.887), Spleen through N=50 (crossover ~N=70, via a GT-free cross-slice prior) — and loses at every budget on Kvasir polyp (structural — no GT-free cue localizes it) and Chest X-ray, where a few-shot static lung box (N=10 labels) lifts the LLM to 0.898 (box quality, not architecture, was the gap) but box-prompted SAM still caps at a GT-box ~0.93 < 0.953. SegFormer is the strongest NN in 10 of 12 sweep cells.
Data-efficiency
Qualitative examples — 3 best / 3 worst per dataset
Green = ground truth, red = prediction. Top row = 3 highest-Dice cases, bottom row = 3 lowest.
ISIC 2018 — skin lesion (pigment cue, LLM's best domain)
MSD Spleen — CT (localizer + cross-slice priors + SAM re-prompt polish)
Kvasir-SEG — polyp (no GT-free cue: structural ceiling)
ChestXray — bilateral lungs (few-shot static box → SAM)
What we learned
- ISIC: the LLM pipeline beats nnU-Net at every budget (+0.116 at N=10 → +0.025 at N=100) — genuinely more label-efficient for dermoscopy.
- Spleen: a stack of GT-free skills — within-volume prior, cross-slice geometry, selective multi-prototype, and a SAM re-prompt boundary polish (gate calibrated on train) — lifts 0.786 → 0.903, zero regressions each step, beating the best trained NN through N=50 and reaching parity at N=100 (0.903 vs 0.907).
- Kvasir: honest 100-image Dice is 0.348 (heuristic→SAM 0.332 + a wired train-calibrated SAM re-prompt boundary polish, +0.016, 0 regressions) — still no GT-free cue localizes the polyp (the 0.86 sometimes cited used GT-hand-redrawn images), so it loses to the NN at every budget.
- Chest X-ray: a few-shot static lung box (mean of N=10 train GT boxes) replaces the heuristic box and lifts the LLM 0.825→0.898 — the gap was box localization, not architecture (richer prompts, shape priors, and dark-field trims add nothing on top). But box-prompted frozen SAM stays architecturally capped — even a perfect GT box reaches only ~0.93 < the trained U-Net's 0.953, so CXR loses at every budget.
Honest-reporting note: earlier drafts carried optimistic hardcoded values (ISIC 0.885, Kvasir 0.860, CXR 0.855). This report uses numbers re-measured from scratch with the current pipeline: ISIC 0.880, Kvasir 0.348, CXR 0.898.