Training-Free Medical Image Segmentation vs Trained Networks
A multi-domain data-efficiency study — frozen LLM polygon traces + a numpy genome + a gated SAM decoder, with zero gradient training.
4 modalitiesmeasured, from scratch
GT-free at inferenceLLM-agnostic skills
updated 2026-07-02
Results — best trained NN across N vs the LLM pipeline
Strongest trained NN per cell — max of SegFormer (mit_b2), pretrained U-Net (resnet34), and nnU-Net, 2-seed means — at matched budgets N = 10 / 25 / 50 / 100, against the LLM pipeline (re-measured from scratch via eval_all_domains.py). The LLM's only label-fit module is a ~5–8-param genome; localization and SAM are GT-free/frozen — so the LLM number is ~flat in N while the NN climbs. Bold = LLM ahead of the best NN at that budget.
| Domain | best NN N=10 | N=25 | N=50 | N=100 | LLM pipeline |
|---|---|---|---|---|---|
| ISIC 2018 (derm) | 0.815 | 0.829 | 0.832 | 0.868 | 0.880 |
| MSD Spleen (CT) | 0.633 | 0.802 | 0.859 | 0.907 | 0.875 |
| Kvasir-SEG (endo) | 0.756 | 0.824 | 0.833 | 0.889 | 0.332 |
| ChestXray (X-ray) | 0.938 | 0.948 | 0.952 | 0.953 | 0.825 |
The LLM pipeline wins the scarce-label regime where it has a cue — ISIC through N=100 (best NN needs N=189 to reach 0.887), Spleen through N=50 (crossover ~N=70, via a GT-free cross-slice prior) — and loses at every budget on the no-cue ceilings: Kvasir polyp (structural — no GT-free cue localizes it) and Chest X-ray (box-prompted SAM caps at a GT-box 0.943 < 0.953). SegFormer is the strongest NN in 10 of 12 sweep cells.
Data-efficiency
Qualitative examples — 3 best / 3 worst per dataset
Green = ground truth, red = prediction. Top row = 3 highest-Dice cases, bottom row = 3 lowest.
ISIC 2018 — skin lesion (pigment cue, LLM's best domain)
MSD Spleen — CT (localizer + within-volume prior)
Kvasir-SEG — polyp (no GT-free cue: structural ceiling)
ChestXray — bilateral lungs (box-prompted SAM)
What we learned
- ISIC: the LLM pipeline beats nnU-Net at every budget (+0.116 at N=10 → +0.025 at N=100) — genuinely more label-efficient for dermoscopy.
- Spleen: a GT-free within-volume prior (re-decode mislocalized/over-seg CT slices from confident sibling slices) lifts 0.786 → 0.840, beating U-Net through N=50.
- Kvasir: honest 100-image Dice is 0.332 — no GT-free cue localizes the polyp (the 0.86 sometimes cited used GT-hand-redrawn images).
- Chest X-ray: box-prompted frozen SAM is architecturally capped — even a perfect GT box reaches only 0.943 < the trained U-Net's 0.953.
Honest-reporting note: earlier drafts carried optimistic hardcoded values (ISIC 0.885, Kvasir 0.860, CXR 0.855). This report uses numbers re-measured from scratch with the current pipeline: ISIC 0.880, Kvasir 0.332, CXR 0.825.