Training-Free Medical Image Segmentation vs Trained Networks

A multi-domain data-efficiency study — frozen LLM polygon traces + a numpy genome + a gated SAM decoder, with zero gradient training.

Pipeline

One image, end to end. A GT-free router selects skills per image; the drawer seam (amber) is the only model-coupled point — an LLM polygon drawer, a frozen SAM-feature prototype localizer, an AMG ranker, or a static box, per domain — and everything downstream is numpy/config. Green boxes (🔄) are iterative loops: SAM fixed-point re-prompts and the spleen two-pass within-volume prior.

Results — best trained NN across N vs the LLM pipeline

Strongest trained NN per cell — max of SegFormer (mit_b2), pretrained U-Net (resnet34), and nnU-Net, 2-seed means — at matched budgets N = 10 / 25 / 50 / 100, against the LLM pipeline (re-measured from scratch via eval_all_domains.py). The LLM's only label-fit module is a ~5–8-param genome; localization and SAM are GT-free/frozen — so the LLM number is ~flat in N while the NN climbs. Bold = LLM ahead of the best NN at that budget.

Data-efficiency

Where is the wall — the decoder, or localization? A fair few-shot probe

Is the residual to the trained NN a decoder limit or a localization limit? MedSAM (SAM fine-tuned on ~1M medical masks) beats box-prompted SAM — but it is data-leaky (its training set likely includes these test distributions), so it is really "import a trained medical net." The fair, leak-free counterpart of the same operation: init from the generic SAM (no medical data), freeze the image + prompt encoders, and fine-tune only the ~4M-param mask decoder on our own N-label train split — GT never touches test. Given a good box (the GT-box oracle, which isolates the decoder), this few-shot-tuned decoder matches or beats the from-scratch NN at every budget on 3 of 4 domains, and it saturates at N=10 (pretrained features + a tiny adapted head), so its curve is flat-high while the NN climbs:

Is the medical pretraining (MedSAM) worth it? — a leaky benchmark

MedSAM is SAM already fine-tuned on ~1M medical masks — a leaky upper reference (its training likely saw these test distributions). Arch-matched (both vit_b, GT-box oracle, N=100), few-shot tuning of the GENERIC SAM matches or beats the leaky MedSAM on 3 of 4 domains: medical pretraining buys no consistent edge once the decoder is adapted, and where generic SAM is already strong (Kvasir/Spleen) it starts above MedSAM even frozen. Bold = better of the two few-shot columns.

Can it be ONE domain-agnostic pipeline? — few-shot skills + a router, SAM frozen

Each domain has a distinct setup — so can it be a single domain-agnostic pipeline that few-shot-fits per domain WITHOUT tuning SAM? We test the transferable skill — a frozen-SAM-feature localizer prototype fit from N labeled images — with identical code across all four domains, SAM never touched.

Qualitative examples — 3 best / 3 worst per dataset

Green = ground truth, red = prediction. Top row = 3 highest-Dice cases, bottom row = 3 lowest.

ISIC 2018 — skin lesion (pigment cue, LLM's best domain)

MSD Spleen — CT (localizer + cross-slice priors + SAM re-prompt polish)

Kvasir-SEG — polyp (LLM-box localizer; the gap was localization)

ChestXray — bilateral lungs (few-shot static box + self-reprompt → SAM)

What we learned

• ISIC: the LLM pipeline beats nnU-Net at every budget (+0.116 at N=10 → +0.025 at N=100) — genuinely more label-efficient for dermoscopy.
• Spleen: a stack of GT-free skills — within-volume prior, cross-slice geometry, selective multi-prototype, a SAM re-prompt boundary polish, and a pyramid keep-best (higher-resolution re-decode of the tiny tip slices) — lifts 0.786 → 0.920, now beating the best trained NN at N=100 (0.920 vs 0.907; CV held-out). The pyramid and the volume prior see the tip slices two different ways — resolution vs siblings — and combine additively via a valid soft-tissue gate.
• Kvasir: an LLM-box localizer + an AMG-candidate margin keep-best (re-localization proposals swapped in only on the clearest mislocalizations) lift the polyp 0.348→0.633→0.659 (median 0.743) — the gap was localization, not a boundary ceiling, and it's solvable both coupled and agnostic. The deployed path has the vision LLM box the polyp per image (box-IoU 0.46, no leak) → SAM decode; a fully LLM-agnostic variant (SAM automatic-mask generator, oracle 0.798, + a frozen-feature ranker) reaches 0.617 (median 0.837) where hand-crafted colour cues (redness) failed. The LLM box wins the mean by +0.016 at a −0.094 median. Still below the NN at every budget (residual = boundary + localizer error), but no longer the "structural ceiling" it looked like.
• Chest X-ray: a few-shot static lung box + a mask-bbox self-reprompt + a repair agent (trims out-of-lung over-seg) + a catastrophic fix (SAM negative-point re-prompt on the worst spilled masks) + a letterbox trim (drops blobs over near-black padding) lift the LLM 0.825→0.898→0.911→0.914→0.920→0.922 — the gap was box localization, not architecture. But box-prompted frozen SAM stays architecturally capped — even a perfect GT box reaches only ~0.93 < the trained U-Net's 0.953, so CXR loses at every budget; the residual is the frozen decoder, not localization.

When does iterative feedback deploy? — the fix is general, the gate is the wall

We built a full repair-agent → verbal + visual feedback → SAM re-prompt → keep-best loop: the repair stage renders a diagnosis overlay (current mask + a GT-free "out-of-region" heat-map + a coordinate grid), the vision LLM looks, says what is wrong verbally, and marks the over-segmentation as SAM negative points; SAM re-decodes; a GT-free keep-best gate accepts or rejects; repeat. On chest X-ray the LLM's visual negatives beat the heuristic catastrophic fix (CHNCXR_0447 0.849→0.902, MCUCXR_0055 0.860→0.955).

But the fix is domain-general — the LLM can place negatives on any over-seg it sees. What decides deployability is the gate: whether a GT-free signal can rank candidate masks by true Dice. We measured that gate↔Dice correlation on all four domains: