Training-Free Medical Image Segmentation vs Trained Networks

A multi-domain data-efficiency study — frozen LLM polygon traces + a numpy genome + a gated SAM decoder, with zero gradient training.

Pipeline

One image, end to end. A GT-free router selects skills per image; the drawer seam (amber) is the only model-coupled point — an LLM polygon drawer, a frozen SAM-feature prototype localizer, an AMG ranker, or a static box, per domain — and everything downstream is numpy/config. Green boxes (🔄) are iterative loops: SAM fixed-point re-prompts and the spleen two-pass within-volume prior.

Results — best trained NN across N vs the LLM pipeline

Strongest trained NN per cell — max of SegFormer (mit_b2), pretrained U-Net (resnet34), and nnU-Net, 2-seed means — at matched budgets N = 10 / 25 / 50 / 100, against the LLM pipeline (re-measured from scratch via eval_all_domains.py). The LLM's only label-fit module is a ~5–8-param genome; localization and SAM are GT-free/frozen — so the LLM number is ~flat in N while the NN climbs. Bold = LLM ahead of the best NN at that budget.

Data-efficiency

Qualitative examples — 3 best / 3 worst per dataset

Green = ground truth, red = prediction. Top row = 3 highest-Dice cases, bottom row = 3 lowest.

ISIC 2018 — skin lesion (pigment cue, LLM's best domain)

MSD Spleen — CT (localizer + cross-slice priors + SAM re-prompt polish)

Kvasir-SEG — polyp (LLM-box localizer; the gap was localization)

ChestXray — bilateral lungs (few-shot static box + self-reprompt → SAM)

What we learned

• ISIC: the LLM pipeline beats nnU-Net at every budget (+0.116 at N=10 → +0.025 at N=100) — genuinely more label-efficient for dermoscopy.
• Spleen: a stack of GT-free skills — within-volume prior, cross-slice geometry, selective multi-prototype, and a SAM re-prompt boundary polish (gate calibrated on train) — lifts 0.786 → 0.903, zero regressions each step, beating the best trained NN through N=50 and reaching parity at N=100 (0.903 vs 0.907).
• Kvasir: an LLM-box localizer lifts the polyp 0.348→0.633 (median 0.743) — the gap was localization, not a boundary ceiling, and it's solvable both coupled and agnostic. The deployed path has the vision LLM box the polyp per image (box-IoU 0.46, no leak) → SAM decode; a fully LLM-agnostic variant (SAM automatic-mask generator, oracle 0.798, + a frozen-feature ranker) reaches 0.617 (median 0.837) where hand-crafted colour cues (redness) failed. The LLM box wins the mean by +0.016 at a −0.094 median. Still below the NN at every budget (residual = boundary + localizer error), but no longer the "structural ceiling" it looked like.
• Chest X-ray: a few-shot static lung box + a mask-bbox self-reprompt + a repair agent (trims out-of-lung over-seg) + a catastrophic fix (SAM negative-point re-prompt on the worst spilled masks) + a letterbox trim (drops blobs over near-black padding) lift the LLM 0.825→0.898→0.911→0.914→0.920→0.922 — the gap was box localization, not architecture. But box-prompted frozen SAM stays architecturally capped — even a perfect GT box reaches only ~0.93 < the trained U-Net's 0.953, so CXR loses at every budget; the residual is the frozen decoder, not localization.

When does iterative feedback deploy? — the fix is general, the gate is the wall

We built a full repair-agent → verbal + visual feedback → SAM re-prompt → keep-best loop: the repair stage renders a diagnosis overlay (current mask + a GT-free "out-of-region" heat-map + a coordinate grid), the vision LLM looks, says what is wrong verbally, and marks the over-segmentation as SAM negative points; SAM re-decodes; a GT-free keep-best gate accepts or rejects; repeat. On chest X-ray the LLM's visual negatives beat the heuristic catastrophic fix (CHNCXR_0447 0.849→0.902, MCUCXR_0055 0.860→0.955).

But the fix is domain-general — the LLM can place negatives on any over-seg it sees. What decides deployability is the gate: whether a GT-free signal can rank candidate masks by true Dice. We measured that gate↔Dice correlation on all four domains: