Training-Free Medical Image Segmentation vs Trained Networks

A multi-domain data-efficiency study — frozen LLM polygon traces + a numpy genome + a gated SAM decoder, with zero gradient training.

4 modalitiesmeasured, from scratch GT-free at inferenceLLM-agnostic skills updated 2026-07-03

Pipeline

One image, end to end. A GT-free router selects skills per image; the drawer seam (amber) is the only model-coupled point — an LLM polygon drawer, a frozen SAM-feature prototype localizer, an AMG ranker, or a static box, per domain — and everything downstream is numpy/config. Green boxes (🔄) are iterative loops: SAM fixed-point re-prompts and the spleen two-pass within-volume prior.

Results — best trained NN across N vs the LLM pipeline

Strongest trained NN per cell — max of SegFormer (mit_b2), pretrained U-Net (resnet34), and nnU-Net, 2-seed means — at matched budgets N = 10 / 25 / 50 / 100, against the LLM pipeline (re-measured from scratch via eval_all_domains.py). The LLM's only label-fit module is a ~5–8-param genome; localization and SAM are GT-free/frozen — so the LLM number is ~flat in N while the NN climbs. Bold = LLM ahead of the best NN at that budget.

Domain	best NN N=10	N=25	N=50	N=100	LLM pipeline
ISIC 2018 (derm)	0.815	0.829	0.832	0.868	0.880
MSD Spleen (CT)	0.633	0.802	0.859	0.907	0.903
Kvasir-SEG (endo)	0.756	0.824	0.833	0.889	0.633
ChestXray (X-ray)	0.938	0.948	0.952	0.953	0.922

The LLM pipeline wins the scarce-label regime where it has a cue — ISIC through N=100 (best NN needs N=189 to reach 0.887), Spleen through N=50 (crossover ~N=70, via a GT-free cross-slice prior) — and loses at every budget on Kvasir polyp (structural — no GT-free cue localizes it) and Chest X-ray, where a static lung box (N=10) + a self-reprompt + a repair agent + a catastrophic fix lift the LLM to 0.920 (localization, not architecture, was the gap) but box-prompted SAM still caps at a GT-box ~0.93 < 0.953. SegFormer is the strongest NN in 10 of 12 sweep cells.

Data-efficiency

Qualitative examples — 3 best / 3 worst per dataset

Green = ground truth, red = prediction. Top row = 3 highest-Dice cases, bottom row = 3 lowest.

ISIC 2018 — skin lesion (pigment cue, LLM's best domain)

MSD Spleen — CT (localizer + cross-slice priors + SAM re-prompt polish)

Spleen best/worst — The worst cases are localizer failures (prediction lands on stomach/liver, Dice≈0). A GT-free cross-slice prior (centroid trajectory + sibling-mask seed), a selective multi-prototype re-decode, and a SAM re-prompt boundary polish (kept only where it lands on stronger image edges) recover them, lifting the spleen pipeline to **0.903** (0.786→0.840→0.868→0.875→0.903 this session, zero regressions) — now level with the best trained NN at N=100.

Kvasir-SEG — polyp (LLM-box localizer; the gap was localization)

Kvasir best/worst — Polyps share colour/texture with mucosa, so the heuristic box and coordinate draws mislocalize (~43/100 to Dice≈0) and a hand-crafted redness box was rejected. Localization turns out to be solvable *two* ways. The deployed path is an **LLM-box localizer**: the vision LLM boxes the polyp per image (box-IoU 0.46 vs GT — genuine perception, no leak) → SAM decode, lifting the pipeline 0.348→**0.633** (median 0.743). A fully **LLM-agnostic** variant reaches 0.617 (median 0.837): SAM's automatic-mask generator proposes the polyp in *every* image (best-candidate oracle 0.798) and a few-shot ranker — standardized cosine-similarity of frozen SAM features to a polyp prototype + a train-fit size prior − a vignette penalty — picks it. The LLM box wins the mean by +0.016 (at a −0.094 median + model-coupling); either way the residual to the NN is the ~0.78–0.80 boundary oracle plus localizer error, not a perception ceiling. Gallery (LLM-box path): top row = hits (the box lands on the polyp → 0.98); bottom row = localization misses (Dice 0 — the ~13 cases where the box lands off the polyp), which is the whole residual.

ChestXray — bilateral lungs (few-shot static box + self-reprompt → SAM)

CXR best/worst — Five GT-free skills: (1) a fixed few-shot lung box (mean of N=10 train GT boxes; box-IoU 0.545→0.67) replaces the heuristic box → 0.825→0.898; (2) a mask-bbox self-reprompt (box-IoU 0.67→0.83) → 0.898→0.911; (3) a **repair agent** that trims SAM's bright "out of lung" over-seg into the mediastinum (a decoder error a redraw loop can't reach), fire-gated + keep-best → 0.911→0.914 zero-regression; (4) a **catastrophic fix** — a SAM negative-point re-prompt (positive points on the dark lung, negative on the spill) that carves the worst engulfing masks the conservative trim can't → 0.914→0.920; (5) a **letterbox-component trim** that drops mask blobs sitting over near-black padding (on letterboxed films the misregistered atlas + darkness both score the black border as lung — only raw intensity separates it, since lung air is dark-*gray*, not pure black), fixing MCUCXR_0060 0.76→**0.88** zero-regression → 0.920→**0.922**. The remaining worst cases (bottom row) are a different, harder failure — inferior over-extension into the abdomen, a decoder-boundary error. CXR's gap was box localization, not architecture — but a perfect GT box still caps SAM at ~0.93 < the U-Net's 0.953, so it stays capped at every budget; the residual is the frozen decoder.

What we learned

ISIC: the LLM pipeline beats nnU-Net at every budget (+0.116 at N=10 → +0.025 at N=100) — genuinely more label-efficient for dermoscopy.
Spleen: a stack of GT-free skills — within-volume prior, cross-slice geometry, selective multi-prototype, and a SAM re-prompt boundary polish (gate calibrated on train) — lifts 0.786 → 0.903, zero regressions each step, beating the best trained NN through N=50 and reaching parity at N=100 (0.903 vs 0.907).
Kvasir: an LLM-box localizer lifts the polyp 0.348→0.633 (median 0.743) — the gap was localization, not a boundary ceiling, and it's solvable both coupled and agnostic. The deployed path has the vision LLM box the polyp per image (box-IoU 0.46, no leak) → SAM decode; a fully LLM-agnostic variant (SAM automatic-mask generator, oracle 0.798, + a frozen-feature ranker) reaches 0.617 (median 0.837) where hand-crafted colour cues (redness) failed. The LLM box wins the mean by +0.016 at a −0.094 median. Still below the NN at every budget (residual = boundary + localizer error), but no longer the "structural ceiling" it looked like.
Chest X-ray: a few-shot static lung box + a mask-bbox self-reprompt + a repair agent (trims out-of-lung over-seg) + a catastrophic fix (SAM negative-point re-prompt on the worst spilled masks) + a letterbox trim (drops blobs over near-black padding) lift the LLM 0.825→0.898→0.911→0.914→0.920→0.922 — the gap was box localization, not architecture. But box-prompted frozen SAM stays architecturally capped — even a perfect GT box reaches only ~0.93 < the trained U-Net's 0.953, so CXR loses at every budget; the residual is the frozen decoder, not localization.

Honest-reporting note: earlier drafts carried optimistic hardcoded values (ISIC 0.885, Kvasir 0.860, CXR 0.855). This report uses numbers re-measured from scratch with the current pipeline: ISIC 0.880, Kvasir 0.617, CXR 0.920.

agenticSeg · training-free LLM+SAM medical segmentation · github.com/R9-Hu/agenticSeg
Skills are LLM-agnostic numpy/config; every result reproducible via eval_all_domains.py. AI-assisted (Claude Code).