Training-Free Medical Image Segmentation vs Trained Networks

A multi-domain data-efficiency study — frozen-foundation-model assembly (an LLM polygon drawer in the deployed path on 1/6 domains, else SAM / op_sam / SegGPT) + a numpy genome, zero gradient training on the target task.

Data-efficiency

Detailed data-efficiency figure — all cached U-Net / SegFormer / nnU-Net seeds + tuned-decoder oracle

Results — best trained NN across N vs the frozen-assembly pipeline

Strongest trained NN per cell — max of SegFormer (mit_b2), pretrained U-Net (resnet34), and nnU-Net using every currently cached seed — at matched budgets N = 10 / 25 / 50 / 100, against the frozen assembly. The point estimates below are current; seed coverage is not uniform. The data-efficiency figure above draws 95% seed intervals only for exact three-seed cells and marks one/two-seed cells as incomplete. Amber denotes a close point estimate, not a significance claim.

Latest Prostate experiments — Codex feedback + SegGPT K sweep / SkillOpt

These are sealed follow-up experiments, kept separate from the deployed six-domain table above. The NN-free Codex pilot uses 15 fixed slices; the frozen SegGPT⊕op_sam lane and K sweep use the 30-case prostate benchmark. Results from different protocols are not subtracted from one another.

Codex crop-zoom + SkillOpt feedback loop

Uncapping SegGPT K

We removed the implicit K=min(8,N) ceiling and reran the same full frozen pipeline on the same 30 held-out slices. K is the number of query-retrieved in-context support pairs, not gradient-training set size. Every run reads from the same pool of 100 leak-free TRAIN pairs, stored at 224×224. Raw joint inference fits through K=10 on a 12 GB TITAN Xp; raw K=12 and K=16 OOM. K>10 therefore uses a memory-safe approximation: feature-ensemble chunks of at most 8, followed by a support-count-weighted pixel vote. Two GPUs ran separate values/shards concurrently; they reduced wall time but did not combine VRAM.

Sealed K-router SkillOpt transfer test

The K sweep also has a GT-isolated router experiment (seggpt_k_skillopt.py, driven by experiments/seggpt_k_feedback.py). It fits on four validation volumes using leave-one-volume-out SkillOpt, freezes the resulting GT-free router, and then opens the six-volume test labels once for scoring.

Segmentation galleries

Per-case GT-vs-prediction overlays for the deployed pipeline on every test image (all 465, sorted worst-Dice first) — open the 6-domain gallery hub → (ISIC · Kvasir · Spleen · Prostate · CXR · FetalHead). Green = ground truth, red = deployed prediction; overlays rendered from the committed mask_cache/.

Where is the wall — the decoder, or localization? A fair few-shot probe

Is the residual to the trained NN a decoder limit or a localization limit? MedSAM (SAM fine-tuned on ~1M medical masks) beats box-prompted SAM — but it is data-leaky (its training set likely includes these test distributions), so it is really "import a trained medical net." The fair, leak-free counterpart of the same operation: init from the generic SAM (no medical data), freeze the image + prompt encoders, and fine-tune only the ~4M-param mask decoder on our own N-label train split — GT never touches test. Given a good box (the GT-box oracle, which isolates the decoder), this few-shot-tuned decoder beats the from-scratch NN at every budget on 4 of 6 domains (ISIC/Spleen/Kvasir/Prostate — Prostate by +0.12) and ties on the other two (CXR, HC18 — both near-ceiling NNs at ~0.95–0.96), and it saturates at N=10 (pretrained features + a tiny adapted head), so its curve is flat-high while the NN climbs:

Is the medical pretraining (MedSAM) worth it? — a leaky benchmark

MedSAM is SAM already fine-tuned on ~1M medical masks — a leaky upper reference (its training likely saw these test distributions). Arch-matched (both vit_b, GT-box oracle, N=100), few-shot tuning of the GENERIC SAM matches or beats the leaky MedSAM on 3 of 6 domains, and stays within 0.015 on the other 3. This table preserves that experiment's arch-matched NN reference; the current strongest cached-NN grid is the data-efficiency section above.

Can it be ONE domain-agnostic pipeline? — few-shot skills + a router, SAM frozen

Each domain has a distinct setup — so can it be a single domain-agnostic pipeline that few-shot-fits per domain WITHOUT tuning SAM? We test the transferable skill — a frozen-SAM-feature localizer prototype fit from N labeled images — with identical code across all six domains, SAM never touched.