Does MLLM localization make a frozen segmenter work on unseen medical images?

SAM-HQ, zero-tuning config, no target labels. n=50/domain (domain-max where smaller: ColonDB/ETIS 40, Prostate 30, Spleen 28, Heart 24; the L2 & L3 columns stay at n=25). Endoscopy·OOD / X-ray·OOD = cross-dataset generalization sets. L2+sr (outlined) — a GT-free self-reprompt — is the best rung on the colour/boundary-rich domains (Kvasir 0.59→0.64, ColonDB 0.70→0.75), but on the grayscale CT/MRI organ stacks at the bottom (Heart, Spleen, Liver, Pancreas) points & self-reprompt don't help — the plain box (L1) is best and the rows stay red because localization, not the decoder, is the wall (their oracles are 0.77–0.94). The router★ column is the one few-shot rung (~10 labels/domain): it routes per domain between the MLLM box and the op_sam correspondence localizer, recovering the grayscale organs (Spleen, Liver) and beating few-shot UniverSeg on average (see below). Two priors that don't help: the L3 judge loop (≈L2, below) and a coarse polygon mask prior fed to SAM (L2m: −0.14 on CXR — a mask prior helps only when it's SAM's own refined mask, i.e. L2+sr).

op_sam breaks the organ wall — Spleen 0.00→0.75, Liver 0.52→0.85, both beating UniverSeg — proving those were MLLM grayscale-localization failures, not task difficulty. But it's no universal winner: it fails on bilateral lungs (CXR/Montgomery — one correspondence peak can't grab two lungs), subtle flat polyps (ETIS 0.23), and tiny Pancreas, where the semantic MLLM box wins. The per-image oracle over the two frozen localizers is 0.757 (> UniverSeg 0.682). Realizing it splits by budget: a GT-free per-image router (route on op_sam's validated prior_iou) reaches only 0.664 — both localizers are confidently wrong in their failure modes (prior_iou is −0.12 anti-correlated with "op_sam is better"), a routing-gate wall. But a few labels/domain fix it — and adding SegGPT as a third complementary method lifts the routed result to 0.760, beating every few-shot baseline (see the method below).

Matched-budget fairness: at the SAME ~10-label budget, UniverSeg drops to 0.618 (N=10) from 0.682 (N=25), SegGPT is 0.666, op_sam 0.637. The 3-way router = 0.749 (per-image oracle 0.757) — beating every few-shot baseline even at matched budget, and its CI lower bound clears UniverSeg-N25. The three methods are genuinely complementary: SegGPT wins the lungs/fetal-head (Montgomery .96, CXR .93), op_sam the organs & clear polyps (Spleen .75, Liver .85, Kvasir .84), the MLLM box the subtle-flat ETIS / tiny Pancreas. Bold = best per domain. Honest caveats: the router is few-shot (~10 support + ~10 calibration labels ≈ 20), not label-free; and a purely GT-free per-image router does not realize this (routing-gate wall) — the few labels are what bank it.

Neither fully solves the failure (audit recovers 41%; the adapter trails on precise small targets) — but both are GT-free and neither lets the localizer judge itself, turning the paper's negative finding into a measurable recovery agenda.