LLM segmentation skills vs trained nets, at a handful of labels

Each domain: the best LLM-skill path (teal) against the strongest matched-N neural net found across seeds and architectures (clay). Ordered by margin. Bars start at 0.50 Dice.

The GPU-free classical decoder wins exactly where the target has an exploitable appearance cue, and needs a different decode skill where it doesn't (isodense CT → shape-prior; filaments → vesselness).

Two lines per domain across N=5/10/25/50/100: the deployed LLM / assembly (teal) and the strongest matched first-N net (clay) = max over best-of-breed SegFormer-b2 / U-Net-r34 and first-N nnU-Net, on the identical first-N images. Same numbers as the table and dataeff_new.png above (scratch_fusion/baselines_dataeff.json). Where the LLM leads, the gap is shaded. A net point is dropped where its matched baseline is deferred/absent (N=5 for Kvasir/Spleen/CXR/FetalHead; Prostate N=5 net = a degenerate single-volume first-N subset, omitted).

Every cell trains the net on the identical first-N sorted images the assembly uses. A subset bug was found and fixed: nnunet_lowdata._subset used a seeded shuffle, not first-N — now patched to first-N (GT-filtered, bit-for-bit verified, ids persisted per JSON). The strongest-NN per cell = max(best-of-breed matched SegFormer-b2 / U-Net-r34, first-N nnU-Net); LLM = the deployed assembly at that budget (fewshot_matched + N=100 deploy/fusion). Δ = LLM − strongest-NN. Methods note: the trained-NN baselines are NOT from-scratch — SegFormer (mit_b2) and U-Net (resnet34) use ImageNet-pretrained encoders; only nnU-Net is trained from scratch. So the LLM's scarce-N wins are against ImageNet-pretrained nets.

Every skill is data / numpy, LLM-agnostic, and SkillOpt-developed (rollout → reflect → strict-improve validation gate → converge). They deploy exactly where a valid GT-free signal exists.

• Corrected margins. The dramatic single-seed leads were an artifact; every number here is vs the strongest matched-N net (SegFormer/U-Net ImageNet-pretrained + nnU-Net from-scratch). At N=10 the net now wins FetalHead (nnU-Net 0.923 > 0.767) — the deployed SegGPT drawer still reaches 0.967 at its full K=8 support budget (−0.004 vs NN); and DRIVE is a tie (nnU-Net 0.772). The scarce-N LLM lead holds on ISIC / Prostate / Kvasir / Spleen.
• Nets are seed-noisy at low N (Spleen N=10: 0.547 ± 0.206), and no architecture dominates — mit_b3 wins FetalHead, UNet++ wins Kvasir/Spleen, unet-resnet34 wins Prostate. The skill paths are near-deterministic.
• Subset caveat. Feedback-loop & isodense numbers are on fixed 15–28-image test subsets (medians reported); net & deployed-pipeline numbers are full test.
• Running now: full-default nnU-Net (1000-epoch, @224, all 6 domains, ~25 h) to make the gold-standard comparison airtight — its reduced run already shows nnU-Net@10 weak on Kvasir (0.559) and only a tie on CXR; SegFormer N=100 fill; and the Synapse multi-organ CT run applying the shape-prior + bilateral-kidney multi-ROI to a fresh abdominal dataset.

The classical loop (Claude box → GrabCut / shape-prior / vesselness + numpy genome + judge — no SAM, no GPU) was pushed stage-by-stage on all six domains. It beats the net where a geometric/appearance cue exists; elsewhere every stage moved (+0.05–0.09) but hit a GT-box-oracle-proven wall that only the deployed GPU frozen-model lane clears.

The SAME LLM-agnostic pipeline, but the drawer/decoder seam is a frozen neural backend (op_sam / SegGPT / SAM) — no gradient training on the target task (one few-shot alt lane, noted). Frozen models + a few labeled support images = the same few-label budget as the genome fit. This lane closes the perception-wall domains the classical [cls] lane can't reach, to ≈NN — training-free. The [cls] deployed column is carried alongside so the two lanes read as a pair.

Six feedback-loop agents each targeted the three lowest-Dice test images of one domain. Four were fixed and land at a different pipeline stage each — the drawer (Kvasir re-box), a cross-slice volume gate (Prostate), and two mask refines (ISIC grow, FetalHead robust ellipse). For those four the montage is before→after: green = GT, red = pre-fix prediction, blue = post-fix. A fifth — Spleen — was then recovered by a NEW GT-free texture-homogeneity discriminant (drift zeros 5→1); only CXR remains a clean perception wall (worst-3 GT-vs-prediction overlay).

The same 6 domains, attributed with an identical structured failure decomposition (an S sub-skill breakdown → a Linker that names the responsible sub-skill → a two-walls verdict), run on both deployed lanes: the NN-free classical drawer [cls] and the frozen-NN backend [gpu] (op_sam / SegGPT / SAM). This is the evidence map — where and why each lane fails, and what the gap between them reveals.

• S_model_selection is the frozen-NN lane's signature sub-skill — the Linker's top cause on 4/6 domains — and it does not exist on the classical single-drawer lane (never the argmax there). Choosing the frozen backend (op_sam vs SegGPT vs SAM vs few-shot) IS a procedural skill, and it is where the frozen lane's residual failures concentrate — FetalHead SegGPT support-dependence: the in-context drawer draws NOTHING on 3/22 held-out → dice 0.0, unrecoverable by any refine/QC edit.
• Backend swap dissolves ~half the classical walls; the other half persist. CXR (soft-border), Spleen (isodense location) and Kvasir-localization DISSOLVE (the NN-free escape grabcut→SegGPT / →op_sam was correct); ISIC-refinement, Kvasir-grow (polyp≈mucosa) and Prostate-isodense-residual PERSIST across BOTH lanes. The NN-free lane's cross_lane_escape field predicted which would dissolve.
• failure_type ↔ wall is a DOMAIN property, not a type property. boundary_drift DEPLOYS on Spleen (edges are anatomy, +0.027) yet is gate-bound on Kvasir/CXR; the same F row resolves oppositely by domain. So the two-walls law is read off the domain, not the failure label.

DRIVE (12 train / 6 test) is the OOD domain — a thin whole-image vessel tree, a hard fit for every method. 3-seed, first-N, 224 Dice, same protocol as the 6 domains but reported separately.

The most label- and compute-lean lane touches no GPU and no neural net at all: Claude traces a polygon, a numpy genome refines it, and an optional crop-zoom re-draws small targets. n≈20/domain.

The fair "does a trained net help?" control: few-shot fine-tune the ACTUAL frozen backbone the pipeline uses — SAM-HQ vit_l — on N labels (mask_decoder only, encoder+prompt frozen, leak-free), then deploy it (decode from the deployed localizer box, NOT the GT box). Compare to the training-free deployed pipeline.