agenticSeg ยท data-efficiency study ยท 2026-07-13

LLM segmentation skills vs trained nets, at a handful of labels

A frozen perception model traces the target; numpy-only skills โ€” a genome, a shape-prior, a vesselness decoder โ€” refine it. No neural net is trained. The question: at N = 5โ€“100 labels, can these skills match a net trained on the same images? Tested against the strongest matched-N baseline (multi-seed SegFormer / UNet++ / nnU-Net), the honest answer is a modest-but-real low-label lead on most domains โ€” and a decisive win where nets need the most data.

5โ€“6 / 7
domains where an LLM skill beats the best matched-N net @ N=10
+0.15
DRIVE vessels โ€” largest lead, where a trained net fails hardest at low N
ยฑ0.21
NN seed-noise at N=10 (Spleen); the skill paths are near-deterministic
0.002โ†’0.52
isodense CT: shape-prior decoder vs GrabCut on the invisible boundary
Full technical report โ†’
head-to-head

Dice at N = 10 labels โ€” skill vs the strongest net

Each domain: the best LLM-skill path (teal) against the strongest matched-N neural net found across seeds and architectures (clay). Ordered by margin. Bars start at 0.50 Dice.

LLM skill (best path) strongest matched-N net ฮ” label = skill โˆ’ net
Read it honestly: earlier single-seed baselines inflated these margins (FetalHead once read +0.34; a properly-chosen net gets 0.816, so the real lead is +0.105). The story survived the correction โ€” it just got modest. CXR is the one clean loss (the net is already 0.921). Matched-N protocol: at each budget both net and skill calibrate on the identical first-N images (verified bit-for-bit); an ISIC id-source divergence was caught and corrected (0.926โ†’0.923, โˆ’0.004, verdict unchanged).
by modality & contrast

Where the skills win is set by the target's visual cue

The GPU-free classical decoder wins exactly where the target has an exploitable appearance cue, and needs a different decode skill where it doesn't (isodense CT โ†’ shape-prior; filaments โ†’ vesselness).

DomainModalityTarget contrast cue best net @10LLM skill @10 ฮ”skill path
Spleen is mixed, not a loss: the deployed skill's mean (0.660) trails the net's lucky best-seed (0.714) but beats its mean (0.547 ยฑ 0.206); its median is 0.929. The net is wildly seed-unstable here; the skill isn't.
data-efficiency ยท N = 10 / 25 / 50

Skill flat, net climbing โ€” the crossover budget is the whole story

The skill line is near-constant in N (the drawer is GT-free; only a light genome uses labels, saturating by Nโ‰ˆ10) while the net climbs with more labels. Where the skill leads at N=10 it holds through N=25/50 until the net catches up โ€” the crossover is the honest measure of the low-label edge. Shaded = skill ahead.

LLM skill matched-N net (arch-max) shaded = skill leads
Crossover per domain: ISIC & FetalHead โ€” skill leads through N=50 (net never catches in range). Prostate โ€” skill wins N=10, net passes by N=25. Kvasir โ€” skill (deployed op_sam) leads to Nโ‰ˆ50, net ties at N=50. Spleen โ€” net's mean passes by N=25, but the skill median (0.93) stays ahead. CXR โ€” net leads throughout. Spleen shown as deployed mean; DRIVE (N=5/10 only, 12-image train pool) is the standout not plotted here: skill 0.773 vs net 0.623.
the skills

Four transferable decode skills โ€” and where each deploys

Every skill is data / numpy, LLM-agnostic, and SkillOpt-developed (rollout โ†’ reflect โ†’ strict-improve validation gate โ†’ converge). They deploy exactly where a valid GT-free signal exists.

GPU-free feedback loop

Draw โ†’ GrabCut โ†’ genome โ†’ judge

0.93 ISIC ยท FetalHead

Claude draws (GT-free), classical GrabCut + numpy genome decodes, a visual judge sends verbal feedback back to the skills. Wins the appearance-cue domains at low N.

โœ“ deployed ยท beats net at every Nโ‰ค50 on wins
isodense shape-prior

Template placed by a spatial prior

0.002 โ†’ 0.52

On isodense CT the boundary is invisible โ€” GrabCut carves noise (0.002). A mean-mask template placed by a GT-free localizer recovers it (~250ร—). The wall is localizing the unseen edge, not the prior (oracle @GT-box 0.84 > net).

โ—‘ localization-bound below net
vesselness ยท learnable

Frangi โ†’ shallow RF decode

0.773 +0.15 vs net

Thin retinal vessels can't be boxed. A ridge-filter stack feeds a shallow RandomForest (โ‰ค10 labels, CPU). Beats a matched-N net decisively โ€” the extreme low-data case where nets fail hardest.

โœ“ largest, most robust lead in the study
decode-router

Route by a GT-free cue

isodense โ†’ prior

A two-scalar cue (saturation + within-box Otsu separation) routes appearance targets โ†’ GrabCut and isodense CT โ†’ the shape-prior. Integrates the isodense skill automatically (Spleen 0.002 โ†’ 0.438 from routing alone).

โœ“ deployed ยท correct on every domain
The recurring lesson โ€” and the honest negatives. Three "push the limit" experiments were run: multiscale (gate-bound, no valid keep-best proxy), mask-refine (mixed โ€” snake grow lifts Prostate +0.028, guided lifts ISIC, but Kvasir/CXR gate-bound), and multi-ROI for CXR's two lungs (fixes the bilateral bookkeeping but +0.003 on test โ€” doesn't close the soft-diaphragm wall). Across all of them: the bottleneck is the GT-free gate / perception, not the fix. A skill deploys iff a valid signal exists.
rigor & status

Caveats kept in view, and what's still running

  • Corrected margins. The dramatic single-seed leads were an artifact; every number here is vs the arch-max best matched-N net. FetalHead: +0.338 โ†’ +0.105.
  • Nets are seed-noisy at low N (Spleen N=10: 0.547 ยฑ 0.206), and no architecture dominates โ€” mit_b3 wins FetalHead, UNet++ wins Kvasir/Spleen, unet-resnet34 wins Prostate. The skill paths are near-deterministic.
  • Subset caveat. Feedback-loop & isodense numbers are on fixed 15โ€“28-image test subsets (medians reported); net & deployed-pipeline numbers are full test.
  • Running now: full-default nnU-Net (1000-epoch, @224, all 6 domains, ~25 h) to make the gold-standard comparison airtight โ€” its reduced run already shows nnU-Net@10 weak on Kvasir (0.559) and only a tie on CXR; SegFormer N=100 fill; and the Synapse multi-organ CT run applying the shape-prior + bilateral-kidney multi-ROI to a fresh abdominal dataset.
gpu-free lane

GPU-free feedback loop โ€” per-stage SkillOpt & the walls

The classical loop (Claude box โ†’ GrabCut / shape-prior / vesselness + numpy genome + judge โ€” no SAM, no GPU) was pushed stage-by-stage on all six domains. It beats the net where a geometric/appearance cue exists; elsewhere every stage moved (+0.05โ€“0.09) but hit a GT-box-oracle-proven wall that only the deployed GPU frozen-model lane clears.

Domainbefore [cls]after [cls] ฮ”stage pushedverdict ยท wall
ISIC.923.940 +.017drawer redraw WIN all N โ€” beats net (NN .835/.847/.825)
FetalHead.921.936 +.015ellipse-fit refine WIN all N โ€” beat old oracle (NN .583/.800/.883)
Kvasir.678.771 +.093multi-polyp boxes WIN N=10; loss Nโ‰ฅ25 โ€” localization wall (GT-box oracle .89) โ†’ [gpu]
Prostate.703.751 +.048blur + CLAHE WIN N=10, tie N=50 โ€” box-instability wall (oracle .82) โ†’ [gpu]
CXR.842.880 +.038CLAHE LOSS all โ€” soft-border DECODE wall, GT-oracle .88 โ‰ช NN .92โ€“.95 โ†’ [gpu]
Spleen.50.73 +.23laterality localizer LOSS (mean) โ€” isodense saturation (oracle .85) โ†’ [gpu]
Every stage moved โ€” but perception is the wall. The GPU-free classical loop cleanly beats the net at N=25/50 on ISIC & FetalHead (geometric/appearance cue); on the other 4 every stage was pushed (+0.05โ€“0.09) but each hit a GT-box-oracle-proven wall needing the deployed GPU frozen-model lane. Every stage moved by: drawer redraw (ISIC), ellipse-fit refine (FetalHead), multi-polyp boxes (Kvasir), blur+CLAHE (Prostate/CXR), laterality localizer (Spleen).
Domain-agnostic router. One GT-free router โ€” zero domain names โ€” routes 7/7 domains to their winning decode skill, non-regression PASS (|ฮ”|โ‰ค0.003), with honest [cls] / [gpu] labels. Routing among the frozen GPU drawers is the open problem.
Failure analysis

Failure cases โ€” where the feedback loop breaks

The three lowest-Dice test images per domain from the deployed GPU-free feedback loop (GrabCut + numpy genome; isodense shape-prior for Spleen). Green = ground truth, red = prediction. Each domain's tail is a single, nameable perception failure โ€” not random noise โ€” which is exactly what the GT-free gate cannot always catch.

ISIC three worst feedback-loop cases: input with GT contour (green) and prediction (red)
ISIC ยท Dermoscopy worst-3 Dice 0.84 ยท 0.86 ยท 0.87
faint diffuse periphery โ€” the pigment fades into skin with no crisp rim for GrabCut to snap to, so the mask stops short of the true lesion extent.
FetalHead three worst feedback-loop cases: input with GT contour (green) and prediction (red)
FetalHead ยท Ultrasound worst-3 Dice 0.86 ยท 0.86 ยท 0.88
acoustically-dropped skull walls โ€” where the bright skull ring fades in the shadowed lateral/inferior arc, the boundary is guessed and the ellipse leaks or clips.
Kvasir three worst feedback-loop cases: input with GT contour (green) and prediction (red)
Kvasir ยท Endoscopy worst-3 Dice 0.16 ยท 0.46 ยท 0.53
polyp โ‰ˆ mucosa mislocalization โ€” a low-contrast sessile polyp is the same colour as the surrounding mucosa, so the box+GrabCut latches the wrong structure or the dark lumen.
Prostate three worst feedback-loop cases: input with GT contour (green) and prediction (red)
Prostate ยท T2 MRI worst-3 Dice 0.32 ยท 0.48 ยท 0.58
isodense gland under-seg โ€” apex/base slices where the gland is small and iso-intense with adjacent muscle/bladder; the colour model under-covers or grabs the neighbour.
CXR three worst feedback-loop cases: input with GT contour (green) and prediction (red)
CXR ยท Chest X-ray worst-3 Dice 0.73 ยท 0.78 ยท 0.81
soft diaphragm over-extension โ€” with no clean intensity edge at the diaphragm/cardiac border, the per-lung box bleeds below the costophrenic angle into the abdomen.
Spleen three worst feedback-loop cases: input with GT contour (green) and prediction (red)
Spleen ยท CT (isodense) worst-3 Dice 0.00 ยท 0.00 ยท 0.00
whole-volume localizer miss on isodense CT โ€” the GT-free membership prior mislocalizes atypical caudal/apical slices entirely (Dice 0.00); no intensity edge exists to recover the crescent.
The tail is perception, not the fix. Every worst case is the documented per-domain wall: no valid boundary cue (ISIC periphery, CXR diaphragm), colour โ‰ˆ background (Kvasir polyp, Prostate/Spleen isodense), or a dropped acoustic boundary (FetalHead). GrabCut/shape-prior decode cannot invent an edge the image doesn't carry, and the medians stay high (0.93โ€“0.96) โ€” these are the 1โ€“3/15 hard slices that pull the mean.