agenticSeg Β· data-efficiency study Β· 2026-07-13

LLM segmentation skills vs trained nets, at a handful of labels

A frozen perception model traces the target; numpy-only skills β€” a genome, a shape-prior, a vesselness decoder β€” refine it. No neural net is trained. The question: at N = 5–100 labels, can these skills match a net trained on the same images? Tested against the strongest matched-N baseline (multi-seed SegFormer / UNet++ / nnU-Net), the honest answer is a modest-but-real low-label lead on most domains β€” and a decisive win where nets need the most data.

5–6 / 7
domains where an LLM skill beats the best matched-N net @ N=10
+0.15
DRIVE vessels β€” largest lead, where a trained net fails hardest at low N
Β±0.21
NN seed-noise at N=10 (Spleen); the skill paths are near-deterministic
0.002β†’0.64
isodense CT: shape-prior vs GrabCut β€” a NEW GT-free texture prior lifts it 0.52β†’0.64 (drift zeros 5β†’1)
Full technical report β†’
head-to-head

Dice at N = 10 labels β€” skill vs the strongest net

Each domain: the best LLM-skill path (teal) against the strongest matched-N neural net found across seeds and architectures (clay). Ordered by margin. Bars start at 0.50 Dice.

LLM skill (best path) strongest matched-N net Ξ” label = skill βˆ’ net
Read it honestly: earlier single-seed baselines inflated these margins (FetalHead once read +0.34; a properly-chosen net gets 0.816, so the real lead is +0.105). The story survived the correction β€” it just got modest. CXR is the one clean loss (the net is already 0.921). Matched-N protocol: at each budget both net and skill calibrate on the identical first-N images (verified bit-for-bit); an ISIC id-source divergence was caught and corrected (0.926β†’0.923, βˆ’0.004, verdict unchanged).
by modality & contrast

Where the skills win is set by the target's visual cue

The GPU-free classical decoder wins exactly where the target has an exploitable appearance cue, and needs a different decode skill where it doesn't (isodense CT β†’ shape-prior; filaments β†’ vesselness).

DomainModalityTarget contrast cue best net @10LLM skill @10 Ξ”skill path
Spleen is mixed, not a loss: the deployed skill's mean (0.660) trails the net's lucky best-seed (0.714) but beats its mean (0.547 Β± 0.206); its median is 0.929. The net is wildly seed-unstable here; the skill isn't.
data-efficiency Β· N = 10 / 25 / 50

Skill flat, net climbing β€” the crossover budget is the whole story

The skill line is near-constant in N (the drawer is GT-free; only a light genome uses labels, saturating by Nβ‰ˆ10) while the net climbs with more labels. Where the skill leads at N=10 it holds through N=25/50 until the net catches up β€” the crossover is the honest measure of the low-label edge. Shaded = skill ahead.

LLM skill matched-N net (arch-max) shaded = skill leads
Crossover per domain: ISIC & FetalHead β€” skill leads through N=50 (net never catches in range). Prostate β€” skill wins N=10, net passes by N=25. Kvasir β€” skill (deployed op_sam) leads to Nβ‰ˆ50, net ties at N=50. Spleen β€” net's mean passes by N=25, but the skill median (0.93) stays ahead. CXR β€” net leads throughout. Spleen shown as deployed mean; DRIVE (N=5/10 only, 12-image train pool) is the standout not plotted here: skill 0.773 vs net 0.623.
the skills

Four transferable decode skills β€” and where each deploys

Every skill is data / numpy, LLM-agnostic, and SkillOpt-developed (rollout β†’ reflect β†’ strict-improve validation gate β†’ converge). They deploy exactly where a valid GT-free signal exists.

GPU-free feedback loop

Draw β†’ GrabCut β†’ genome β†’ judge

0.93 ISIC Β· FetalHead

Claude draws (GT-free), classical GrabCut + numpy genome decodes, a visual judge sends verbal feedback back to the skills. Wins the appearance-cue domains at low N.

βœ“ deployed Β· beats net at every N≀50 on wins
isodense shape-prior

Template placed by a spatial prior

0.002 β†’ 0.64

On isodense CT the boundary is invisible β€” GrabCut carves noise (0.002). A mean-mask template placed by a GT-free localizer recovers it (~250Γ—). A NEW GT-free texture-homogeneity prior then separates the smooth spleen from more-textured wrong blobs, recovering the drift zeros (5β†’1, 0.52β†’0.64, median 0.61β†’0.70); the residual is the caudal-volume location wall, not the prior (oracle @GT-box 0.84 > net).

β—‘ localization-bound below net
vesselness Β· learnable

Frangi β†’ shallow RF decode

0.773 +0.15 vs net

Thin retinal vessels can't be boxed. A ridge-filter stack feeds a shallow RandomForest (≀10 labels, CPU). Beats a matched-N net decisively β€” the extreme low-data case where nets fail hardest.

βœ“ largest, most robust lead in the study
decode-router

Route by a GT-free cue

isodense β†’ prior

A two-scalar cue (saturation + within-box Otsu separation) routes appearance targets β†’ GrabCut and isodense CT β†’ the shape-prior. Integrates the isodense skill automatically (Spleen 0.002 β†’ 0.438 from routing alone).

βœ“ deployed Β· correct on every domain
The recurring lesson β€” and the honest negatives. Three "push the limit" experiments were run: multiscale (gate-bound, no valid keep-best proxy), mask-refine (mixed β€” snake grow lifts Prostate +0.028, guided lifts ISIC, but Kvasir/CXR gate-bound), and multi-ROI for CXR's two lungs (fixes the bilateral bookkeeping but +0.003 on test β€” doesn't close the soft-diaphragm wall). Across all of them: the bottleneck is the GT-free gate / perception, not the fix. A skill deploys iff a valid signal exists.
rigor & status

Caveats kept in view, and what's still running

  • Corrected margins. The dramatic single-seed leads were an artifact; every number here is vs the arch-max best matched-N net. FetalHead: +0.338 β†’ +0.105.
  • Nets are seed-noisy at low N (Spleen N=10: 0.547 Β± 0.206), and no architecture dominates β€” mit_b3 wins FetalHead, UNet++ wins Kvasir/Spleen, unet-resnet34 wins Prostate. The skill paths are near-deterministic.
  • Subset caveat. Feedback-loop & isodense numbers are on fixed 15–28-image test subsets (medians reported); net & deployed-pipeline numbers are full test.
  • Running now: full-default nnU-Net (1000-epoch, @224, all 6 domains, ~25 h) to make the gold-standard comparison airtight β€” its reduced run already shows nnU-Net@10 weak on Kvasir (0.559) and only a tie on CXR; SegFormer N=100 fill; and the Synapse multi-organ CT run applying the shape-prior + bilateral-kidney multi-ROI to a fresh abdominal dataset.
gpu-free lane

GPU-free feedback loop β€” per-stage SkillOpt & the walls

The classical loop (Claude box β†’ GrabCut / shape-prior / vesselness + numpy genome + judge β€” no SAM, no GPU) was pushed stage-by-stage on all six domains. It beats the net where a geometric/appearance cue exists; elsewhere every stage moved (+0.05–0.09) but hit a GT-box-oracle-proven wall that only the deployed GPU frozen-model lane clears.

Domainbefore [cls]after [cls] Ξ”stage pushedverdict Β· wall
ISIC.923.946 +.023faint-periphery grow (refine) WIN all N β€” beats net (NN .835/.847/.825); worst-case grow 0019910 .836β†’.916
FetalHead.921.939 +.018robust ellipse (refine) WIN all N β€” beats net (NN .583/.800/.883); worst-3 fixed 2/3
Kvasir.678.844 +.166drawer per-image re-box worst-3 fixed 3/3 (.17β†’.86) β€” drawer localizes by shape, not colour
Prostate.703.763 +.060cross-slice volume gate WIN N=50 (.763>NN .756) β€” cross-slice gate rescues apex/base 3/3
CXR.842.881 +.039soft-border wall LOSS β€” worst-3 trim valid but wrong geometry; soft-border wall β†’ [gpu]
Spleen.50.64 +.14texture-homogeneity prior (preprocess→localize) NEW GT-free feature — texture separates the smooth spleen from textured wrong blobs; drift zeros 5→1, median .61→.70; residual caudal-volume location wall → [gpu]
Every stage moved β€” but perception is the wall. The GPU-free classical loop cleanly beats the net at N=25/50 on ISIC & FetalHead (geometric/appearance cue); on the other 4 every stage was pushed (+0.05–0.09) but each hit a GT-box-oracle-proven wall needing the deployed GPU frozen-model lane. The decisive lever differs per domain β€” drawer re-box (Kvasir), cross-slice volume gate (Prostate), faint-periphery grow (ISIC), robust-ellipse refine (FetalHead); a NEW GT-free texture-homogeneity prior then lifted Spleen 0.52β†’0.64 (drift zeros 5β†’1), leaving CXR the one clean perception wall.
Worst-case fixes β€” 4 wins, 2 walls. Six agents each attacked the worst-3 of one domain. 4/6 fixed, and every win landed at a different pipeline stage: the drawer (Kvasir per-image re-box, 3/3, .17β†’.86), a cross-slice volume gate (Prostate apex/base collapse, 3/3, now beats NN at N=50), and two refines (ISIC geodesic faint-periphery grow 1/3; FetalHead gradient-weighted robust ellipse 2/3) β€” all GT-free, near-zero regression. The other 2 are perception walls: CXR’s worst-3 over-extension is at lung intensity (a valid trim, wrong geometry) and Spleen’s spleen_40 is caudal, outside the train location atlas (two-walls). Montages below.
Domain-agnostic router. One GT-free router β€” zero domain names β€” routes 7/7 domains to their winning decode skill, non-regression PASS (|Ξ”|≀0.003), with honest [cls] / [gpu] labels. Routing among the frozen GPU drawers is the open problem.
Failure analysis

Worst-case fixes β€” 4 wins at 4 stages, a texture-perception recovery, 1 residual wall

Six feedback-loop agents each targeted the three lowest-Dice test images of one domain. Four were fixed and land at a different pipeline stage each — the drawer (Kvasir re-box), a cross-slice volume gate (Prostate), and two mask refines (ISIC grow, FetalHead robust ellipse). For those four the montage is before→after: green = GT, red = pre-fix prediction, blue = post-fix. A fifth — Spleen — was then recovered by a NEW GT-free texture-homogeneity discriminant (drift zeros 5→1); only CXR remains a clean perception wall (worst-3 GT-vs-prediction overlay).

Kvasir worst-3 before to after: GT (green) with pre-fix prediction (red) beside post-fix prediction (blue)
Kvasir Β· Endoscopy worst-3 fixed 3/3 0.771β†’0.844 Β· drawer per-image re-box
green GT Β· red pre-fix Β· blue post-fix. polyp β‰ˆ mucosa mislocalization. The LLM drawer re-places the box on the visible smooth, vessel-free polyp using shape/texture cues the colour GMM can’t use β€” recovering all three near the GT-box oracle (0.17β†’0.86). Boxes are per-image, so the 12 good frames are bit-identical (zero regression).
Prostate worst-3 before to after: GT (green) with pre-fix prediction (red) beside post-fix prediction (blue)
Prostate Β· T2 MRI worst-3 fixed 3/3 0.751β†’0.763 Β· cross-slice volume gate
green GT Β· red pre-fix Β· blue post-fix. isodense apex/base collapse. A GT-free CROSS-SLICE (volume) area/centroid gate flags the collapsed slices and re-selects the family candidate on the volume trajectory; good slices are untouched (zero regression). The loop now beats matched-NN at N=50 (0.763 > 0.756).
ISIC worst-3 before to after: GT (green) with pre-fix prediction (red) beside post-fix prediction (blue)
ISIC Β· Dermoscopy worst-3 fixed 1/3 0.940β†’0.9458 Β· geodesic faint-periphery grow (refine)
green GT Β· red pre-fix Β· blue post-fix. faint diffuse periphery. A colour-distance geodesic GROW keyed to the inner-boundary LAB colour follows the fade outward, gated GT-free (osim + box_fill). It fires on the one growable case (0019910 0.836β†’0.916); the other two are genuinely un-growable (a precision case / no colour-coherent extent) β€” zero regression.
FetalHead worst-3 before to after: GT (green) with pre-fix prediction (red) beside post-fix prediction (blue)
FetalHead Β· Ultrasound worst-3 fixed 2/3 0.9357β†’0.9387 Β· gradient-weighted robust ellipse (refine)
green GT Β· red pre-fix Β· blue post-fix. acoustically-dropped skull walls. A Sobel-gradient-weighted ellipse fit is dominated by the true bright ring and EXTRAPOLATES the missing wall (673 0.889β†’0.899, 108 0.898β†’0.911). The very worst case (154) β€” a strongly-tilted GT whose inferior wall is genuinely absent β€” stays a residual.
CXR three worst cases: input with GT contour (green) and prediction (red)
CXR Β· Chest X-ray perception wall β€” no GT-free fix worst-3 Dice 0.73 Β· 0.78 Β· 0.81
green GT Β· red prediction. soft-border wall. The addressable bright-abdomen diaphragm bleed IS trimmable (VAL +0.001, recovers 0.79β†’0.85), but the TEST worst-3 over-extension is a lateral/medial box-corner fill at LUNG intensity (0.44–0.55), a wide block β€” indistinguishable from lung by any CPU brightness cue. Trim valid but wrong geometry: no worst-3 recovery.
Spleen three worst cases: input with GT contour (green) and prediction (red)
Spleen Β· CT (isodense) texture-perception recovery 0.52β†’0.64 Β· median 0.61β†’0.70 Β· drift zeros 5β†’1
green GT Β· red pre-fix Β· blue post-fix. A NEW GT-free texture-homogeneity discriminant (a local-std map computed in preprocessβ†’localize) separates the smooth isodense spleen from more-textured wrong blobs, flipping blob selection on the drift zeros: spleen_21_z041 0.00β†’0.72 (fully recovered), spleen_40 caudal 0.00β†’0.21–0.37 (partial). This relocates the two-walls story β€” the drift zeros were a MISSING feature, not an absent one β€” while the caudal-volume location wall (spleen_40 sits outside the train location atlas 108–143) remains, quantified.
4 stage-wins, a texture recovery, 1 residual wall. The fixable tails were fixed by whichever stage owned the error: drawer (Kvasir per-image re-box, 3/3, 0.17β†’0.86), cross-slice gate (Prostate volume-consistency, 3/3, now beats NN at N=50), and refine Γ—2 (ISIC geodesic grow 1/3; FetalHead gradient-weighted ellipse 2/3) β€” all GT-free and (bar FetalHead’s tilted residual) zero-regression. Perception push (preprocess+localize): a NEW GT-free texture-homogeneity discriminant then recovered Spleen’s drift zeros (5β†’1, 0.52β†’0.64, median 0.61β†’0.70) β€” relocating the two-walls story: those zeros were a MISSING feature, not an absent one; the caudal-volume location wall remains, quantified. Only CXR stays a clean wall: its worst-3 over-extension is at lung intensity (a valid trim, wrong geometry). Perception, or its GT-free proxy, remains the lever β€” not iteration.