agenticSeg · data-efficiency study · 2026-07-13

LLM segmentation skills vs trained nets, at a handful of labels

A frozen perception model traces the target; numpy-only skills — a genome, a shape-prior, a vesselness decoder — refine it. No neural net is trained. The question: at N = 5–100 labels, can these skills match a net trained on the same images? Tested against the strongest matched-N baseline (multi-seed SegFormer / UNet++ / nnU-Net), the honest answer is a modest-but-real low-label lead on most domains — and a decisive win where nets need the most data.

5–6 / 7
domains where an LLM skill beats the best matched-N net @ N=10
+0.15
DRIVE vessels — largest lead, where a trained net fails hardest at low N
±0.21
NN seed-noise at N=10 (Spleen); the skill paths are near-deterministic
0.002→0.643
isodense CT: shape-prior vs GrabCut — a NEW GT-free texture prior lifts it 0.52→0.643 (drift zeros 5→1)
Full technical report →
head-to-head

Dice at N = 10 labels — skill vs the strongest net

Each domain: the best LLM-skill path (teal) against the strongest matched-N neural net found across seeds and architectures (clay). Ordered by margin. Bars start at 0.50 Dice.

LLM skill (best path) strongest matched-N net Δ label = skill − net
Read it honestly: earlier single-seed baselines inflated these margins (FetalHead once read +0.34; a properly-chosen net gets 0.816, so the real lead is +0.105). The story survived the correction — it just got modest. CXR is the one clean loss (the net is already 0.921). Matched-N protocol: at each budget both net and skill calibrate on the identical first-N images (verified bit-for-bit); an ISIC id-source divergence was caught and corrected (0.926→0.923, −0.004, verdict unchanged).
by modality & contrast

Where the skills win is set by the target's visual cue

The GPU-free classical decoder wins exactly where the target has an exploitable appearance cue, and needs a different decode skill where it doesn't (isodense CT → shape-prior; filaments → vesselness).

DomainModalityTarget contrast cue best net @10LLM skill @10 Δskill path
Spleen is mixed, not a loss: the deployed skill's mean (0.660) trails the net's lucky best-seed (0.714) but beats its mean (0.547 ± 0.206); its median is 0.929. The net is wildly seed-unstable here; the skill isn't.
data-efficiency · N = 5 / 10 / 25 / 50

Skill flat, net climbing — the crossover budget is the whole story

The NN-free [cls] skill line is flat in N — the drawer is GT-free, so only the ~5–8-param genome ever touches a label and it saturates by N≈10. The flat curve is the point: label-efficiency comes from the labels-free drawer, not from more training data. A third [gpu] line adds the frozen-NN deployed value (frozen SAM / SegGPT / op_sam + a few supports) — also training-free, so it too plots flat, except Kvasir whose op_sam localizer has a measured data-eff curve (N=10 .824, N=25 .824, deployed .8646 @N=30). The arch-max best matched-N net (SegFormer / U-Net / UNet++ envelope, stronger_nn_baselines.md) climbs with labels; where the skill leads at N=5/10 it holds until the net crosses over. Shaded = skill ahead.

NN-free [cls] skill (flat) frozen-NN [gpu] deployed (flat*) matched-N net (arch-max), N=5/10/25/50 shaded = [cls] leads net *flat except Kvasir [gpu] (op_sam measured curve)
Crossover per domain (now across N=5/10/25/50): ISIC & FetalHead — the flat NN-free [cls] skill leads at every budget through N=50 (net never catches in range; the low-N gap is largest at N=5 — ISIC +.20, FetalHead +.27). Kvasir — skill leads N=5/10/25, net crosses by N=50 (.872 > .844). Prostate — skill wins N=5/10, net passes by N=25 (.791). Spleen — skill's mean (.643) leads only at N=5, net passes by N=10; but the deployed skill median (.93) stays ahead throughout. CXR — net leads at all N (already .902 at N=5) — the one clean perception wall on this lane. N=5 NN = arch-max best of the multi-arch/seed sweep; per-domain N=5 per-image jsons live only for DRIVE in this checkout, the rest read from stronger_nn_baselines.md. DRIVE (N=5/10 only, 12-image train pool) is the standout not plotted: skill 0.773 vs net 0.623 (+.15). The [gpu] lane (frozen-NN, flat) sits above [cls] where perception walls bind — CXR .9497, Spleen .920, FetalHead .9667 — reaching ≈NN training-free; both deployed lanes are training-free & label-efficient ([cls] saturates at the ~5–8-param genome fit, [gpu] at its K/N support budget), so both plot flat — except Kvasir op_sam (measured curve). The GPU is currently down, so the [gpu] support-budget curve was not freshly re-measured at every N; flat = the saturated deployed value, and no per-N [gpu] points are fabricated beyond Kvasir's documented curve.
the skills

Four transferable decode skills — and where each deploys

Every skill is data / numpy, LLM-agnostic, and SkillOpt-developed (rollout → reflect → strict-improve validation gate → converge). They deploy exactly where a valid GT-free signal exists.

GPU-free feedback loop

Draw → GrabCut → genome → judge

0.93 ISIC · FetalHead

Claude draws (GT-free), classical GrabCut + numpy genome decodes, a visual judge sends verbal feedback back to the skills. Wins the appearance-cue domains at low N.

✓ deployed · beats net at every N≤50 on wins
isodense shape-prior

Template placed by a spatial prior

0.002 → 0.643

On isodense CT the boundary is invisible — GrabCut carves noise (0.002). A mean-mask template placed by a GT-free localizer recovers it (~250×). A NEW GT-free texture-homogeneity prior then separates the smooth spleen from more-textured wrong blobs, recovering the drift zeros (5→1, 0.52→0.643, median 0.61→0.70); the residual is the caudal-volume location wall, not the prior (oracle @GT-box 0.84 > net).

◑ localization-bound below net
vesselness · learnable

Frangi → shallow RF decode

0.773 +0.15 vs net

Thin retinal vessels can't be boxed. A ridge-filter stack feeds a shallow RandomForest (≤10 labels, CPU). Beats a matched-N net decisively — the extreme low-data case where nets fail hardest.

✓ largest, most robust lead in the study
decode-router

Route by a GT-free cue

isodense → prior

A two-scalar cue (saturation + within-box Otsu separation) routes appearance targets → GrabCut and isodense CT → the shape-prior. Integrates the isodense skill automatically (Spleen 0.002 → 0.438 from routing alone).

✓ deployed · correct on every domain
The recurring lesson — and the honest negatives. Three "push the limit" experiments were run: multiscale (gate-bound, no valid keep-best proxy), mask-refine (mixed — snake grow lifts Prostate +0.028, guided lifts ISIC, but Kvasir/CXR gate-bound), and multi-ROI for CXR's two lungs (fixes the bilateral bookkeeping but +0.003 on test — doesn't close the soft-diaphragm wall). Across all of them: the bottleneck is the GT-free gate / perception, not the fix. A skill deploys iff a valid signal exists.
rigor & status

Caveats kept in view, and what's still running

  • Corrected margins. The dramatic single-seed leads were an artifact; every number here is vs the arch-max best matched-N net. FetalHead: +0.338 → +0.105.
  • Nets are seed-noisy at low N (Spleen N=10: 0.547 ± 0.206), and no architecture dominates — mit_b3 wins FetalHead, UNet++ wins Kvasir/Spleen, unet-resnet34 wins Prostate. The skill paths are near-deterministic.
  • Subset caveat. Feedback-loop & isodense numbers are on fixed 15–28-image test subsets (medians reported); net & deployed-pipeline numbers are full test.
  • Running now: full-default nnU-Net (1000-epoch, @224, all 6 domains, ~25 h) to make the gold-standard comparison airtight — its reduced run already shows nnU-Net@10 weak on Kvasir (0.559) and only a tie on CXR; SegFormer N=100 fill; and the Synapse multi-organ CT run applying the shape-prior + bilateral-kidney multi-ROI to a fresh abdominal dataset.
gpu-free lane

GPU-free feedback loop — per-stage SkillOpt & the walls

The classical loop (Claude box → GrabCut / shape-prior / vesselness + numpy genome + judge — no SAM, no GPU) was pushed stage-by-stage on all six domains. It beats the net where a geometric/appearance cue exists; elsewhere every stage moved (+0.05–0.09) but hit a GT-box-oracle-proven wall that only the deployed GPU frozen-model lane clears.

Domainbefore [cls]after [cls] Δstage pushedverdict · wall
ISIC.923.946 +.023faint-periphery grow (refine) WIN all N — beats net (NN .835/.847/.825); worst-case grow 0019910 .836→.916
FetalHead.921.940 +.019robust ellipse + gated ring-snap (refine) WIN all N — beats net (NN .583/.800/.883); worst-3 fixed 2/3
Kvasir.678.844 +.166drawer per-image re-box worst-3 fixed 3/3 (.17→.86) — drawer localizes by shape, not colour
Prostate.703.763 +.060cross-slice volume gate WIN N=50 (.763>NN .756) — cross-slice gate rescues apex/base 3/3
CXR.842.881 +.039soft-border wall LOSS — deployable static-box 0.8817 (= GT-oracle-box decode exactly; box near-constant); worst-3 trim valid but wrong geometry; soft-border wall → [gpu]
Spleen.50.643 +.14texture-homogeneity prior (preprocess→localize) NEW GT-free feature — texture separates the smooth spleen from textured wrong blobs; drift zeros 5→1, median .61→.70; residual caudal-volume location wall → [gpu]
Every stage moved — but perception is the wall. The GPU-free classical loop cleanly beats the net at N=25/50 on ISIC & FetalHead (geometric/appearance cue); on the other 4 every stage was pushed (+0.05–0.09) but each hit a GT-box-oracle-proven wall needing the deployed GPU frozen-model lane. The decisive lever differs per domain — drawer re-box (Kvasir), cross-slice volume gate (Prostate), faint-periphery grow (ISIC), robust-ellipse refine (FetalHead); a NEW GT-free texture-homogeneity prior then lifted Spleen 0.52→0.643 (drift zeros 5→1), leaving CXR the one clean perception wall.
Worst-case fixes — 4 wins, 2 walls. Six agents each attacked the worst-3 of one domain. 4/6 fixed, and every win landed at a different pipeline stage: the drawer (Kvasir per-image re-box, 3/3, .17→.86), a cross-slice volume gate (Prostate apex/base collapse, 3/3, now beats NN at N=50), and two refines (ISIC geodesic faint-periphery grow 1/3; FetalHead gradient-weighted robust ellipse 2/3) — all GT-free, near-zero regression. The other 2 are perception walls: CXR’s worst-3 over-extension is at lung intensity (a valid trim, wrong geometry) and Spleen’s spleen_40 is caudal, outside the train location atlas (two-walls). Montages below.
The loop is now a reusable meta-harness. The worst-case-driven SkillOpt loop is packaged as worstcase_loop.py: eval → rank worst-3 → an agent diagnoses {failure_mode, responsible_stage, proposed_fix} → apply-and-gate (VAL strict-improve + non-regression) → loop until the two-walls stop (gate-wall: can't separate at the decision boundary; headroom-wall: no better candidate in the family). An ISIC dry-run correctly REJECTED a non-improving edit (grow_frac 0.05→0.12, VAL 0.9443→0.9416) without mutating the deployed config — reproducing the documented two-walls finding. It frames the whole project's method as an automatable loop over the non-NN classical pipeline.
Domain-agnostic router. One GT-free router — zero domain names — routes 7/7 domains to their winning decode skill, non-regression PASS (|Δ|≤0.003), with honest [cls] / [gpu] labels. Routing among the frozen GPU drawers is the open problem.
frozen-nn lane

Pipeline WITH frozen NN — the deployed [gpu] backends

The SAME LLM-agnostic pipeline, but the drawer/decoder seam is a frozen neural backend (op_sam / SegGPT / SAM) — no gradient training on the target task (one few-shot alt lane, noted). Frozen models + a few labeled support images = the same few-label budget as the genome fit. This lane closes the perception-wall domains the classical [cls] lane can't reach, to ≈NN — training-free. The [cls] deployed column is carried alongside so the two lanes read as a pair.

Domain[cls] deployed[gpu] frozen-NN drawer / decoderbest-NNvs best-NN
ISIC.9458.880 genome + SAM-forced.866 +.014
FetalHead.9400.9667 SegGPT in-context, K=8 query-retrieval training-free · few-shot alt .9328.9703 −.004
Kvasir.8440.8646 op_sam DINOv2-corr localizer → frozen SAM · median .9484.889 −.025
Prostate.7633.7877 SegGPT⊕op_sam edge-gated ensemble training-free · median .8483.816 −.028
CXR.8817.9497 SegGPT in-context, K=8 training-free · few-shot alt .9488.9546 −.005
Spleen.6432.920 frozen SAM + within-volume prior + pyramid keep-best · median .926.907 +.019
Within ≤0.005 of best-NN on 4/6 domains, LEADS on ISIC (+0.014) & Spleen (+0.019), and all 6 deployed paths are TRAINING-FREE — frozen models + a few labeled support images (the same few-label budget as the genome fit), zero gradient training on the target task. FetalHead and CXR each also have a few-shot-tuned decoder ALT lane (.9328 / .9488), but the deployed path on both is the training-free SegGPT in-context drawer. FetalHead −0.004 and CXR −0.005 are near-ties; Kvasir (−0.025) and Prostate (−0.028) are the two residual perception gaps.
The two tables are a pair. “Pipeline WITHOUT NN” [cls] (classical, GPU-free, CPU — Claude box → GrabCut / shape-prior / vesselness + numpy genome) vs “Pipeline WITH frozen NN” [gpu] (frozen SAM / SegGPT / op_sam, GPU). The [cls] lane wins the scarce-label regime on the appearance / geometry domains (ISIC, FetalHead); the [gpu] lane closes the perception-wall domains (Spleen, CXR, Kvasir) to ≈NN, training-free. Both lanes are LLM-agnostic and label-scarce — only the drawer/decoder seam swaps.
Failure analysis

Worst-case fixes — 4 wins at 4 stages, a texture-perception recovery, 1 residual wall

Six feedback-loop agents each targeted the three lowest-Dice test images of one domain. Four were fixed and land at a different pipeline stage each — the drawer (Kvasir re-box), a cross-slice volume gate (Prostate), and two mask refines (ISIC grow, FetalHead robust ellipse). For those four the montage is before→after: green = GT, red = pre-fix prediction, blue = post-fix. A fifth — Spleen — was then recovered by a NEW GT-free texture-homogeneity discriminant (drift zeros 5→1); only CXR remains a clean perception wall (worst-3 GT-vs-prediction overlay).

Kvasir worst-3 before to after: GT (green) with pre-fix prediction (red) beside post-fix prediction (blue)
Kvasir · Endoscopy worst-3 fixed 3/3 0.771→0.844 · drawer per-image re-box
green GT · red pre-fix · blue post-fix. polyp ≈ mucosa mislocalization. The LLM drawer re-places the box on the visible smooth, vessel-free polyp using shape/texture cues the colour GMM can’t use — recovering all three near the GT-box oracle (0.17→0.86). Boxes are per-image, so the 12 good frames are bit-identical (zero regression).
Prostate worst-3 before to after: GT (green) with pre-fix prediction (red) beside post-fix prediction (blue)
Prostate · T2 MRI worst-3 fixed 3/3 0.751→0.763 · cross-slice volume gate
green GT · red pre-fix · blue post-fix. isodense apex/base collapse. A GT-free CROSS-SLICE (volume) area/centroid gate flags the collapsed slices and re-selects the family candidate on the volume trajectory; good slices are untouched (zero regression). The loop now beats matched-NN at N=50 (0.763 > 0.756).
ISIC worst-3 before to after: GT (green) with pre-fix prediction (red) beside post-fix prediction (blue)
ISIC · Dermoscopy worst-3 fixed 1/3 0.940→0.9458 · geodesic faint-periphery grow (refine)
green GT · red pre-fix · blue post-fix. faint diffuse periphery. A colour-distance geodesic GROW keyed to the inner-boundary LAB colour follows the fade outward, gated GT-free (osim + box_fill). It fires on the one growable case (0019910 0.836→0.916); the other two are genuinely un-growable (a precision case / no colour-coherent extent) — zero regression.
FetalHead worst-3 before to after: GT (green) with pre-fix prediction (red) beside post-fix prediction (blue)
FetalHead · Ultrasound worst-3 fixed 2/3 0.9357→0.9400 · gradient-weighted robust ellipse + gated ring-snap (refine)
green GT · red pre-fix · blue post-fix. acoustically-dropped skull walls. A Sobel-gradient-weighted ellipse fit is dominated by the true bright ring and EXTRAPOLATES the missing wall (673 0.889→0.899, 108 0.898→0.911). The very worst case (154) — a strongly-tilted GT whose inferior wall is genuinely absent — stays a residual.
CXR three worst cases: input with GT contour (green) and prediction (red)
CXR · Chest X-ray perception wall — no GT-free fix worst-3 Dice 0.73 · 0.78 · 0.81
green GT · red prediction. soft-border wall. The addressable bright-abdomen diaphragm bleed IS trimmable (VAL +0.001, recovers 0.79→0.85), but the TEST worst-3 over-extension is a lateral/medial box-corner fill at LUNG intensity (0.44–0.55), a wide block — indistinguishable from lung by any CPU brightness cue. Trim valid but wrong geometry: no worst-3 recovery. CXR's deployed number is a GT-free static per-lung box (mean of N=25 train GT boxes), 0.8817 — NOT a GT-oracle ceiling: the static box reproduces the GT-oracle-box decode exactly (0.8812 static = 0.8812 oracle) because CXR anatomy makes the box near-constant, so box localization was never the CXR gap. Confirmed a clean perception wall from both sides — a preprocess/localize perception push (bilateral+unsharp +0.0005 TEST; homomorphic/rib-suppression rejected) AND a boundary-refine shape-prior push (lung shape atlas, all 14 configs regressed VAL) both failed to beat the soft diaphragm/costophrenic wall.
Spleen three worst cases: input with GT contour (green) and prediction (red)
Spleen · CT (isodense) texture-perception recovery 0.52→0.643 · median 0.61→0.70 · drift zeros 5→1
green GT · red pre-fix · blue post-fix. A NEW GT-free texture-homogeneity discriminant (a local-std map computed in preprocess→localize) separates the smooth isodense spleen from more-textured wrong blobs, flipping blob selection on the drift zeros: spleen_21_z041 0.00→0.72 (fully recovered), spleen_40 caudal 0.00→0.21–0.37 (partial). This relocates the two-walls story — the drift zeros were a MISSING feature, not an absent one — while the caudal-volume location wall (spleen_40 sits outside the train location atlas 108–143) remains, quantified. A decode-stage locate-and-trim skill (homogeneity_organ_constraint) then constrains the mask to the homogeneous organ region and trims the non-organ bleed → 0.639→0.6432, zero-regression; the worst over-extension spleen_44_z066 0.335→0.795 (a 5× bleed trimmed). Gate-bound (only absolute kept-area separates over-extension from mislocalization), so modest but real.
4 stage-wins, a texture recovery, 1 residual wall. The fixable tails were fixed by whichever stage owned the error: drawer (Kvasir per-image re-box, 3/3, 0.17→0.86), cross-slice gate (Prostate volume-consistency, 3/3, now beats NN at N=50), and refine ×2 (ISIC geodesic grow 1/3; FetalHead gradient-weighted ellipse 2/3) — all GT-free and (bar FetalHead’s tilted residual) zero-regression. Perception push (preprocess+localize): a NEW GT-free texture-homogeneity discriminant then recovered Spleen’s drift zeros (5→1, 0.52→0.643, median 0.61→0.70) — relocating the two-walls story: those zeros were a MISSING feature, not an absent one; the caudal-volume location wall remains, quantified. Only CXR stays a clean wall: its worst-3 over-extension is at lung intensity (a valid trim, wrong geometry). Perception, or its GT-free proxy, remains the lever — not iteration.
structured failure attribution

Where each lane fails — one structured decomposition, run on both lanes

The same 6 domains, attributed with an identical structured failure decomposition (an S sub-skill breakdown → a Linker that names the responsible sub-skill → a two-walls verdict), run on both deployed lanes: the NN-free classical drawer [cls] and the frozen-NN backend [gpu] (op_sam / SegGPT / SAM). This is the evidence map — where and why each lane fails, and what the gap between them reveals.

Domain NN-free [cls] — responsible / wall frozen-NN [gpu] — responsible / wall cross-lane verdict
ISIC S_prompting / S_refinementno_gt_free_gate S_refinementno_family_headroom (genome≈NN) PERSISTS
refinement gate; frozen-NN can't beat the genome (SegGPT +0.11 measured, not deployed)
FetalHead S_refinement (ellipse over-seg) — gate S_model_selection (support-dependence: 3/22 → dice 0.0) FLIPS
refine → model_selection; a NEW failure mode absent classically
Kvasir S_prompting / S_segmentationno_gt_free_gate (polyp≈mucosa) S_model_selection / S_prompting (mislocalization tail) PARTLY DISSOLVES
op_sam localizes (0.865), but the grow/boundary gate persists
Prostate S_prompting / S_refinement — gate / no_family_headroom S_model_selection (SegGPT⊕op_sam) — residual no_family_headroom PARTLY DISSOLVES
complementary drawer drops the gate wall (+0.0277); isodense residual persists (−0.028 to NN)
CXR S_segmentation / S_prompting (soft-border bleed) — no_gt_free_gate S_segmentation — low-severity boundary noise (held-out 0.9512) DISSOLVES
the SegGPT backend swap kills the soft-border wall
Spleen S_prompting / mislocalize (isodense location) — no_gt_free_gate (isodense → SAM + volume) boundary_drift DEPLOYS (edges = anatomy) DISSOLVES
SAM + within-volume prior (0.920, beats NN)
  • S_model_selection is the frozen-NN lane's signature sub-skill — the Linker's top cause on 4/6 domains — and it does not exist on the classical single-drawer lane (never the argmax there). Choosing the frozen backend (op_sam vs SegGPT vs SAM vs few-shot) IS a procedural skill, and it is where the frozen lane's residual failures concentrate — FetalHead SegGPT support-dependence: the in-context drawer draws NOTHING on 3/22 held-out → dice 0.0, unrecoverable by any refine/QC edit.
  • Backend swap dissolves ~half the classical walls; the other half persist. CXR (soft-border), Spleen (isodense location) and Kvasir-localization DISSOLVE (the NN-free escape grabcut→SegGPT / →op_sam was correct); ISIC-refinement, Kvasir-grow (polyp≈mucosa) and Prostate-isodense-residual PERSIST across BOTH lanes. The NN-free lane's cross_lane_escape field predicted which would dissolve.
  • failure_type ↔ wall is a DOMAIN property, not a type property. boundary_drift DEPLOYS on Spleen (edges are anatomy, +0.027) yet is gate-bound on Kvasir/CXR; the same F row resolves oppositely by domain. So the two-walls law is read off the domain, not the failure label.
Provenance, kept honest. NN-free: all 6 freshly measured test-blind (held-out opt splits). Frozen-NN: FetalHead + CXR freshly measured test-blind (held-out Training, GPU); ISIC / Kvasir / Spleen / Prostate doc-grounded from the deployed path + documented residual (worst-case ids marked (repr)), because the full op_sam+SAM+3D stack on held-out was out of scope in one foreground GPU pass.
Does it push the limit? (rigor). Under strict test-blind discipline (held-out select → one-shot test), the classical lane is at its perception ceiling: the all-6 worstcase_loop run moved only FetalHead (+0.0014, the gated ring-snap ellipse — the ONE edit that survived across all six domains). The structured F shows 15/18 residuals are no_gt_free_gate with every escape out-of-lane. Both advanced levers — a few-shot learned gate and a per-image drawer-swap router — confirmed the two-walls law rather than broke it. What DOES push numbers is the domain-level drawer swap (S_model_selection), already deployed per-domain: CXR→SegGPT 0.95, Spleen→SAM 0.92, Kvasir→op_sam 0.86. The loop's contribution is auditable diagnosis + routing to that swap — not squeezing post-processing. And the gate-walls are now confound-free — re-tested with Claude-drawn held-out boxes (regime-matched to the deployed test), the residual headroom still does not transfer: genuine perception walls, not a GT-box artifact (the de-confounding did correct two prior framings — Kvasir’s “signal” was the box regime; ISIC post_grow is neutral, not negative).