head-to-head
Dice at N = 10 labels — skill vs the strongest net
Each domain: the best LLM-skill path (teal) against the strongest matched-N neural net found across seeds and architectures (clay). Ordered by margin. Bars start at 0.50 Dice.
LLM skill (best path)
strongest matched-N net
Δ label = skill − net
Read it honestly: earlier single-seed baselines inflated these margins (FetalHead
once read +0.34; a properly-chosen net gets 0.816, so the real lead is +0.105). The story survived the
correction — it just got modest. CXR is the one clean loss (the net is already 0.921). Matched-N protocol: at each budget both net and skill calibrate on the identical first-N images (verified bit-for-bit); an ISIC id-source divergence was caught and corrected (0.926→0.923, −0.004, verdict unchanged).
by modality & contrast
Where the skills win is set by the target's visual cue
The GPU-free classical decoder wins exactly where the target has an exploitable appearance cue, and needs a different decode skill where it doesn't (isodense CT → shape-prior; filaments → vesselness).
| Domain | Modality | Target contrast cue | best net @10 | LLM skill @10 | Δ | skill path |
|---|
Spleen is mixed, not a loss: the deployed skill's mean (0.660)
trails the net's lucky best-seed (0.714) but beats its mean (0.547 ± 0.206);
its median is 0.929. The net is wildly seed-unstable here; the skill isn't.
data-efficiency · N = 5 / 10 / 25 / 50
Skill flat, net climbing — the crossover budget is the whole story
The NN-free [cls] skill line is flat in N — the drawer is GT-free, so only the ~5–8-param genome ever touches a label and it saturates by N≈10. The flat curve is the point: label-efficiency comes from the labels-free drawer, not from more training data. A third [gpu] line adds the frozen-NN deployed value (frozen SAM / SegGPT / op_sam + a few supports) — also training-free, so it too plots flat, except Kvasir whose op_sam localizer has a measured data-eff curve (N=10 .824, N=25 .824, deployed .8646 @N=30). The arch-max best matched-N net (SegFormer / U-Net / UNet++ envelope, stronger_nn_baselines.md) climbs with labels; where the skill leads at N=5/10 it holds until the net crosses over. Shaded = skill ahead.
NN-free [cls] skill (flat)
frozen-NN [gpu] deployed (flat*)
matched-N net (arch-max), N=5/10/25/50
shaded = [cls] leads net
*flat except Kvasir [gpu] (op_sam measured curve)
Crossover per domain (now across N=5/10/25/50): ISIC & FetalHead — the flat
NN-free [cls] skill leads at every budget through N=50 (net never catches in range;
the low-N gap is largest at N=5 — ISIC +.20, FetalHead +.27). Kvasir — skill leads N=5/10/25, net crosses by
N=50 (.872 > .844). Prostate — skill wins N=5/10, net passes by N=25 (.791). Spleen — skill's mean
(.643) leads only at N=5, net passes by N=10; but the deployed skill median (.93) stays ahead throughout.
CXR — net leads at all N (already .902 at N=5) — the one clean perception wall on this lane. N=5 NN =
arch-max best of the multi-arch/seed sweep; per-domain N=5 per-image jsons live only for DRIVE in this checkout,
the rest read from stronger_nn_baselines.md. DRIVE (N=5/10 only, 12-image train pool) is
the standout not plotted: skill 0.773 vs net 0.623 (+.15). The [gpu] lane
(frozen-NN, flat) sits above [cls] where perception walls bind — CXR
.9497, Spleen .920, FetalHead .9667 — reaching ≈NN training-free; both deployed lanes are
training-free & label-efficient ([cls] saturates at the ~5–8-param genome fit,
[gpu] at its K/N support budget), so both plot flat — except Kvasir op_sam
(measured curve). The GPU is currently down, so the [gpu] support-budget curve was not freshly re-measured
at every N; flat = the saturated deployed value, and no per-N [gpu] points are fabricated beyond Kvasir's
documented curve.
the skills
Four transferable decode skills — and where each deploys
Every skill is data / numpy, LLM-agnostic, and SkillOpt-developed (rollout → reflect → strict-improve validation gate → converge). They deploy exactly where a valid GT-free signal exists.
GPU-free feedback loop
Draw → GrabCut → genome → judge
0.93 ISIC · FetalHead
Claude draws (GT-free), classical GrabCut + numpy genome decodes, a visual judge sends verbal feedback back to the skills. Wins the appearance-cue domains at low N.
✓ deployed · beats net at every N≤50 on wins
isodense shape-prior
Template placed by a spatial prior
0.002 → 0.643
On isodense CT the boundary is invisible — GrabCut carves noise (0.002). A mean-mask template placed by a GT-free localizer recovers it (~250×). A NEW GT-free texture-homogeneity prior then separates the smooth spleen from more-textured wrong blobs, recovering the drift zeros (5→1, 0.52→0.643, median 0.61→0.70); the residual is the caudal-volume location wall, not the prior (oracle @GT-box 0.84 > net).
◑ localization-bound below net
vesselness · learnable
Frangi → shallow RF decode
0.773 +0.15 vs net
Thin retinal vessels can't be boxed. A ridge-filter stack feeds a shallow RandomForest (≤10 labels, CPU). Beats a matched-N net decisively — the extreme low-data case where nets fail hardest.
✓ largest, most robust lead in the study
decode-router
Route by a GT-free cue
isodense → prior
A two-scalar cue (saturation + within-box Otsu separation) routes appearance targets → GrabCut and isodense CT → the shape-prior. Integrates the isodense skill automatically (Spleen 0.002 → 0.438 from routing alone).
✓ deployed · correct on every domain
The recurring lesson — and the honest negatives. Three "push the limit" experiments
were run: multiscale (gate-bound, no valid keep-best proxy), mask-refine (mixed — snake grow
lifts Prostate +0.028, guided lifts ISIC, but Kvasir/CXR gate-bound), and multi-ROI for CXR's two lungs
(fixes the bilateral bookkeeping but +0.003 on test — doesn't close the soft-diaphragm wall). Across all of them:
the bottleneck is the GT-free gate / perception, not the fix. A skill deploys iff a valid signal exists.
rigor & status
Caveats kept in view, and what's still running
- Corrected margins. The dramatic single-seed leads were an artifact; every number here is vs the arch-max best matched-N net. FetalHead: +0.338 → +0.105.
- Nets are seed-noisy at low N (Spleen N=10: 0.547 ± 0.206), and no architecture dominates — mit_b3 wins FetalHead, UNet++ wins Kvasir/Spleen, unet-resnet34 wins Prostate. The skill paths are near-deterministic.
- Subset caveat. Feedback-loop & isodense numbers are on fixed 15–28-image test subsets (medians reported); net & deployed-pipeline numbers are full test.
- Running now: full-default nnU-Net (1000-epoch, @224, all 6 domains, ~25 h) to make the gold-standard comparison airtight — its reduced run already shows nnU-Net@10 weak on Kvasir (0.559) and only a tie on CXR; SegFormer N=100 fill; and the Synapse multi-organ CT run applying the shape-prior + bilateral-kidney multi-ROI to a fresh abdominal dataset.
gpu-free lane
GPU-free feedback loop — per-stage SkillOpt & the walls
The classical loop (Claude box → GrabCut / shape-prior / vesselness + numpy genome + judge — no SAM, no GPU) was pushed stage-by-stage on all six domains. It beats the net where a geometric/appearance cue exists; elsewhere every stage moved (+0.05–0.09) but hit a GT-box-oracle-proven wall that only the deployed GPU frozen-model lane clears.
| Domain | before [cls] | after [cls] | Δ | stage pushed | verdict · wall |
|---|---|---|---|---|---|
| ISIC | .923 | .946 | +.023 | faint-periphery grow (refine) | WIN all N — beats net (NN .835/.847/.825); worst-case grow 0019910 .836→.916 |
| FetalHead | .921 | .940 | +.019 | robust ellipse + gated ring-snap (refine) | WIN all N — beats net (NN .583/.800/.883); worst-3 fixed 2/3 |
| Kvasir | .678 | .844 | +.166 | drawer per-image re-box | worst-3 fixed 3/3 (.17→.86) — drawer localizes by shape, not colour |
| Prostate | .703 | .763 | +.060 | cross-slice volume gate | WIN N=50 (.763>NN .756) — cross-slice gate rescues apex/base 3/3 |
| CXR | .842 | .881 | +.039 | soft-border wall | LOSS — deployable static-box 0.8817 (= GT-oracle-box decode exactly; box near-constant); worst-3 trim valid but wrong geometry; soft-border wall → [gpu] |
| Spleen | .50 | .643 | +.14 | texture-homogeneity prior (preprocess→localize) | NEW GT-free feature — texture separates the smooth spleen from textured wrong blobs; drift zeros 5→1, median .61→.70; residual caudal-volume location wall → [gpu] |
Every stage moved — but perception is the wall. The GPU-free classical loop cleanly beats the
net at N=25/50 on ISIC & FetalHead (geometric/appearance cue); on the other 4 every stage was pushed
(+0.05–0.09) but each hit a GT-box-oracle-proven wall needing the deployed GPU
frozen-model lane. The decisive lever differs per domain — drawer re-box (Kvasir), cross-slice volume gate (Prostate),
faint-periphery grow (ISIC), robust-ellipse refine (FetalHead); a NEW GT-free texture-homogeneity prior then lifted Spleen 0.52→0.643 (drift zeros 5→1), leaving CXR the one clean perception wall.
Worst-case fixes — 4 wins, 2 walls. Six agents each attacked the worst-3 of one
domain. 4/6 fixed, and every win landed at a different pipeline stage: the drawer (Kvasir
per-image re-box, 3/3, .17→.86), a cross-slice volume gate (Prostate apex/base collapse, 3/3, now beats
NN at N=50), and two refines (ISIC geodesic faint-periphery grow 1/3; FetalHead gradient-weighted robust
ellipse 2/3) — all GT-free, near-zero regression. The other 2 are perception walls: CXR’s worst-3
over-extension is at lung intensity (a valid trim, wrong geometry) and Spleen’s spleen_40
is caudal, outside the train location atlas (two-walls). Montages below.
The loop is now a reusable meta-harness. The worst-case-driven SkillOpt loop is packaged as
worstcase_loop.py: eval → rank worst-3 → an agent diagnoses
{failure_mode, responsible_stage, proposed_fix} → apply-and-gate (VAL strict-improve +
non-regression) → loop until the two-walls stop (gate-wall: can't separate at the decision boundary;
headroom-wall: no better candidate in the family). An ISIC dry-run correctly REJECTED a non-improving edit
(grow_frac 0.05→0.12, VAL 0.9443→0.9416) without mutating the deployed config — reproducing the documented
two-walls finding. It frames the whole project's method as an automatable loop over the non-NN classical pipeline.
Domain-agnostic router. One GT-free router — zero domain names — routes 7/7 domains to
their winning decode skill, non-regression PASS (|Δ|≤0.003), with honest
[cls] / [gpu] labels. Routing among the frozen GPU
drawers is the open problem.
frozen-nn lane
Pipeline WITH frozen NN — the deployed [gpu] backends
The SAME LLM-agnostic pipeline, but the drawer/decoder seam is a frozen neural backend (op_sam / SegGPT / SAM) — no gradient training on the target task (one few-shot alt lane, noted). Frozen models + a few labeled support images = the same few-label budget as the genome fit. This lane closes the perception-wall domains the classical [cls] lane can't reach, to ≈NN — training-free. The [cls] deployed column is carried alongside so the two lanes read as a pair.
| Domain | [cls] deployed | [gpu] frozen-NN | drawer / decoder | best-NN | vs best-NN |
|---|---|---|---|---|---|
| ISIC | .9458 | .880 | genome + SAM-forced | .866 | +.014 |
| FetalHead | .9400 | .9667 | SegGPT in-context, K=8 query-retrieval training-free · few-shot alt .9328 | .9703 | −.004 |
| Kvasir | .8440 | .8646 | op_sam DINOv2-corr localizer → frozen SAM · median .9484 | .889 | −.025 |
| Prostate | .7633 | .7877 | SegGPT⊕op_sam edge-gated ensemble training-free · median .8483 | .816 | −.028 |
| CXR | .8817 | .9497 | SegGPT in-context, K=8 training-free · few-shot alt .9488 | .9546 | −.005 |
| Spleen | .6432 | .920 | frozen SAM + within-volume prior + pyramid keep-best · median .926 | .907 | +.019 |
Within ≤0.005 of best-NN on 4/6 domains, LEADS on ISIC (+0.014) & Spleen (+0.019), and all 6
deployed paths are TRAINING-FREE — frozen models + a few labeled support images (the same few-label budget as
the genome fit), zero gradient training on the target task. FetalHead and CXR each also have a few-shot-tuned
decoder ALT lane (.9328 / .9488), but the deployed path on both is the training-free SegGPT in-context
drawer. FetalHead −0.004 and CXR −0.005 are near-ties; Kvasir (−0.025) and Prostate (−0.028) are the two residual
perception gaps.
The two tables are a pair. “Pipeline WITHOUT NN” [cls]
(classical, GPU-free, CPU — Claude box → GrabCut / shape-prior / vesselness + numpy genome) vs “Pipeline WITH
frozen NN” [gpu] (frozen SAM / SegGPT / op_sam, GPU). The [cls]
lane wins the scarce-label regime on the appearance / geometry domains (ISIC, FetalHead); the
[gpu] lane closes the perception-wall domains (Spleen, CXR, Kvasir) to ≈NN,
training-free. Both lanes are LLM-agnostic and label-scarce — only the drawer/decoder seam swaps.
Failure analysis
Worst-case fixes — 4 wins at 4 stages, a texture-perception recovery, 1 residual wall
Six feedback-loop agents each targeted the three lowest-Dice test images of one domain. Four were fixed and land at a different pipeline stage each — the drawer (Kvasir re-box), a cross-slice volume gate (Prostate), and two mask refines (ISIC grow, FetalHead robust ellipse). For those four the montage is before→after: green = GT, red = pre-fix prediction, blue = post-fix. A fifth — Spleen — was then recovered by a NEW GT-free texture-homogeneity discriminant (drift zeros 5→1); only CXR remains a clean perception wall (worst-3 GT-vs-prediction overlay).
green GT · red pre-fix · blue post-fix. polyp ≈ mucosa mislocalization. The LLM drawer re-places the box on the visible smooth, vessel-free polyp using shape/texture cues the colour GMM can’t use — recovering all three near the GT-box oracle (0.17→0.86). Boxes are per-image, so the 12 good frames are bit-identical (zero regression).
green GT · red pre-fix · blue post-fix. isodense apex/base collapse. A GT-free CROSS-SLICE (volume) area/centroid gate flags the collapsed slices and re-selects the family candidate on the volume trajectory; good slices are untouched (zero regression). The loop now beats matched-NN at N=50 (0.763 > 0.756).
green GT · red pre-fix · blue post-fix. faint diffuse periphery. A colour-distance geodesic GROW keyed to the inner-boundary LAB colour follows the fade outward, gated GT-free (osim + box_fill). It fires on the one growable case (0019910 0.836→0.916); the other two are genuinely un-growable (a precision case / no colour-coherent extent) — zero regression.
green GT · red pre-fix · blue post-fix. acoustically-dropped skull walls. A Sobel-gradient-weighted ellipse fit is dominated by the true bright ring and EXTRAPOLATES the missing wall (673 0.889→0.899, 108 0.898→0.911). The very worst case (154) — a strongly-tilted GT whose inferior wall is genuinely absent — stays a residual.
green GT · red prediction. soft-border wall. The addressable bright-abdomen diaphragm bleed IS trimmable (VAL +0.001, recovers 0.79→0.85), but the TEST worst-3 over-extension is a lateral/medial box-corner fill at LUNG intensity (0.44–0.55), a wide block — indistinguishable from lung by any CPU brightness cue. Trim valid but wrong geometry: no worst-3 recovery. CXR's deployed number is a GT-free static per-lung box (mean of N=25 train GT boxes), 0.8817 — NOT a GT-oracle ceiling: the static box reproduces the GT-oracle-box decode exactly (0.8812 static = 0.8812 oracle) because CXR anatomy makes the box near-constant, so box localization was never the CXR gap. Confirmed a clean perception wall from both sides — a preprocess/localize perception push (bilateral+unsharp +0.0005 TEST; homomorphic/rib-suppression rejected) AND a boundary-refine shape-prior push (lung shape atlas, all 14 configs regressed VAL) both failed to beat the soft diaphragm/costophrenic wall.
green GT · red pre-fix · blue post-fix. A NEW GT-free texture-homogeneity discriminant (a local-std map computed in preprocess→localize) separates the smooth isodense spleen from more-textured wrong blobs, flipping blob selection on the drift zeros: spleen_21_z041 0.00→0.72 (fully recovered), spleen_40 caudal 0.00→0.21–0.37 (partial). This relocates the two-walls story — the drift zeros were a MISSING feature, not an absent one — while the caudal-volume location wall (spleen_40 sits outside the train location atlas 108–143) remains, quantified. A decode-stage locate-and-trim skill (homogeneity_organ_constraint) then constrains the mask to the homogeneous organ region and trims the non-organ bleed → 0.639→0.6432, zero-regression; the worst over-extension spleen_44_z066 0.335→0.795 (a 5× bleed trimmed). Gate-bound (only absolute kept-area separates over-extension from mislocalization), so modest but real.
4 stage-wins, a texture recovery, 1 residual wall. The fixable tails were fixed by whichever
stage owned the error: drawer (Kvasir per-image re-box, 3/3, 0.17→0.86), cross-slice gate
(Prostate volume-consistency, 3/3, now beats NN at N=50), and refine ×2 (ISIC geodesic grow 1/3;
FetalHead gradient-weighted ellipse 2/3) — all GT-free and (bar FetalHead’s tilted residual) zero-regression.
Perception push (preprocess+localize): a NEW GT-free texture-homogeneity discriminant then recovered
Spleen’s drift zeros (5→1, 0.52→0.643, median 0.61→0.70) — relocating the two-walls story: those zeros
were a MISSING feature, not an absent one; the caudal-volume location wall remains, quantified. Only CXR
stays a clean wall: its worst-3 over-extension is at lung intensity (a valid trim, wrong geometry). Perception, or
its GT-free proxy, remains the lever — not iteration.
structured failure attribution
Where each lane fails — one structured decomposition, run on both lanes
The same 6 domains, attributed with an identical structured failure decomposition (an S sub-skill breakdown → a Linker that names the responsible sub-skill → a two-walls verdict), run on both deployed lanes: the NN-free classical drawer [cls] and the frozen-NN backend [gpu] (op_sam / SegGPT / SAM). This is the evidence map — where and why each lane fails, and what the gap between them reveals.
| Domain | NN-free [cls] — responsible / wall | frozen-NN [gpu] — responsible / wall | cross-lane verdict |
|---|---|---|---|
| ISIC | S_prompting / S_refinement — no_gt_free_gate | S_refinement — no_family_headroom (genome≈NN) | PERSISTS refinement gate; frozen-NN can't beat the genome (SegGPT +0.11 measured, not deployed) |
| FetalHead | S_refinement (ellipse over-seg) — gate | S_model_selection (support-dependence: 3/22 → dice 0.0) | FLIPS refine → model_selection; a NEW failure mode absent classically |
| Kvasir | S_prompting / S_segmentation — no_gt_free_gate (polyp≈mucosa) | S_model_selection / S_prompting (mislocalization tail) | PARTLY DISSOLVES op_sam localizes (0.865), but the grow/boundary gate persists |
| Prostate | S_prompting / S_refinement — gate / no_family_headroom | S_model_selection (SegGPT⊕op_sam) — residual no_family_headroom | PARTLY DISSOLVES complementary drawer drops the gate wall (+0.0277); isodense residual persists (−0.028 to NN) |
| CXR | S_segmentation / S_prompting (soft-border bleed) — no_gt_free_gate | S_segmentation — low-severity boundary noise (held-out 0.9512) | DISSOLVES the SegGPT backend swap kills the soft-border wall |
| Spleen | S_prompting / mislocalize (isodense location) — no_gt_free_gate | (isodense → SAM + volume) boundary_drift DEPLOYS (edges = anatomy) | DISSOLVES SAM + within-volume prior (0.920, beats NN) |
- S_model_selection is the frozen-NN lane's signature sub-skill — the Linker's top cause on 4/6 domains — and it does not exist on the classical single-drawer lane (never the argmax there). Choosing the frozen backend (op_sam vs SegGPT vs SAM vs few-shot) IS a procedural skill, and it is where the frozen lane's residual failures concentrate — FetalHead SegGPT support-dependence: the in-context drawer draws NOTHING on 3/22 held-out → dice 0.0, unrecoverable by any refine/QC edit.
- Backend swap dissolves ~half the classical walls; the other half persist. CXR (soft-border), Spleen (isodense location) and Kvasir-localization DISSOLVE (the NN-free escape grabcut→SegGPT / →op_sam was correct); ISIC-refinement, Kvasir-grow (polyp≈mucosa) and Prostate-isodense-residual PERSIST across BOTH lanes. The NN-free lane's cross_lane_escape field predicted which would dissolve.
- failure_type ↔ wall is a DOMAIN property, not a type property. boundary_drift DEPLOYS on Spleen (edges are anatomy, +0.027) yet is gate-bound on Kvasir/CXR; the same F row resolves oppositely by domain. So the two-walls law is read off the domain, not the failure label.
Provenance, kept honest. NN-free: all 6 freshly measured test-blind (held-out opt
splits). Frozen-NN: FetalHead + CXR freshly measured test-blind (held-out Training, GPU); ISIC / Kvasir /
Spleen / Prostate doc-grounded from the deployed path + documented residual (worst-case ids marked
(repr)), because the full op_sam+SAM+3D stack on held-out was out of scope in one
foreground GPU pass.
Does it push the limit? (rigor). Under strict test-blind discipline (held-out select →
one-shot test), the classical lane is at its perception ceiling: the all-6 worstcase_loop
run moved only FetalHead (+0.0014, the gated ring-snap ellipse — the ONE edit that survived across all six
domains). The structured F shows 15/18 residuals are no_gt_free_gate with every
escape out-of-lane. Both advanced levers — a few-shot learned gate and a per-image drawer-swap router —
confirmed the two-walls law rather than broke it. What DOES push numbers is the domain-level drawer swap
(S_model_selection), already deployed per-domain: CXR→SegGPT 0.95, Spleen→SAM 0.92,
Kvasir→op_sam 0.86. The loop's contribution is auditable diagnosis + routing to that swap — not squeezing post-processing. And the gate-walls are now confound-free — re-tested with Claude-drawn held-out boxes (regime-matched to the deployed test), the residual headroom still does not transfer: genuine perception walls, not a GT-box artifact (the de-confounding did correct two prior framings — Kvasir’s “signal” was the box regime; ISIC post_grow is neutral, not negative).