Training-free LLM segmentation vs trained networks

A frozen LLM traces the target GT-free and a tiny numpy post-process refines it — no neural network is trained. Here it is compared against a U-Net trained at matched label budgets (N = 10, 25, 50, 100) across four modalities. The LLM line is flat (it never trains); the NN line climbs with labels. Where they cross is the whole story.

Summary — new pipeline (LLM + frozen SAM) vs trained NN

DatasetLLM pipeline (training-free)LLM old→newNN N=10N=25N=50N=100Crossover
ISIC 2018draw → numpy genome (pigment cue); SAM gated OFF0.882→0.892 (+0.01)0.7490.8300.8280.858never (≤100)
Kvasir-SEGdraw → frozen SAM decoder (no GT-free cue)0.580→0.783 (+0.20)0.6280.7770.8080.845N≈50
Chest X-rayCLAHE → draw 2 lungs → SAM per-component → darkness-grow0.788→0.899 (+0.11)0.9250.9430.9500.956N≈10
Spleensoft-tissue window → relocate → SAM decode0.420→0.563 (+0.14)0.5370.6860.8380.885N≈25
All pipelines are training-free (frozen LLM draws + frozen SAM + numpy, ≤8 genome params fit on a few labels — no target-task NN trained). The router picks each domain's path GT-free: cue-rich → numpy genome (ISIC); no boundary cue → frozen SAM decoder (polyp/lung/spleen); SAM never touches a cue-rich genome boundary. Red cells = the training-free LLM matches/beats the trained net at that budget. NN = U-Net/resnet34, ImageNet-pretrained.

ISIC 2018 (dermoscopy)

Dermoscopy

NLLMNN
100.8920.749
250.8920.830
500.8920.828
1000.8920.858
Cue-rich pigment boundary → the numpy genome recovers it (0.892), beating the trained net at every N≤100. SAM is gated OFF here (it would regress an already-good boundary: 0.887→0.817). The standout positive case.
GTLLMNN @N
✓ Good cases
isic · ISIC_0036159
raw image
ground truth
LLM
0.99
NN N=10
0.91
NN N=25
0.91
NN N=50
0.95
NN N=100
0.94
isic · ISIC_0023694
raw image
ground truth
LLM
0.95
NN N=10
0.77
NN N=25
0.92
NN N=50
0.93
NN N=100
0.95
isic · ISIC_0023836
raw image
ground truth
LLM
0.97
NN N=10
0.86
NN N=25
0.96
NN N=50
0.96
NN N=100
0.97
✗ Bad cases
isic · ISIC_0023306
raw image
ground truth
LLM
0.17
NN N=10
0.54
NN N=25
0.30
NN N=50
0.15
NN N=100
0.04
isic · ISIC_0022313
raw image
ground truth
LLM
0.06
NN N=10
0.36
NN N=25
0.35
NN N=50
0.71
NN N=100
0.52
isic · ISIC_0022007
raw image
ground truth
LLM
0.06
NN N=10
0.04
NN N=25
0.13
NN N=50
0.10
NN N=100
0.15

Kvasir-SEG (polyp / endoscopy)

Polyp / endoscopy

NLLMNN
100.7830.628
250.7830.777
500.7830.808
1000.7830.845
No GT-free boundary cue (polyp ≈ mucosa) → a FROZEN SAM decoder, box-prompted by the LLM draw, recovers the boundary the polygon can't: 0.58 → 0.783, now BEATING the NN @N=25 (0.750), with zero target-task labels.
GTLLMNN @N
✓ Good cases
kvasir · cju7fq7mm2pw508176uk5ugtx
raw image
ground truth
LLM
0.90
NN N=10
0.79
NN N=25
0.82
NN N=50
0.94
NN N=100
0.92
kvasir · cju5o1vu9gz8a0818eyy92bns
raw image
ground truth
LLM
0.87
NN N=10
0.72
NN N=25
0.95
NN N=50
0.93
NN N=100
0.96
kvasir · cju7amjna1ly40871ugiokehb
raw image
ground truth
LLM
0.85
NN N=10
0.69
NN N=25
0.91
NN N=50
0.96
NN N=100
0.96
✗ Bad cases
kvasir · cju3xl264ingx0850rcf0rshj
raw image
ground truth
LLM
0.12
NN N=10
0.72
NN N=25
0.70
NN N=50
0.96
NN N=100
0.95
kvasir · cju5u8gz4kj5b07552e2wpkwp
raw image
ground truth
LLM
0.10
NN N=10
0.13
NN N=25
0.56
NN N=50
0.01
NN N=100
0.42
kvasir · cju884985nlmx0817vzpax3y4
raw image
ground truth
LLM
0.06
NN N=10
0.26
NN N=25
0.55
NN N=50
0.88
NN N=100
0.85
🧪 Skill: gated polyp ROI contrast-enhance new
Bad Kvasir cases mislocate — the draw lands on the wrong region. A GT-free contrast enhancer (specular-inpaint + CLAHE-L + saturation) makes the polyp salient enough to find. It rescues mislocations but distorts already-good boundaries, so it is gated on K-draw self-consistency (applied only to low-confidence draws). Skill polyp_roi_enhance (b45cd77).
case groupLLM rawLLM enhancedΔ
bad0.0940.434+0.340
good0.8740.729-0.145
gated (raw on good, enhance on bad)0.4840.654+0.170
GTLLM rawLLM on enhanced view
✗ Bad cases — rescued by enhancement
kvasir · cju3xl264ingx0850rcf0rshj
raw
enhanced view
GT
LLM raw
0.12
LLM enhanced
0.63
kvasir · cju5u8gz4kj5b07552e2wpkwp
raw
enhanced view
GT
LLM raw
0.10
LLM enhanced
0.61
kvasir · cju884985nlmx0817vzpax3y4
raw
enhanced view
GT
LLM raw
0.06
LLM enhanced
0.06
✓ Good cases — enhancement not needed (gated off)
kvasir · cju7fq7mm2pw508176uk5ugtx
raw
enhanced view
GT
LLM raw
0.90
LLM enhanced
0.65
kvasir · cju5o1vu9gz8a0818eyy92bns
raw
enhanced view
GT
LLM raw
0.87
LLM enhanced
0.81
kvasir · cju7amjna1ly40871ugiokehb
raw
enhanced view
GT
LLM raw
0.85
LLM enhanced
0.73

Chest X-ray (lung fields)

Chest X-ray

NLLMNN
100.8990.925
250.8990.943
500.8990.950
1000.8990.956
Two lung fields → SAM PER-COMPONENT (one box per lung) recovers the precise boundary, then a darkness-grow recovers under-segmentation: 0.79 → 0.899, near the trained NN (0.925), GT-free. The per-component prompt was essential (single-box hurts: 0.72).
GTLLMNN @N
✓ Good cases
lungseg · MCUCXR_0150_1
raw image
ground truth
LLM
0.86
NN N=10
0.89
NN N=25
0.91
NN N=50
0.91
NN N=100
0.93
lungseg · CHNCXR_0330_1
raw image
ground truth
LLM
0.85
NN N=10
0.96
NN N=25
0.97
NN N=50
0.98
NN N=100
0.98
lungseg · CHNCXR_0421_1
raw image
ground truth
LLM
0.85
NN N=10
0.94
NN N=25
0.95
NN N=50
0.96
NN N=100
0.95
✗ Bad cases
lungseg · CHNCXR_0229_0
raw image
ground truth
LLM
0.71
NN N=10
0.91
NN N=25
0.94
NN N=50
0.94
NN N=100
0.94
lungseg · MCUCXR_0055_0
raw image
ground truth
LLM
0.71
NN N=10
0.96
NN N=25
0.97
NN N=50
0.97
NN N=100
0.98
lungseg · CHNCXR_0027_0
raw image
ground truth
LLM
0.70
NN N=10
0.94
NN N=25
0.95
NN N=50
0.95
NN N=100
0.96
🧪 Skill: X-ray CLAHE + non-convex lung trace new
Bad lung draws came out as rounded ovals — the polygon rasterizer already supports non-convex (PIL even-odd fill), so this was a drawing issue, not a skill limit. Fix = the literature-standard CXR CLAHE enhancer (xray_clahe) + a guide to trace the true shape (concave mediastinal border + costophrenic angle). Net-positive and non-destructive on good cases (lungs are large/consistent) → applied ungated. Part of the new modality→preprocess registry (8b535f1).
case groupLLM rawLLM enhancedΔ
bad0.7070.762+0.055
good0.8550.846-0.009
all (ungated)0.7810.804+0.023
GTLLM rawLLM on enhanced view
✗ Bad cases — round ovals fixed
lungseg · CHNCXR_0229_0
raw
enhanced view
GT
LLM raw
0.71
LLM enhanced
0.78
lungseg · MCUCXR_0055_0
raw
enhanced view
GT
LLM raw
0.71
LLM enhanced
0.74
lungseg · CHNCXR_0027_0
raw
enhanced view
GT
LLM raw
0.70
LLM enhanced
0.77
✓ Good cases — unaffected
lungseg · MCUCXR_0150_1
raw
enhanced view
GT
LLM raw
0.86
LLM enhanced
0.85
lungseg · CHNCXR_0330_1
raw
enhanced view
GT
LLM raw
0.85
LLM enhanced
0.83
lungseg · CHNCXR_0421_1
raw
enhanced view
GT
LLM raw
0.85
LLM enhanced
0.86

Spleen (abdominal CT)

Abdominal CT

NLLMNN
100.5630.537
250.5630.686
500.5630.838
1000.5630.885
Small invisible-boundary organ → a soft-tissue window relocates the mislocated draw, then SAM decodes: 0.22 → 0.42 → 0.563 (+SAM). Still below the NN (0.70@25) — the residual bottleneck is LOCALIZATION (SAM is prompted by a weak 0.42 draw), not the boundary.
GTLLMNN @N
✓ Good cases
spleen · spleen_8_z024
raw image
ground truth
LLM
0.76
NN N=10
0.68
NN N=25
0.49
NN N=50
0.91
NN N=100
0.96
spleen · spleen_33_z057
raw image
ground truth
LLM
0.75
NN N=10
0.72
NN N=25
0.70
NN N=50
0.81
NN N=100
0.96
spleen · spleen_6_z098
raw image
ground truth
LLM
0.68
NN N=10
0.71
NN N=25
0.90
NN N=50
0.86
NN N=100
0.96
✗ Bad cases
spleen · spleen_21_z058
raw image
ground truth
LLM
0.07
NN N=10
0.55
NN N=25
0.94
NN N=50
0.85
NN N=100
0.87
spleen · spleen_21_z041
raw image
ground truth
LLM
0.01
NN N=10
0.36
NN N=25
0.52
NN N=50
0.65
NN N=100
0.59
spleen · spleen_40_z068
raw image
ground truth
LLM
0.00
NN N=10
0.70
NN N=25
0.75
NN N=50
0.95
NN N=100
0.96