Lumen

Pathology vision-language models are usually built the expensive way: pretrain or fine-tune large image and text encoders end to end on paired image-caption data. Lumen asks whether that is necessary. Instead of training a multimodal model from scratch, it takes two strong encoders that already exist, freezes them, and learns only the mapping between them.

The image tower is Virchow2 [2], a self-supervised vision transformer trained on 3.1 million whole slide images. The text tower is BioMedBERT [3], trained from scratch on PubMed abstracts and full-text articles. Both stay frozen. Alignment is learned by rank-4 low-rank adapters (LoRA) inserted into the attention projections of each encoder, plus a linear projection head that maps both modalities into a shared 512-dimensional space. Training uses a symmetric InfoNCE objective on the public QUILT-1M corpus [4], with gradient caching to keep 1024 in-batch negatives despite the memory budget.

That comes to roughly 3 million trainable parameters, 0.40% of the model, and about 12 MB of released weights. The pretrained backbones are never touched.

Key results [1]:

• Zero-shot patch classification: highest mean chance-corrected balanced accuracy across nine public benchmarks, 0.546 against 0.461 for the strongest baseline (paired difference 0.086, 95% CI 0.042-0.136)
• Lymph node metastasis detection: AUROC 0.964 on 4,214 held-out internal slides and 0.955 on 2,368 slides from nine external cohorts spanning six organs
• One locked decision rule: a single threshold calibrated on 215 internal cases, applied unchanged everywhere, giving balanced accuracy 0.909 internally and 0.915 externally with no per-site adjustment
• Cross-modal retrieval: the one axis where Lumen does not lead, ranking third by Recall@5 on ARCH while still separating matched from random image-caption pairs at AUROC 0.957

The lymph node result is the interesting one. Internally, several models are indistinguishable from Lumen. They only separate under distribution shift: Lumen loses 0.009 AUROC going from internal to external data, while some baselines give up as much as 0.18. Against MetAssist 2.0, a supervised specialist trained end to end for this exact task, Lumen reaches higher balanced accuracy on four of nine external cohorts without ever seeing a lymph node label during training.

We also trained a matched control that unfroze both backbones under the same corpus, objective, negative pool and step budget. It fit the training objective better and improved retrieval, but gave no downstream benefit, and was measurably worse under external distribution shift. Updating 250 times more parameters appears to erode the invariances the unimodal pretraining put there in the first place.

A remaining limitation sits in the read-out rather than the features. Prompting handles broad diagnostic categories well but struggles with adjacent grades and subtypes: on SICAPv2, prompting recovers Gleason patterns at 0.226, while a linear probe on the same frozen image features reaches 0.518. The visual representation is not the bottleneck. How effectively language reaches it is.