When Saliency Over-Credits Anatomy: Token-Level Evidence Flow in Frozen Medical Vision Transformers

Abstract

In medical imaging, saliency maps are among the most common ways to communicate model reasoning, yet a plausible-looking map may not faithfully reflect the evidence a model actually uses. This limitation is amplified in frozen foundation-model pipelines, where large vision transformers separate representation learning from downstream decision-making and combine global [CLS] summaries with local patch-token evidence. We audited the plausibility-faithfulness gap in a frozen chest X-ray foundation model for osteoporosis screening. Exploiting its linear pooled-patch classifier pathway, we derived an exact per-patch attribution for the pooled-patch contribution to the decision margin and used it as a branch-level faithfulness reference. On external radiographs (n = 295), input-gradient saliency assigns, on average, 1.92x more mass to the spine than the exact pooled-patch attribution and exceeds it in all 295 cases, while the spine is clinically plausible for osteoporosis. This reveals a false-trust failure mode: saliency can make a model appear more clinically grounded than its exact decision attribution supports. We present this as a focused interpretability audit and argue for quantitative faithfulness checks before deploying heatmap-based explanations on medical foundation models.

more