Switch-KD

Abstract

Vision-Language Models (VLMs) have shown remarkable capabilities in joint vision–language understanding, but their large scale poses significant challenges for deployment in resource-constrained scenarios. Knowledge Distillation (KD) offers a viable way to improve model capabilities without increasing model size, making deployment more efficient.

However, applying KD to VLMs is challenged by modality-specific supervision: although multimodal knowledge in VLMs is fused within the language space, current methods supervise each modality separately without explicitly addressing multimodal alignment, leading to inconsistent multimodal knowledge transfer.

To address this, we propose Switch-KD, a visual-switch distillation framework that unifies vision–language knowledge transfer within a shared text-probability space. Switch-KD comprises two key components:

(1) Visual-Switch Distillation, which switches the student's visual outputs into the teacher's language pathway to construct cross-modal probabilistic references for implicit visual knowledge transfer; and

(2) Dynamic Bi-directional Logits Difference (DBiLD) loss, which adaptively aligns informative probability regions while preserving the distributional structures of teacher and student through bidirectional supervision.

Guided by Switch-KD, a 0.5B TinyLLaVA effectively distills rich multimodal knowledge from its 3B teacher, yielding an average improvement of 3.6 points across 10 multimodal benchmarks without any architectural modification.

Method

1. Visual-Switch Distillation

Switches the student's visual encoder outputs into the teacher's language pathway, creating a hybrid inference route where the teacher's powerful language modules interpret the student's visual features. This design enables implicit cross-modal knowledge transfer within the unified text-probability space without modifying the student's architecture.

Key Insight: If the student visual encoder learns meaningful representations, its visual features should be correctly interpreted and decoded by the teacher's language modules, enabling effective knowledge transfer through a simple switch mechanism.

2. DBiLD Loss

Dynamic Bi-directional Logits Difference loss adaptively aligns informative probability regions while preserving the distributional structures of both teacher and student:

• Adaptive Top-K Selection: Uses Kneedle algorithm to automatically find the optimal boundary between information-rich and long-tail regions, focusing on the most informative and confident predictions
• Bidirectional Alignment: Performs alignment from both teacher-guided and student-guided perspectives, ensuring comprehensive knowledge transfer by verifying confident predictions from both sides
• Reverse KL Divergence: Focuses on high-confidence regions for stable optimization, avoiding over-emphasis on noisy long-tail distributions and enabling more reliable knowledge transfer

Results

Switch-KD achieves state-of-the-art performance on 10 multimodal benchmarks, surpassing existing distillation methods and lightweight VLM approaches.

Comparison with LLaVA-KD on 10 Benchmarks

Comparison with Align-KD on 6 Benchmarks

^* Align-KD results are from the long instruction subset (3.6M samples). Switch-KD uses only 1.2M samples with a lighter backbone.

Citation

@article{sun2026switchkd,
  title={Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models},
  author={Sun, Haoyi and Wang, Xiaoxiao and Mao, Ning and Wang, Qian and Mu, Lifu and Zheng, Wen and Wei, Tao and Chen, Wei},
  journal={arXiv preprint arXiv:2604.14629},
  year={2026}
}