Modality-Aware Expert Pruning Reduces Memory in MoE-MLLMs
Modality-aware expert pruning (MAEP) reduces BF16 expert weight footprint by 27 GB in MoE-MLLMs, enabling 58 GB models to fit within 40 GB GPUs while maintaining performance.
Problem with Expert Parameters in MoE-MLLMs
Mixture-of-Experts (MoE) architectures allow multimodal large language models (MLLMs) to scale efficiently, but they face a significant challenge: expert parameters consume the majority of memory, even though only a small subset is activated per token. In modern MoE-MLLMs, such as InternVL3.5-30B-A3B and Qwen3-VL-30B-A3B, expert parameters account for over 90% of the model size, with one example reaching 54 GB out of a total of about 58 GB in BF16 format [arXiv CV]. Existing expert pruning methods often treat all tokens uniformly, failing to account for the differing roles experts play in vision versus language processing. This oversight limits the effectiveness of compression strategies. Modality-aware expert pruning (MAEP) addresses this by considering the distinct contributions of experts across modalities, achieving significant memory reduction while maintaining performance. Experiments show that MAEP reduces the BF16 expert weight footprint by about 27 GB (approximately 47% of total memory), enabling a 58 GB model to fit within a single 40 GB GPU [arXiv CV].
Introduction of MAEP Framework
To address the challenge of expert importance decomposition in MoE-based multimodal large language models, the MAEP (Modality-Aware Expert Pruning) framework introduces a training-free approach that operates within a single forward pass. MAEP utilizes two key components: Delta-H Output ($\Delta H$), a redistribution-aware metric that quantifies changes in MoE output when an expert is removed, and low-attention vision tokens, which serve as a dataset-stable pruning signal for the vision side. On the text side, importance acts as a preservation signal, while on the vision side, it functions as a pruning signal. Experts are ranked globally across layers, enabling efficient and effective pruning without the need for additional training. This method ensures that the expert-importance ranking remains consistent across different datasets, enhancing the robustness and adaptability of the model.
Performance and Memory Reduction Achieved by MAEP
Experiments on three MoE-based MLLMs across multimodal and text benchmarks show that MAEP outperforms modality-agnostic baselines on the majority of multimodal benchmarks while remaining competitive on text-only tasks, reducing the BF16 expert weight footprint by about 27 GB (about 47% of total memory) so that the full weight footprint of an approximately 58 GB model fits within a single 40 GB GPU. This reduction in memory footprint is attributed to the modality-aware pruning strategy, which selectively retains experts that are most relevant to specific modalities, thereby optimizing both performance and resource usage.
Sources and reporting notes
Based on published source material. No independent hands-on testing is claimed.
