• Computer Science > Distributed, Parallel, and Cluster Computing [Submitted on 25 Feb 2026] Title:DHP: Efficient Scaling of MLLM Training with Dynamic Hybrid Parallelism View PDF HTML (experimental)Abstract:Scaling long-context capabilities is crucial for Multimodal Large Language Models (MLLMs) • However, real-world multimodal datasets are extremely heterogeneous • Existing training frameworks predominantly rely on static parallelism strategies, which suffer from severe load imbalance, redundant communication, and suboptimal hardware utilization under data heterogeneity • In this work, we propose Dynamic Hybrid Parallelism (DHP), an efficient parallelism strategy that adaptively reconfigures communication groups and parallelism degrees during MLLM training • We generalize the non-power-of-two parallelism degrees and develop a polynomial-time algorithm to generate near-optimal parallelism strategies with only millisecond-level overhead per training batch • DHP is able to maintain high hardware efficiency even under extreme data variability

Article Summaries:

  • Computer Science > Distributed, Parallel, and Cluster Computing [Submitted on 25 Feb 2026] Title:DHP: Efficient Scaling of MLLM Training with Dynamic Hybrid Parallelism View PDF HTML (experimental)Abstract:Scaling long-context capabilities is crucial for Multimodal Large Language Models (MLLMs). However, real-world multimodal datasets are extremely heterogeneous. Existing training frameworks predominantly rely on static parallelism strategies, which suffer from severe load imbalance, redundant communication, and suboptimal hardware utilization under data heterogeneity. In this work, we propose

Sources: