Mixture-of-Experts

EMS-FL: Federated Tuning of Mixture-of-Experts in Satellite-Terrestrial Networks via Expert-Driven Model Splitting

• Combines Mixture‑of‑Experts (MoE) with satellite‑terrestrial networks (STN) to overcome data scarcity and compute limits in federated learning. • Introduces EMS‑FL, an expert‑dri

Optimizing Communication for Mixture-of-Experts Training with Hybrid Expert Parallel

• In LLM training, Expert Parallel (EP) communication for hyperscale mixture-of-experts (MoE) models is challenging. • EP communication is essentially all-to-all, but due to its dy