ExpertFlow: Optimized Expert Activation and Token Allocation for Efficient Mixture-of-Experts Inference

This work introduces ExpertFlow, a comprehensive system specifically designed to enhance inference efficiency by accommodating flexible routing and enabling efficient expert scheduling between CPU and GPU, and implements a dynamic token scheduling strategy that optimizes MoE inference by rearranging input tokens across different batches.