Quantized Mixture-of-LoRA-Experts for Low-Cost Training of Large Language Models

This work demonstrates the potential of combining quantization with mixture-of-experts routing for efficient and effective adaptation of large language models, and it is hoped this work will inspire further research in this direction.