Distributed Inference of Large Language Models on Edge Devices

We aim to propose a novel architecture to improve the performance of Large Language Models (LLMs) during inference on edge devices, particularly focused on mobile phones through a distributed computing environment. By utilizing the resources available across the local network, our architecture optimizes inference while maintaining the quality of generated text. Our study compares the number of nodes across the network while inferring an LLM and highlights the performance optimization in the average inference time per token with a greater number of nodes. We saw an improvement of up to 74% when the distributed system was scaled to four nodes. Our study advances the democratization of local LLMs and paves the way for research and applications that follow it.