Collaborative Multi-Device Edge Inference for Vision-Language Models with Speculative Decoding

Large language models (LLMs) have demonstrated remarkable success across various domains. However, the substantial computational and memory demands pose significant challenges for deploying LLMs at the network edge. To address this issue, the existing studies mainly focus on either reducing the size of LLMs or distributing LLMs across multiple devices. However, the former approach suffers from performance degradation, while the latter incurs high communication cost due to the transmission of high-dimensional intermediate features. To tackle these issues, we propose a novel collaborative framework to support vision-language model (VLM) inference at the network edge, expanding speculative decoding into a multi-device scenario. In this framework, each edge device first utilizes its small VLM to generate draft tokens in an auto-regressive manner. These tokens are then transmitted to an edge server, where they are corrected in parallel by a large VLM. By benefiting from speculative decoding, the number of calls to the large VLM is reduced without degrading the inference performance. Furthermore, we minimize the average latency of inference tasks by developing a deep reinforcement learning algorithm to optimize the number of draft tokens generated at each iteration. Simulation results confirm that the proposed algorithm achieves a lower average latency compared to other baselines.