Poster: Distributed Pipelined Collaborative Inference for LLMs on Edge Device Clusters

The Large language models (LLMs)-based intelligent applications are increasingly executed on user-owned edge devices such as phones, gateways, and smart embedding devices [2]. A key challenge for efficient on-device LLM is the memory wall. On resource-limited edge devices, model with billions of parameters and scaling KV cache often cannot remain resident in fast device memory. Existing solutions migrate part of the burden from GPU to CPU, but this usually inflates per-token latency during auto-regressive decoding stage due to repeated movement of CPU-GPU transfers. Collaborative inference across a group of nearby idle devices offers a direct way to break this constraint. However, conventional tensor parallelism [1] is poorly suited to this case since its frequent all-reduce collectives introduces significant latency. In this poster, we develop and validate a real-time collaborative inference design for edge networks based on pipeline parallelism. We partition the transformer decoder horizontally by layers, assign each device a contiguous block of layers, and execute the blocks as a distributed pipeline. During autoregressive decoding, each generated token flows stage-by-stage through the pipeline: a device runs its local forward computation, updates and retains the KV cache for its resident layers, and forwards only the intermediate activations to the next device. This architecture has the following two main advantages. First, each device holds only a fraction of the layers and their cache state, avoiding heavyweight swapping. Second, pipeline parallelism supports heterogeneous devices through flexible partitioning.