SHEPHERD: Serving DNNs in the Wild
SHEPHERD: Serving DNNs in the Wild
Hong Zhang,Yupeng Tang,Anurag Khandelwal,Ion Stoica
2023 · DBLP: conf/nsdi/0025TKS23
Symposium on Networked Systems Design and Implementation · 72 Citations
TLDR
S HEPHERD uses a novel online algo-rithm that provides guaranteed goodput under workload un-predictability by carefully leveraging preemptions and model-specific batching properties and achieves up to 18 .