• Computer Science > Distributed, Parallel, and Cluster Computing [Submitted on 19 Feb 2026] Title:Collaborative Processing for Multi-Tenant Inference on Memory-Constrained Edge TPUs View PDFAbstract:IoT applications are increasingly relying on on-device AI accelerators to ensure high performance, especially in limited connectivity and safety-critical scenarios. • However, the limited on-chip memory of these accelerators forces inference runtimes to swap model segments between host and accelerator memory, substantially inflating latency. • While collaborative processing by partitioning the model processing between CPU and accelerator resources can reduce accelerator memory pressure and latency, naive partitioning may worsen end-to-end latency by either shifting excessive computation to the CPU or failing to sufficiently curb swapping, a problem that is further amplified in multi-tenant and dynamic environments. • To address these issues, we present SwapLess, a system for adaptive, multi-tenant TPU-CPU collaborative inference for memory-constrained Edge TPUs. • SwapLess utilizes an analytic queueing model that captures partition-dependent CPU/TPU service times as well as inter- and intra-model swapping overheads across different workload mixes and request rates. • Using this model, SwapLess continuously adjusts both the partition point and CPU core allocation online to minimize end-to-end response time with low decision overhead.
Article Summaries:
- Summary
Researchers have introduced SwapLess, a system that dynamically balances inference workloads between CPUs and Edge TPUs to reduce memory pressure and latency. By modeling partition‑dependent service times and swapping overheads, SwapLess continuously adjusts the split point of a neural‑network model and the number of CPU cores allocated to each request. Experiments on Edge TPU‑equipped devices show that the approach cuts mean inference latency by up to 63.8 % for single‑tenant workloads and 77.4 % for multi‑tenant scenarios compared with the default compiler. The technique offers a lightweight, adaptive solution for high‑performance on‑device AI in memory‑constrained, multi‑tenant IoT environments.
Sources: