Real-Time Edge Classification: Optimal Offloading under Token Bucket Constraints

We consider an edge-computing setting where machine learning-based algorithms are used for real-time classification of inputs acquired by devices, e.g., cameras. Computational resources on the devices are constrained, and therefore only capable of running machine learning models of limited accuracy. A subset of inputs can be offloaded to the edge for processing by a more accurate but resource-intensive machine learning model. Both models process inputs with low-latency, but offloading incurs network delays. To manage these delays and meet application deadlines, a token bucket constrains transmissions from the device. We introduce a Markov Decision Process-based framework to make offload decisions under such constraints. Decisions are based on the local model's confidence and the token bucket state, with the goal of minimizing a specified error measure for the application. We extend the approach to configurations involving multiple devices connected to the same access switch to realize the benefits of a shared token bucket. We evaluate and analyze the policies derived using our framework on the standard ImageNet image classification benchmark.