Appendix: Recent Research

These are some notes from recent papers which don’t fit neatly into any of the sections because they require more context and framing to bridge the gap from the earlier, more fundamental works. The notes are less polished than the chapters and the hope is to eventually weave them into the broader narrative, but until then they’ll be collected here.

Routing and Scheduling

LMetric (OSDI’26) is a request routing/load-balancing system from Alibaba which has been deployed and serves production traffic. The deployment model is assumed to be with prefill-decode colocation and chunked prefills. Every instance has the entire model and can serve any request. In practice, this may be a group of hosts or GPUs the model is sharded across; however, from the perspective of traffic distribution it can be thought of as a single instance.

LLM routing is a classic data-aware routing problem which has to balance the twin concerns of locality and load. Serving instances keep the KV cache around after servicing the request for some time so that any potential follow-up requests get a prefix cache hit, which saves a significant amount of compute (cached prefixes are around a tenth of the cost of new input tokens at frontier labs’ API pricing). This is attractive for both multi-turn conversation chatbots and coding agents; on the other hand, purely prefix-cache based routing ends up with hotspots overwhelming individual instances.

There is a long body of work trying to balance these two parameters using threshold functions to move between data and load-based routing (called filter-based combinations in the paper) or using simulation to model the cost of recomputing prefill tokens and using that for decision making. Prior work from Alibaba extends basic shortest-queue routing with a scoring function for prefix hits, which uses hyperparameters to determine how heavily to weight prefix matches vs load.

All these methods require knowledge of the underlying model and hardware and need to be tuned extensively to work well. In contrast, LMetric uses a simple hardware- and model-agnostic function which is just a multiplication of the estimated number of prefill tokens (P) that an instance would have to calculate, including already queued requests and accounting for prefix cache hits, and the number of active requests at the instance, called the batch size (BS). The instance with the lowest ($P \times BS$) score is chosen since a low score indicates either less recomputation, lower load, or both.

This is almost deceptively simple with only two variables and no hyperparameter tuning, but works surprisingly well for the evaluated workloads. The pathological case is where some instances are repeatedly scored low, for instance, with near-complete prefix cache hits causing P and the score to be almost zero regardless of load, while the recomputation cost is high on other instances, so that all requests end up getting sent to the same few instances and overload them. The system detects and temporarily excludes hotspots or shifts to queue-length based routing in these cases.

GPU Sharing

Prism (OSDI’26) is a GPU memory management system designed to increase GPU utilization in multi-model serving workloads, such as those seen at Token-as-a-Service providers. These providers serve several models in parallel across their GPU fleet with models sharing GPUs; production traces reveal that the models tend to receive requests in short bursts punctuated with long idle intervals and that the popularity of models fluctuates over time. Time sharing approaches focus on efficiently swapping in and out model weights and violate SLOs when multiple models have overlapping bursts, while static space sharing focuses on efficiently sharing capacity for the KV cache across models and suffers from SLO violations during bursts since idle capacity cannot be reused.

Prism combines both time and space sharing approaches. It allows memory to be shared across different models for either model weights or the KV cache by virtualizing the physical address space of the GPU into a number of contiguous address spaces for the inference engines. It operates at page granularity and is agnostic to the contents of the virtual memory page, which is controlled by the engine; however, pages are not shared across models, which can result in internal fragmentation. Weights of inactive models are swapped out from GPU memory to host CPU memory.

Swapping models back in on demand can violate SLOs, so Prism optimizes this path by pre-allocating engines as well as reusable virtual address spaces for different models, rather than creating a new engine instance from scratch. Further, if multiple GPUs are available, it parallelizes loading model weights into GPU memory by partitioning the load across multiple GPUs in the same node to maximize the host-to-GPU bandwidth and then collecting the weights at the target GPU using NVLink interconnects. The KV capacity of other models is reduced to accommodate the swapped-in model, similar to classical memory ballooning techniques.

Finally, Prism is responsible for placement and scheduling decisions. It periodically sorts the models in descending order of their resource demands, calculated as a combination of load, KV size, and the TPOT SLO, and greedily assigns models to GPUs to reduce KV pressure. For scheduling, it holds a single, shared per-device request queue, and prioritizes running the maximum number of requests that can meet their TTFT SLO.