SHADOWPREFILL: Keeping Your Next Model Warm for Per-Step Model Routing in Agentic LLM Serving
arXiv:2611.13176v1 Announce Type: new Abstract: Per-step model routers send each LLM call of an agent to the cost-optimal model dynamically within each session.…