Orbit: Efficient Agentic Inference using Priority Scheduling

Abstract

Large language models (LLMs) are increasingly deployed as autonomous agents that execute complex tasks through long sequences of reasoning steps and tool calls. Serving these agentic workloads at scale is a growing priority, yet existing LLM inference engines are mostly designed for single-pass, chat-style interactions. Agentic tasks, however, can fail, and under system congestion, policies such as first come first serve (FCFS) allocate GPU resources equally to all tasks regardless of their likelihood of success, letting failing tasks inflate queue waiting times for tasks that will complete correctly. We introduce Orbit, a trajectory-aware scheduler that addresses this by dynamically deprioritizing tasks predicted to fail, shifting resources toward likely-correct tasks. We present two scheduling algorithms. Orbit-step is a lightweight heuristic that assigns priority inversely proportional to the number of steps a task has taken, leveraging the empirical observation that longer-running tasks are more likely to fail. Orbit-judge uses an asynchronous LLM-as-a-judge that reads partial execution trajectories and predicts task correctness online, enabling finer-grained prioritization. We evaluate both variants on the GAIA benchmark using the Magentic-One benchmark and the GPT-OSS-120B model. Compared to vLLM's default FCFS policy, Orbit-step and Orbit-judge show 11.2% and 4.3% decrease in average end-to-end latency, respectively, for correct tasks. These results demonstrate that real-time trajectory observation is a practical and effective signal for resource allocation in agentic serving systems.