KV Cache and Inference Scheduling: Energy Modeling for High-QPS Services

Abstract

High-QPS (Queries Per Second) services, such as large language model (LLM) inference and real-time recommendation systems, are increasingly pervasive in AI-driven applications, but their energy consumption has become a critical challenge—accounting for up to 40% of data center operational costs. Existing optimization efforts primarily focus on latency reduction and throughput improvement, overlooking the intricate interplay between KV cache management (a core component of transformer-based model inference) and inference scheduling in energy efficiency. Traditional energy modeling methods (e.g., linear regression, hardware-centric power meters) fail to capture dynamic dependencies between cache behavior, scheduling policies, and workload volatility, leading to inaccurate energy predictions and suboptimal resource allocation. To address these gaps, this study proposes a hybrid energy modeling framework tailored for high-QPS services, integrating KV cache characteristics and inference scheduling dynamics. First, we construct a multi-dimensional energy factor system encompassing four core dimensions: KV Cache Configuration (e.g., cache size, eviction policy, hit ratio), Inference Scheduling Strategy (e.g., batching size, task prioritization, resource partitioning), System Environment (e.g., CPU/GPU utilization, memory bandwidth, power capping), and Workload Traits (e.g., QPS volatility, request complexity, sequence length distribution). Second, we design a two-stage modeling approach: a data-driven component (Gradient Boosting Tree, GBT) to capture non-linear relationships between factor interactions and energy consumption, and an analytical component (Queueing Theory-based latency-energy tradeoff model) to ensure QPS and latency constraints are satisfied. Third, we validate the framework using a real-world dataset from an LLM inference service (2022–2024) with QPS ranging from 5k to 30k, comparing it against three baseline methods. Experimental results show that the proposed framework outperforms traditional models: it achieves an energy prediction accuracy of 92.7% (vs. 78.3% for linear regression and 83.5% for hardware-centric modeling), reduces energy consumption by 18.9%–25.3% while maintaining target QPS and latency SLAs (Service Level Agreements), and identifies key optimization levers (e.g., adaptive KV cache resizing based on QPS fluctuations reduces energy by 14.2%). This study provides a practical tool for system administrators to balance performance and energy efficiency, supporting the development of sustainable high-QPS AI services.