A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing

This paper analyzes production LLM serving workloads over one year, revealing how traffic patterns and model usage have evolved, and proposes improved caching and load-balancing strategies to optimize inference performance and cost. The findings provide empirical data on real-world LLM deployment challenges, including token cache management and request distribution across inference clusters.

This paper analyzes production LLM serving workloads over one year, revealing how traffic patterns and model usage have evolved, and proposes improved caching and load-balancing strategies to optimize inference performance and cost. The findings provide empirical data on real-world LLM deployment challenges, including token cache management and request distribution across inference clusters. Enterprise leaders deploying LLMs at scale will find actionable insights on infrastructure efficiency and cost reduction.

Cloud-native AI architecture choices made today will shape flexibility and cost for years. QuettaMinds designs systems that avoid vendor lock-in while maximizing the infrastructure investments clients have already made.