LLM Infrastructure

Breaking the KV Wall: Scaling LLM Inference at Scale

As LLMs handle longer contexts and more concurrent users, the KV cache has become the dominant bottleneck in inference. New architectural approaches aim to break through this memory wall for next-generation serving.

LLM Infrastructure

Moonshot AI's PrfaaS Rethinks LLM Serving at Scale

Moonshot AI and Tsinghua researchers unveil PrfaaS, a cross-datacenter KVCache architecture that decouples prefill from decode to dramatically improve LLM serving efficiency at scale.

LLM Infrastructure

Semantic Caching: Making LLM Embeddings Faster and Smarter

New research explores semantic caching strategies for LLM embeddings, moving beyond exact-match lookups to approximate retrieval methods that could dramatically reduce computational costs.

LLM Infrastructure

PROTEUS: Lagrangian RL Optimizes Multi-LLM Routing for SLAs

New research introduces PROTEUS, a reinforcement learning framework using Lagrangian optimization to intelligently route requests across multiple LLMs while meeting strict service level agreements.

LLM Infrastructure

AIConfigurator Speeds Up LLM Serving Optimization Dramatically

New research introduces AIConfigurator, a system that dramatically accelerates configuration optimization for multi-framework LLM serving, enabling faster deployment of AI inference infrastructure.

agentic AI

Agentic AI Scaling Demands New Memory Architecture

As AI agents tackle complex multi-step tasks, traditional memory systems are hitting fundamental scaling limits. New architectural approaches are emerging to handle persistent context across extended workflows.

LLM Infrastructure

Joint KV-Cache Encoding: A New Approach to Scalable LLM Serving

New research proposes joint encoding of KV-cache blocks to improve memory efficiency in large language model inference, addressing a key bottleneck in scalable AI deployment.

LLM Infrastructure

FlashInfer-Bench: New Framework Optimizes LLM Kernel Performance

Researchers introduce FlashInfer-Bench, a comprehensive benchmarking suite that creates a virtuous cycle for optimizing attention kernels in LLM serving systems, addressing critical infrastructure needs.

LLM Infrastructure

FPGA-Based CXL Memory Architecture Tackles LLM KV-Cache Bottlenec

New research proposes CXL-SpecKV, a disaggregated FPGA architecture using CXL memory pooling and speculative prefetching to overcome memory bottlenecks in large language model inference at datacenter scale.