LLM Infrastructure

PROTEUS: Lagrangian RL Optimizes Multi-LLM Routing for SLAs

New research introduces PROTEUS, a reinforcement learning framework using Lagrangian optimization to intelligently route requests across multiple LLMs while meeting strict service level agreements.

LLM Infrastructure

AIConfigurator Speeds Up LLM Serving Optimization Dramatically

New research introduces AIConfigurator, a system that dramatically accelerates configuration optimization for multi-framework LLM serving, enabling faster deployment of AI inference infrastructure.

Agentic AI

Agentic AI Scaling Demands New Memory Architecture

As AI agents tackle complex multi-step tasks, traditional memory systems are hitting fundamental scaling limits. New architectural approaches are emerging to handle persistent context across extended workflows.

LLM Infrastructure

Joint KV-Cache Encoding: A New Approach to Scalable LLM Serving

New research proposes joint encoding of KV-cache blocks to improve memory efficiency in large language model inference, addressing a key bottleneck in scalable AI deployment.

LLM Infrastructure

FlashInfer-Bench: New Framework Optimizes LLM Kernel Performance

Researchers introduce FlashInfer-Bench, a comprehensive benchmarking suite that creates a virtuous cycle for optimizing attention kernels in LLM serving systems, addressing critical infrastructure needs.

LLM Infrastructure

FPGA-Based CXL Memory Architecture Tackles LLM KV-Cache Bottlenec

New research proposes CXL-SpecKV, a disaggregated FPGA architecture using CXL memory pooling and speculative prefetching to overcome memory bottlenecks in large language model inference at datacenter scale.