Speculative Decoding
Speculative Decoding: 3x Faster LLM Inference Explained
Speculative decoding uses a small draft model to predict tokens that a larger model verifies in parallel, delivering up to 3x faster inference without sacrificing output quality. Here's how the technique works and why it matters for generative AI.