Ray Summit 2024

LLM inference is one of the most resource-intensive AI workloads today. Over the last year at Anyscale, we've focused on contributing critical advances to open source inference engines while building enterprise and production features on the Anyscale Platform. We've collaborated closely with the vLLM team to release key features such as FP8 support, chunked prefill, multi-step decoding, and speculative decoding. Together, these optimizations have improved vLLM performance by over 2x in both throughput and latency. We’ll also cover some of the optimizations we’ve made on the Anyscale Platform as well, including custom kernels, optimizations for batch inference, and accelerated large model loading for autoscaling deployments.