Ray Summit 2024

As large language models emerge, multi-GPU inference becomes a necessary requirement for model serving libraries. Due to the dynamic nature of distributed inference, it poses a different set of challenges from distributed training. The talk covers differences between distributed train and inference, how different parallelism strategies such as tensor parallelism, pipeline parallelism, expert parallelism, work in detail, and how to build an optimized architecture for a fast distributed inference engine, with vLLM as an example.