Ray Summit 2024

Ray Serve and NVIDIA Triton Inference Server are two popular open-source inference serving solutions with unique capabilities. While Ray Serve is Python-first and allows for multi-process model composition and scaling, Triton Inference Server is primarily C / C++ and focuses on in-process model ensembles and optimized framework backends. Working closely together the maintainers of Ray Serve and Triton Inference Server collaborated to give users the best of both worlds. Starting in early 2024, Triton Inference Server includes a Python API that ​​allows developers to seamlessly embed Triton Inference Server inside their Python applications running on Ray Serve. For Ray Serve users, this allows them to improve the performance of their ML models as can be demonstrated with our stable diffusion demo. It also allows users to leverage advanced inference serving optimization and analysis tools like Performance and Model Analyzer. Users can use these to find optimal model configurations based on their application throughput and latency requirements. Triton Inference Server users can now leverage Ray Serve to build, load balance and scale complex multi-process applications all with the ease-of-use of Python. In this talk we’ll demonstrate how Ray Serve users can take models optimized for Triton Inference Server and quickly deploy and scale them within a Ray cluster.

We’ll discuss the challenges and opportunities in combining these Open Source projects to meet the demands of Generative AI applications.