Ray Summit 2024

In this talk, we explore the integration and utilization of Ray within Roblox's machine learning platform to handle large-scale batch inference jobs effectively for deep learning models and LLMs, leading to a significant cost reduction. As an immersive platform for communication and connection, Roblox requires robust, scalable solutions to manage and deploy machine learning models efficiently across various use cases. We will delve into the specific applications of Ray at Roblox, highlighting its pivotal role in enhancing throughput and reliability in model batch inference scenarios. Our discussion will cover the architectural design decisions that led to the adoption of KubeRay, showcasing how it fits into our Kubernetes-managed environments to bolster our machine learning operations.