AI/ML Platform & Applications

BentoML or RayServe, You Can Choose Both with BentoRay

September 19, 3:15 PM - 3:45 PM
View Slides

Ray and BentoML are two popular libraries in the machine learning ecosystem, both beloved by developers for their unique strengths. Ray is particularly appreciated for its powerful architecture, which enables developers to parallelize and scale Python code with ease, without having to worry about the complexities of distributed computing. Meanwhile, developers love BentoML for its simplicity in developing, managing, and collaborating on AI applications. With BentoML, developers can build AI applications once and deploy them anywhere, without having to modify any code. The industry is now seeking a solution that can seamlessly integrate both libraries, enabling developers to streamline ML model training and AI application development.

In this talk, we will introduce BentoRay, a new project that enables developers to effortlessly run BentoML workloads (Bentos) on the RayServe architecture leveraging Ray's resource management and autoscaling capabilities in a Ray cluster. We will dive deep into the technical details of how BentoML's distributed runner architecture is transformed into Ray's deployment graph, and how BentoML's dependency environment is transparently mapped and bootstrapped in Ray's runtime environment. We will also demonstrate the simple steps involved in moving from a BentoML single container deployment to a distributed microservice based BentoRay deployment in a Ray cluster, allowing developers to seamlessly scale their workloads all without having to modify any code.

Overall, attendees will gain a comprehensive understanding of how BentoRay can help streamline the deployment of machine learning models and applications on Ray's powerful distributed computing framework. With BentoRay, developers can leverage the best of both worlds, combining the simplicity and collaboration capabilities of BentoML with the power and scalability of Ray.

About Sean

Sean Sheng is the Head of Engineering at BentoML supporting product design, development and roadmap. Prior to joining, he led engineering teams in the Service Infrastructure org at LinkedIn responsible for building the platform that powers all backend distributed services at LinkedIn.

About Chaoyu

Chaoyu is the CEO of BentoML and the original creator of the BentoML open-source project.

Sean Sheng

Head Of Engineering, BentoML

Chaoyu Yang

CEO, BentoML
Photo of Ray Summit pillows
Ray Summit 23 logo

Ready to Register?

Come connect with the global community of thinkers and disruptors who are building and deploying the next generation of AI and ML applications.

Photo of Ray pillows and Raydiate sign
Photo of Raydiate sign

Join the Conversation

Ready to get involved in the Ray community before the conference? Ask a question in the forums. Open a pull request. Or share why you’re excited with the hashtag #RaySummit on Twitter.