Ray Deep Dives

Faster and Cheaper Offline Batch Inference with Ray

September 19, 2:30 PM - 3:00 PM
View Slides

The popularity of machine learning (ML) in the real world has exploded recently, with offline batch inference as one of the primary workloads. Yet existing production ML systems fall short for this workload in scale and simplicity. To address this, the Ray community has built Ray Data, an open-source library for building large-scale data processing for ML applications. In this talk we'll discuss:

• How to use Ray Data for efficient inference of Terabytes of data and a pretrained model

• Why traditional data processing tools are difficult, expensive, and inefficient, particularly for modern deep learning applications.

• How to easily leverage modern ML models, multiple times faster and cheaper than other common solutions (like Spark or SageMaker) using - Ray Data

• How and why offline inference can be useful with LLMs and when building LLM applications

• Demonstrate an e2e offline batch inference use case with Ray Data


• Ray Data is the best solution for offline batch inference/processing, particularly when working with unstructured data and with deep learning models

• Discuss why having a performant batch inference solution is important for LLM workloads, and show how Ray Data can help

• Share user success stories of using Ray Data for batch inference

About Amog

Amog Kamsetty is a software engineer at Anyscale where he works on building distributed ML libraries and integrations on top of Ray. He is one of the lead developers of Ray's distributed training and offline batch inference libraries.

About Balaji

Balaji Veeramani is a software engineer at Anyscale. He works on open source libraries built on Ray. Before joining Anyscale, he was a student at UC Berkeley.

Amog Kamsetty

Software Engineer, Anyscale

Balaji Veeramani

Software Engineer, Anyscale
Photo of Ray Summit pillows
Ray Summit 23 logo

Ready to Register?

Come connect with the global community of thinkers and disruptors who are building and deploying the next generation of AI and ML applications.

Photo of Ray pillows and Raydiate sign
Photo of Raydiate sign

Join the Conversation

Ready to get involved in the Ray community before the conference? Ask a question in the forums. Open a pull request. Or share why you’re excited with the hashtag #RaySummit on Twitter.