_1722956449156001fz1R.png "Anyscale Logo")
Uber is actively embracing Generative AI to enhance user experience, productivity, and efficiency. Central to this effort is Michelangelo, Uber's centralized machine learning platform, which supports GenAI application development. This presentation delves into Michelangelo’s batch prediction for GenAI applications, leveraging Ray's parallel processing capabilities and vLLM's state-of-the-art natural language processing (NLP) performance.
Traditionally, Michelangelo has managed batch predictions using Spark data transforms, where model prediction is executed via a user-defined function within a Spark pipeline model. However, this approach has shown limitations in handling the increasingly complex GenAI tasks Uber is deploying, which often require GPUs to meet stringent latency and throughput requirements. Ray provides better resource utilization, simpler parallelism, and more straightforward scalability for GenAI tasks.
vLLM is at the forefront of generative AI batch prediction, offering significant advantages by generating high-quality predictions in parallel, drastically reducing latency and improving throughput. This makes it exceptionally well-suited for batch prediction tasks that demand quick, reliable, and scalable performance.
By integrating Ray and vLLM, we have developed a batch prediction workflow capable of handling various open-source models like LLaMA and Mixtral. This workflow unifies model parallelism, distributes tasks across large-scale instances, and processes massive amounts of data efficiently. Leveraging Ray and vLLM, we have created a scalable pipeline to support Uber’s GenAI applications, including Eats carousel, rider search, and customer obsession.
In this presentation, we will showcase the architecture of our batch prediction system, its integration with Kubernetes (K8s) and the Michelangelo LLM evaluation workflow, benchmarking results, and the issues and learnings encountered. We will illustrate how this combination accelerates prediction tasks, reduces latency, and scales seamlessly across clusters, highlighting its transformative impact on Uber’s GenAI capabilities.
Traditionally, Michelangelo has managed batch predictions using Spark data transforms, where model prediction is executed via a user-defined function within a Spark pipeline model. However, this approach has shown limitations in handling the increasingly complex GenAI tasks Uber is deploying, which often require GPUs to meet stringent latency and throughput requirements. Ray provides better resource utilization, simpler parallelism, and more straightforward scalability for GenAI tasks.
vLLM is at the forefront of generative AI batch prediction, offering significant advantages by generating high-quality predictions in parallel, drastically reducing latency and improving throughput. This makes it exceptionally well-suited for batch prediction tasks that demand quick, reliable, and scalable performance.
By integrating Ray and vLLM, we have developed a batch prediction workflow capable of handling various open-source models like LLaMA and Mixtral. This workflow unifies model parallelism, distributes tasks across large-scale instances, and processes massive amounts of data efficiently. Leveraging Ray and vLLM, we have created a scalable pipeline to support Uber’s GenAI applications, including Eats carousel, rider search, and customer obsession.
In this presentation, we will showcase the architecture of our batch prediction system, its integration with Kubernetes (K8s) and the Michelangelo LLM evaluation workflow, benchmarking results, and the issues and learnings encountered. We will illustrate how this combination accelerates prediction tasks, reduces latency, and scales seamlessly across clusters, highlighting its transformative impact on Uber’s GenAI capabilities.
