Large Language Models

Serving Large Language Models with KubeRay on TPUs

September 19, 3:15 PM - 3:45 PM
View Slides

Large language models (LLMs) have become increasingly prevalent in recent years, due to their ability to generate outputs that are close to indistinguishable from human-written text. However, serving these models have presented new technical challenges, as they contain hundreds of billions of model parameters and require massive computational resources. In this talk, we will discuss how to serve large language models using KubeRay on TPUs, and how it can help improve the performance of your models.

KubeRay is a Kubernetes operator that makes it easy to deploy and manage Ray clusters on cloud. Ray is an open-source framework for distributed machine learning. It enables ML engineers and data scientists to scale their workloads out to large clusters of machines.

TPUs are Tensor Processing Units, which are specialized processors that are designed for high throughput, large batch workloads. They are highly efficient at running neural network workloads, and can provide a significant boost in performance for large models.

By integrating KubeRay with TPUs, you can create a powerful and efficient platform for serving large language models. KubeRay will make it easy to deploy and manage your Ray cluster, and the TPUs will provide the performance boost that your models need.

In this talk, we will demonstrate the following benefits of using KubeRay with TPUs:

  • Increased performance: TPUs can provide a significant boost in performance for language models, which can lead to faster inference times and better results.

  • Improved scalability: KubeRay's autoscaling capabilities make it easy to scale out your Ray cluster as needed, so you can easily add more machines to meet demand.

  • Reduced costs: TPUs are more efficient at running batch workloads, which can lead to lower costs for running your language models.

  • Increased flexibility: KubeRay gives you the flexibility to choose the best hardware for your needs, so you can get the best performance at the best price.

  • Increased monitoring: Integration with Prometheus and other tools make it easy to collect performance metrics for your large language models in production.

About Richard

Richard Liu is a Senior Software Engineer at Google Kubernetes Engine, where his main area of focus is building ML experience on Kubernetes. He is the co-author of "Kubeflow for Machine Learning: From Lab to Production" (O'Reilly Media, 2020). Previously he has worked on the ML Platform at Waymo.

About Winston

Winston Chiang is currently the product lead for AI/ML on Google Kubernetes Engine. He has led cloud Machine Learning Platforms at both Google and Amazon. Winston completed his PhD in System Design Theory and MS in Computer Science (AI Robotics). In his extra time, he is the personal chauffeur for 3 young and energetic boys.

Richard Liu

Senior Software Engineer, Google

Winston Chiang

Product Manager, Google
Photo of Ray Summit pillows
Ray Summit 23 logo

Ready to Register?

Come connect with the global community of thinkers and disruptors who are building and deploying the next generation of AI and ML applications.

Photo of Ray pillows and Raydiate sign
Photo of Raydiate sign

Join the Conversation

Ready to get involved in the Ray community before the conference? Ask a question in the forums. Open a pull request. Or share why you’re excited with the hashtag #RaySummit on Twitter.