_1722956449156001fz1R.png "Anyscale Logo")
Engineering ML solutions at Workday has always been an interesting challenge due to our strongly tenanted architecture and our position as a processor of our customers' data, not an owner. Almost every model we design needs to be trained individually on each tenant's data and training must be done in many isolated pockets of the cloud with tightly-controlled data access permissions. With LLMs, this becomes even more challenging, in no small part due to GPU scarcity! To meet these demands in a cost-efficient way, we designed a platform around PEFT techniques (e.g. LoRA) and leveraged KubeRay's autoscaling capabilities to source GPUs when needed, then release them afterwards. As this scalable architecture supports both research and (re-)training, it paves a seamless path to production and broadens access to custom LLM-based features across the whole of the Workday product.
In this talk, we'll discuss how we use Ray at all scales to support a solution that's extremely flexible in how it's deployed, making full-stack development as accessible as full-scale production. We'll cover our LLM platform architecture and dive into lessons we've learned about building robust systems that idle with (nearly) no compute, but can easily scale to meet ephemeral demands for expensive resources like GPUs.
In this talk, we'll discuss how we use Ray at all scales to support a solution that's extremely flexible in how it's deployed, making full-stack development as accessible as full-scale production. We'll cover our LLM platform architecture and dive into lessons we've learned about building robust systems that idle with (nearly) no compute, but can easily scale to meet ephemeral demands for expensive resources like GPUs.
