It can often be useful to leverage short-lived Ray clusters as a part of a broader ML workflow (ex: to perform distributed training as part of a pipeline with multiple data and modeling steps). Using these single-purpose, ephemeral Ray clusters can unlock opportunities for improved reproducibility, efficiency, and observability. KubeRay provides a natural way to manage these ephemeral Ray clusters on Kubernetes. Sematic (https://sematic.dev) shares learnings from leveraging KubeRay & Ray in this way.
Josh Bauer has written code for everything from particle accelerator data analysis for data from the LHC to ML infra for self-driving cars at Cruise. Most recently, he's been a founding engineer at https://sematic.dev trying to bring world-class ML workflow tooling to teams of all shapes and sizes.
Come connect with the global community of thinkers and disruptors who are building and deploying the next generation of AI and ML applications.