Lightning Talks

Building a Distributed Query Engine with Ray

September 19, 2:15 PM - 2:30 PM
View Slides

In this talk I will describe Quokka, a distributed fault-tolerant pipelined SQL query engine built on top of Ray. Quokka uses Ray as a cluster manager similar to how Spark uses Yarn. Quokka delegates single node compute kernels to Polars and Apache Arrow, while relying on Redis for global coordination. To learn more about Quokka, please see Robert's tweet:, and the Github repo linked.

I will first describe the motivations of Quokka, and how it's designed to support feature engineering workloads on time-series data that current query engines don't support very well. These include esoteric window computations and special streaming joins typically seen in real-time systems for finance, recommendation or cybersecurity.

Typically, testing these real-time applications on historical data is difficult: people run historical data as streams through the streaming engine (e.g. Flink), which is not optimized for good performance on static data. Quokka speeds up these computations on historical data to allow rapid iteration of streaming pipelines, feature backfilling, and fast streaming failure recovery.

I will focus my talk on how Quokka uses Ray. I will showcase how the simplicity and stability of Ray allows Quokka's core execution engine to be only less than 1000 lines of Python. I will also describe alternative Ray designs that we considered and why we ended up on our current architecture, where long running Ray actors handle user-defined Quokka tasks.

I will end with a list of feature requests for Ray (which might be addressed before the talk even!), for example automatic swapping out of actors to disk instead of running out of memory.

Key takeaways:

  1. Ray is amazing. Its flexibility allows you to build distributed applications limited only by your imagination. Even if your current design doesn't quite work, you will eventually be able to use Ray to find a working design.

  2. Ray offers a lot of functionalities out of the box, like object communication and fault tolerance. However depending on your application you might want to build your own components while using the rest of Ray.

  3. Stream processing is hard -- the hardest part is testing your application on historical data. By offering a unified stream/batch API with custom optimizations, Quokka speeds up common stream processing primitives on historical data.

About Tony

I'm a PhD student at Stanford University. To see what I am working on, please check out Dr. Robert Nishihara's tweet:

Tony Wang

PhD student, Stanford
Photo of Ray Summit pillows
Ray Summit 23 logo

Ready to Register?

Come connect with the global community of thinkers and disruptors who are building and deploying the next generation of AI and ML applications.

Photo of Ray pillows and Raydiate sign
Photo of Raydiate sign

Join the Conversation

Ready to get involved in the Ray community before the conference? Ask a question in the forums. Open a pull request. Or share why you’re excited with the hashtag #RaySummit on Twitter.