AI/ML Platform & Applications

Build Augmented Reality Maps With Ray

September 19, 1:00 PM - 1:30 PM
View Slides

Niantic is a leading augmented reality company. The AR mapping organization at Niantic is responsible for building and serving the best 3D maps to users all around the world through Niantic Lightship visual positioning system (VPS).

VPS is like GPS for the virtual version of the outside world, only far more precise, down to centimeter. With VPS developers can build persistent AR content that is anchored at real-world locations. This means people can have more meaningful and compelling immersive AR experiences in the real world because the visuals are blended perfectly.

We leverage Ray to solve 3 problems: 1. Research to production 2. Scan processing 3. Map building.

1: VPS is new and we need to constantly ship research outcomes into production. Often researchers, CV engineers and server engineers use different development environments, codebases and toolchains. Such gaps make debugging, testing and deployment difficult and extremely painful. With Ray we are building a serverless framework which different groups can run without code changes which fills the gaps.

2: Millions of people playing Niantic titles like Pokemon Go and Ingress upload a large volume of scans of places of interest (POIs) from all around the world to Niantic servers, every minute. There are a number of steps in scan processing, including blurring potentially recognizable objets like faces and license plates, running AI models for quality control, generating scan previews and etc. Different steps run on different hardware platform, e.g. CPUs and GPUs. Low overhead scheduling, efficient data passing between machines (because the scans files could be large) are critical for good performance and low cost, which Ray is good at.

  1. Creating 3D maps from scans is a complex process. There are number of steps involved - splitting scans, location mapping, connecting relevant maps, 3D dense reconstruction, context + semantics, map coordinate system accuracy improvement, merging connected maps etc. The overall pipeline is a large distributed system and some individual steps are also distributed. Easily connecting different components and distributing workloads, efficiently passing data around and scheduling different steps onto different hardware platforms are very important. With Ray we are able to make the pipeline simple, efficient and performant. What's more, with Ray we can also make overall system highly composable. Each step can be made a building block that can be run both locally and remotely, and users can easily assemble different building blocks to achieve their goals.

About Qi

Qi's a software engineer at Niantic, leading AR mapping infrastructure work. Qi has experience with large scale distributed system, machine learning infrastructure and performance optimizations. Before Niantic, Qi's a tech lead at Facebook AI, leading PyTorch productionization.

Qi Zhou

Staff Software Engineer, Niantic
Photo of Ray Summit pillows
Ray Summit 23 logo

Ready to Register?

Come connect with the global community of thinkers and disruptors who are building and deploying the next generation of AI and ML applications.

Photo of Ray pillows and Raydiate sign
Photo of Raydiate sign

Join the Conversation

Ready to get involved in the Ray community before the conference? Ask a question in the forums. Open a pull request. Or share why you’re excited with the hashtag #RaySummit on Twitter.