Meta’s Orion holographic avatars will (eventually) be in VR too

The biggest reveal at Meta’s Connect event was its long-promised AR glasses, Orion. As expected, the prototype, each of which reportedly costs around $10,000, won’t be ready for the public any time soon.

In the meantime, Meta offered a glimpse of its new holographic avatars, which will allow people to talk with lifelike holograms in augmented reality. The holograms are Meta’s Codec Avatars, a technology it’s been working on for several years. Mark Zuckerberg teased a version of this last year when he participated in a podcast interview “in the metaverse.”

That technology may now be closer than we think. Following the keynote at Connect, I sat down with Mark Rabkin, a VP at Meta leading Horizon OS and Quest, who shared more about Meta’s codec avatars and how they will one day come to the company’s VR headsets as well.

“Generally, pretty much everything you can do on Orion you can do on Quest,” Rabkin said. The Codec Avatars in particular have also gotten much easier to create. While they once required advanced camera scans, most of the internal avatars are now created with phone scans, Rabkin explains.

“It’s an almost identical process in many ways in generating the stylized avatars [for VR], but with a different training set and a different amount of computation required,” Rabkin explained. “For the stylized avatars, the model has to be trained on a lot of stylized avatars and how they look and how they move. [It has to] get a lot of training data on what people perceive to look like their picture, and what they perceive to move nicely.”

“For the Codec avatars … it’s the same process. You gather a tremendous amount of data. You gather data from very high-quality, fancy camera scans. You gather data from phone scans, because that’s how people will be really creating, and you just build a model until it improves. And one of the challenges with both problems is to make it fast enough and computationally cheap enough so that millions and millions can use it.”

Rabkin said that he eventually expects these avatars to be able to play in virtual reality on the company’s headsets. Right now, the Quest 3 and 3S don’t have the necessary sensors, including eye tracking, necessary for the photorealistic avatars. But that could change for the next-generation VR headset, he said: “I think probably, if we do really well, it should be possible in the next generation [of headset].”