Updated October 22, 2024

What We Learned

Background

The way people interact with computers has evolved considerably over the last few decades, transitioning from bulky mainframes to desktop PCs in millions of homes and offices. Improvements in processors and economies of scale have enabled the mobile computing revolution—dominated by laptops and then smartphones, smartwatches, and tablets.

Experts have pointed to an emergent class of mixed reality products that let users experience immersive worlds via wearable glasses or goggles as the next step in the evolution.

What’s the Difference?

The technology combines elements of virtual and augmented reality. The differences between these twin technologies are minor but important.

Virtual reality relies on technologies with screens where every pixel is generated, together making up a “virtual” world. In principle, traditional video games are examples of virtual reality. Advanced VR takes a display and translates it to high-tech goggles—like Meta’s Quest—where everything in a user’s field of vision is simulated. The result is the feeling of being fully immersed in a separate world. 

Augmented reality takes views of the real world and "augments" the user’s field of vision with virtualized content. For example, an app using the camera on a smartphone to show what a new sofa might look like in a living room.

Mixed reality products and software combine aspects of both AR and VR such that the user can interact with both physical and digital objects in real time.

Real-World Applications 

Apple’s new Vision Pro headset is one example of a mixed reality device, combining both AR and VR into a single experience.

The two micro-OLED displays seen through each eye of the headset present a combined 23 million pixels to the user. While a modern iPhone display features 460 pixels per inch, Vision Pro achieves 3,386.

As an example of real-world applications, some surgeons are now wearing Vision Pro headsets during surgeries. The surgeon uses the device’s virtual reality capabilities to walk through the entire surgery in a simulation before the real operation begins. At the same time, the augmented reality elements let the surgeon use informational overlays during the actual surgery—such as image scans or vital signs superimposed onto their field of view while wearing the headset.

A Bet on the Future

It remains to be seen whether mixed reality devices will become as widespread as other personal electronics like smartphones and smartwatches. Analysts say technology companies are positioning themselves under the assumption smartphones won't be a user’s primary computing device forever.

As processors become more powerful and companies and users crave experiences that can’t be confined to the flat screen of a smartphone, these new products have arisen to complement phones (and maybe replace them, eventually).

Factors holding back mixed reality from mass adoption, for now, include the high cost of hardware and limited content availability. Battery life—the Vision Pro only gets a couple of hours on a full battery—is also an issue, as is the discomfort that sometimes accompanies protracted use.

Dive Deeper

Relevant articles, podcasts, videos, and more from around the internet — curated and summarized by our team

Open link on podcasts.apple.com

Mixed reality headsets aren’t a mainstream consumer electronics product yet, but that hasn’t stopped a growing number of tech companies from continuing to release new ones. In this podcast interview, Paramount Pictures futurist Ted Schilowitz and Microsoft HoloLens director of Mixed Reality Greg Sullivan walk listeners through terms like VR, AR, and mixed reality, while also explaining how using products like Meta’s Quest can feel a little like having a superpower.

Open link on youtube.com

When mixed reality headsets are shown off, it’s often in the context of entertainment or personal use cases at home. Front-line manufacturing workers, meanwhile, are the backbone of many industries, and this video from Microsoft shows how a mixed reality headset can facilitate everything from those workers’ hands-on education to immersive guides overlaid onto a workers field of vision that support training and knowledge transfer across an organization.

Open link on youtube.com

Mixed reality opens up new possibilities for entertainment, particularly when it comes to video games. MR, for example, makes it possible to utilize a gamer’s actual, physical space as part of the gameplay. Hauntify is one example; this game, available via the Meta Quest’s app store, maps the gamer’s physical space, from which evil spirits seem to suddenly appear that must be repelled.

Open link on neurosciencenews.com

Mixed reality technologies aren’t just about fun and games. A new study, led by a team from Stanford University, shows that virtual reality can actually be a tool that helps successfully treat depression and improve mental health. This article explains how participants who were given VR headsets to engage in activities, like VR board games and dancing, saw reduced symptoms of depression that made their overall treatment more enjoyable.

Open link on youtube.com

Apple device owners already have a variety of screens in their lives, from iPhones to iPads. Vision Pro, the company’s new $3,499 headset, adds one more—this one, essentially strapped to the user’s face. Apple has touted the device as pointing toward a new generation of computing, and this Wall Street Journal video explains what it’s like to use the device for more or less 24 consecutive hours.

Open link on youtube.com

Regarding the educational side of mixed reality, headsets like Meta’s Quest offer plenty of opportunity for instruction and learning new skills. One such example is an app like PianoVision, which uses the passthrough visibility of the headset to illuminate which keys to play on the user’s actual piano or keyboard (alternatively, a virtual piano can be used).

Explore all Mixed Reality

Search and uncover even more interesting information in our vast database of curated Mixed Reality resources