Augmented reality and mixed reality: Extending the real world

In short: Augmented reality overlays digital content on the real environment, virtual reality replaces it entirely, and mixed reality lets both interact.

In more detail: VR (virtual reality) puts you with a closed headset into a purely digital world. AR (augmented reality) overlays graphics on the image of reality, for example on a phone display. MR (mixed reality) goes further: digital objects recognise the room, disappear behind furniture and can be touched.

In Depth

Technology

• Camera and sensors capture the surroundings: motion sensors (IMU), depth cameras or LiDAR, infrared.
• SLAM (simultaneous localization and mapping): the device builds a map of the room and at the same time knows where it is in it.
• Rendering of graphics exactly to the viewing direction (low latency, otherwise you feel sick).
• Display: smartphone display, pass-through (cameras show the surroundings on displays in the headset) or see-through optics (waveguides) as in HoloLens.

Applications

• Phone AR: placing furniture virtually in the room, filters, navigation with arrows, games (Pokémon GO).
• Industry: assembly instructions and maintenance with overlays, remote help from experts.
• Medicine and training: surgery planning, anatomy, training dangerous procedures.
• Architecture and design: viewing models at 1:1 scale.
• Headsets such as Apple Vision Pro or Meta Quest (with pass-through) bring MR into everyday life.

Development

Frameworks: ARKit (iOS), ARCore (Android), WebXR (in the browser), engines such as Unity and Unreal. Important are performance, legibility of overlays, safety (no distraction while walking) and operation without a mouse.

Privacy and health

Cameras capture homes, people and surroundings; check where data is processed. Longer use can lead to fatigue, dizziness and eye strain, so plan breaks. For children the manufacturers’ age recommendations apply.

See also: VR headset, Smartwatch and fitness band