Immersive User Experiences: Trends and Challenges of Using XR Technologies

Extended reality (XR) is a range of technologies offering immersive and interactive user experiences across a continuum of virtual reality (VR), mixed reality (MR), and augmented reality (AR) devices. Specifically, VR technology-enabled device immerses the user in a virtual environment such that the user's sensory perception is completely synthetic. Devices with AR technology, on the other hand, superimpose digital information over the user's view of the real world, and the goal of AR is to enhance the user's experience with the real-world environment. MR devices include a combination of real-world environments with AR and VR technologies. In all these XR devices, there is a common set of perception workloads that need to be executed. These are predominantly computer vision tasks and typically involve tasks for simultaneous localization and mapping (SLAM) such as interest point detection, feature matching, and motion tracking, 3D reconstruction-related tasks such as depth estimation and scene understanding, virtual avatar-related tasks such as egocentric hand, eye, and/or body tracking. An XR application will often require many of these tasks to be run concurrently requiring excessive computing resources and resulting in high power consumption. Given the form factor of head-mounted devices (HMD) developed for XR, and limits on heat dissipation around the head of the user wearing the HMD, high power consumption and large computing resource for the application are often conflicting requirements. In this chapter, we will provide an overview of the perception tasks related to an immersive XR experience and discuss how through the use of dedicated hardware and distributed hardware some of these challenges can be mitigated.