Introduction

In medicine, the trainee must acquire significant experience to obtain the knowledge base necessary to identify, interpret, and correctly manage findings or perform procedures. Additionally, an instructor must give up control of the patient interaction for the trainee to build technique and acquire mastery [1]. Trainees typically complete tasks at a slower pace than the experts, which consumes time and reduces care efficiency as experts are ‘slowed down’ [1]. However, the highest concern is the risk to patient safety as trainees are asked to learn new procedural skills in high-stress clinical settings. The practice of new skills on patients by novices may result in inappropriate application of procedures, misdiagnoses, lower rates of procedure completion and higher rates of complications, all of which could jeopardize patient safety [2,3]. Patient safety and comfort should not be compromised for the purpose of training. In consideration of medicolegal consequences, training on patients is becoming less acceptable during the early stages of training [4]. Furthermore, although instructors may be widely available in a resource-rich setting allowing multiple instructors for any quantity of trainees, resource-constrained settings are limited by a dearth of experts. Therefore, experienced clinicians in resource-limited areas may be challenged in balancing clinical and instructional responsibilities.

Advances in technology can transform medical education and augment current training methods for trainees. In the Institute of Medicine Report, ‘To Err is Human’, interactive digital technology for medical education was proposed as an efficient application to allow skill acquisition and the practice of invasive procedures without the risk of harm to a patient [5]. Simulation-based technology creates a learner-centred model [1]. Supervisors will have the confidence and time to focus on the needs of the trainee rather than emphasizing the comfort of the patient [1]. Additionally, this ensures a cost-effective and interactive learning experience for trainees [6]. According to the Institute of Medicine Report, lapses in patient safety, particularly procedure-related complications, account for nearly $9 billion in healthcare spending and morbidity or mortality to 7.5 million patients annually [5]. The cost of acquiring and implementing interactive technology can be offset by the overall savings within a year. Immersion Medical’s Laparoscopy AccuTouch System, a virtual reality (VR) laparoscopy training system, had a return on the initial investment in approximately 6 months. Similarly, the endoscopy AccuTouch VR simulator was estimated to accomplish a return on investment in approximately 131 days by instructor time savings, error reduction, increased trainee efficiency during live procedures and reduced breakage of endoscopic equipment during live training [7].

In this report, we discuss the various modes and applications of extended reality, reflections from the National Aeronautics and Space Administration (NASA)’s contributions to training utilizing extended reality, and the limitations and future of extended reality in medical training.

Modes of extended reality

Multiple types of extended reality have been developed to enhance the user experience and allow for immersion in the subject at hand. These tools have been implemented in medical education and are described as follows:

Augmented reality

Augmented reality (AR) is another educational advancement that allows a trainee to observe instruction overlaying a live background through a lens, which may be the camera of a smart device, allowing for a budget-friendly option, or an advanced pair of glasses. Glasses-based lenses allow a hands-free option for a trainee to practise procedural techniques on live models or simulation-based mannequins while receiving live guidance through the AR lens. The live guidance may be in the form of a pre-installed AR application offering directions on screen (Figure 1) or via live remote guidance and feedback by an instructor observing from the same point of view as the trainee.

Figure 1:

Augmented reality procedural training. Augmented reality application offering directions on how to perform laboratory procedures through glasses-based lenses.

Virtual reality

VR employs an immersive experience to simulate various scenarios that may be encountered by a trainee. VR applications function entirely without the requirement of a live or mannequin-based model and can be uploaded into any device capable of VR; therefore, removing the need to replace models. It may use a combination of visual and haptic interfaces to rehearse the cognitive and technical skills required for a procedure [1]. VR creates a learner-centred environment for novices to practise skills in a high-stress environment without the concern for mistakes or harming a live patient and provides a means for standardization of feedback through objective measures [1,8–11]. In this context, interactive VR also allows for gamification, which can adapt to the level of expertise of the trainee. For example, trainees may start at the beginner level and as they complete each level, according to defined objective metrics, the trainee may advance until the increasingly difficult scenarios are overcome [8,12]. Achievement of increasingly difficult scenarios and satisfactory completion of objective measures can increase competence and instill confidence prior to exposure to live patients. Since the intention is to introduce VR prior to live patient exposure, there are improvements in safety and comfort for patients as trainees will have acquired the skill sets necessary to practise on the live patient. With VR, an instructor can allow a trainee to experience the full range of errors and allow these errors to progress, which may rearrange future behaviours by the trainees to avoid and resolve potential mistakes [1,3,13,14]. Not only may the technology be used to provide instruction to novice trainees, but also provides advanced professionals with ‘just-in-time’ training, or refreshers, as well as patient-specific rehearsal for procedures that may not be commonly performed [6].

The advantages and disadvantages of each modality of training discussed above in detail are summarized in Table 1.

Table 1:

Advantages and disadvantages for each modality of training.

	Advantages	Disadvantages
Physical reality	• Hands-on training • Most effective for gaining expertise	• Risk, patient safety • Cost-efficiency
Smartphone-based	• Interactive didactic material • Cost-efficiency • Video conferencing	• No ‘hands-free’ options
Augmented reality	• Remote guidance • Hands-free options • Use in live cases • Confidence improvement • Cost-efficient (handheld device applications)	• Distractibility during cases • Lack of availability of platforms and devices
Virtual reality	• Low patient risk • Immersive procedural training • Preoperative rehearsal • Confidence improvement • Reduction of procedural time • Cost-efficient • Adaptability • Objective measures available	• Cannot be used during a live case • Lack of availability of platforms and devices

Medical and laboratory applications of AR and VR training methods

Multiple companies have employed interactive technology in programs for gaming as well as training. NASA has been at the forefront of developing extended reality training aids. NASA uses VR to address situational awareness during astronaut-controlled robotics operations in space. A VR platform was developed to simulate astronaut control of the Mobile Servicing System Remote Manipulator System during on-orbit operations [15]. Therefore, VR training through the platform may increase crew efficiency and reduce the risk of collision during true operations [15]. The success that NASA has had in developing training aids, particularly pertaining to interactive technology, is ingrained in the development of advanced training mechanisms for medical trainees. Performing complex tasks in space without the availability of the abundant resources on Earth requires immense pre-launch training participation and refreshment of these skills during flight. NASA has employed its understanding of interactive technology to provide medical education to crew members to ensure they are better equipped to handle minor medical emergencies [16,17]. NASA has developed and studied a stand-alone skill enhancement software tool for performing ultrasound examinations in orbit on the International Space Station (ISS). It has been shown to significantly improve performance when used pre-launch and reviewed again in-flight prior to performing the examination as a ‘just-in-time’ refresher [16,17].

Similarly, medical educators can pull from innovative concepts used in pre-launch and in-flight training for astronauts to develop platforms capable of educating medical trainees on medical examinations and procedures prior to exposure to live patients. These advancements in medical education are already present and being evaluated. At Baylor College of Medicine, we have developed an application designed for smart devices to provide interactive instruction for gastroenterology fellows on the basics of performing upper endoscopy (Figure 2). To provide technical training on endoscopy, we have developed a VR platform designed to simulate endoscopy with high-fidelity graphics and controls to instruct novice endoscopists step-by-step on the manoeuvres and controls required to successfully perform an upper endoscopy (Figure 3). Our preliminary surveys before and after the use of the didactic application and VR tools indicated that trainees had increased confidence in their endoscopic skills following approximately 30 hours of training on the VR endoscopy application supplemented by unlimited access to the didactic endoscopy application [18]. In addition, we have created a VR walkthrough of the disinfection process in an isolation clinic simulating potential exposure to highly contagious pathogens (Figure 4). This provides a walkthrough tutorial of the steps for disinfection of the room as well as a virtual indication of the location of the essential equipment. These pieces of training are aimed at equipping a learner with didactic and cognitive materials, technical practice and guidelines, environmental experience, and situational exercise without patient involvement.

Figure 2:

Endoscopy didactic application. Didactic application designed for smart devices to provide interactive instruction for gastroenterology fellows on the basics of performing upper endoscopy.

Figure 3:

Baylor College of Medicine Endoscopy VR application. Virtual reality platform designed to simulate endoscopy with high-fidelity graphics and controls to instruct novice endoscopists step-by-step on the manoeuvres and controls required to successfully perform an upper endoscopy.

Figure 4:

Isolation clinic and disinfection procedure. Virtual reality walkthrough of the disinfection process in an isolation clinic simulating potential exposure to highly contagious pathogens.

While our interactive smartphone-based, AR and VR platforms and applications are still under development, multiple reports have demonstrated the benefits of interactive technology as a learner-centred approach to medical education. Gastroenterology is a procedural field in which there is significant concordance concerning the advantages of interactive technology prior to practice on a live patient. Mahmood et al. summarized the results of 21 randomized controlled trials and commented that 16 of the randomized controlled trials reported positive results for simulation-based training including higher achievements in distance advanced with the scope in the colon in colonoscopy, comprehensive assessment of applicable technical and non-technical skills, as well as patient comfort [8]. Similarly, simulation-based training has demonstrated an advantage in robotic surgery training [19–23]. Lerner et al. and Korets et al. assigned trainees to groups with one group training on the da Vinci-Trainer, a VR trainer and another group training on da Vinci dry laboratory equipment [19,20,23]. The groups participated in a pre-test to evaluate baseline robotic surgery skills. Groups were then allowed to train with their respective assigned equipment, which was followed by a post-training test to evaluate for improvement in skills from the pre-test. Both studies agreed that experience with the da Vinci-Trainer improved performance on the actual da Vinci Surgery System similar to that observed with traditional robotic dry laboratory simulation training [19,20,23]. However, Rehman et al. also described that VR simulators could save approximately $622,784 over 1 year if used to replace robotic dry laboratory training systems; therefore, conferring a cost–benefit advantage to simulation-based training models [24]. There are also positive results observed for mobile applications and AR simulators in medical education. The BIFURCAID mobile application has been developed to provide instruction on complex coronary bifurcation intervention and has garnered over 2,170 downloads as of January 2018 [25]. According to a survey of 103 users, of which 74% of respondents are practising interventional cardiologists, the mobile application helped improve knowledge (87%) and procedural practice (86%) of coronary bifurcation interventions and demonstrated an ‘Excellent’ rating based on the standard System Usability Scale, all of which indicate a positive user and educational experience for learners [25]. Additionally, the ImmersiveTouch AR simulator has been employed in multiple surgical disciplines and has been reported as a valid training method with a positive learning effect by accelerating the learning curve in various neurosurgical procedures including an increased ability to locate anatomical landmarks for interventional techniques as well as performing ventriculostomy and vertebroplasty procedures [26–28]. For example, Yudkowsky et al. identified, using pre-test and post-test assessment data of 16 neurosurgical residents, that following training with the ImmersiveTouch AR platform, there was a 32% probability of successful cannulation for ventriculostomy versus the 12% probability of successful cannulation prior to the AR training [27]. Furthermore, in consideration of the novel mechanics and objective advantages of interactive technology, the technology has garnered support from trainees. In a survey assessing the readiness of trainees to use simulation-based technology for procedural training, 91% of the participants agreed that VR simulators would have utility in their training [29,30]. This indicates that interactive smartphone-based AR and VR technology is viewed favourably by trainees in the improvement of decision-making ability.

There is rapid development and acceptance of simulation-based technology in resource-rich areas. However, the potential for simulation-based technology in resource-constrained settings is similarly significant. NASA has been at the forefront of this technology, and interplanetary space flight may be considered the ultimate remote location, where autonomous medical management is required due to communications delays. On Martian missions, Earth Independent Medical Operations (EIMO) are required because the communication latency may be as long as 23 minutes. AR scripts must be pre-loaded or uplinked on a non-emergent basis as real-time remote control or telemonitoring is not possible. As exemplified above, NASA has integrated EIMO technology by developing a software tool for performing ultrasound examinations on the ISS, which may serve as a ‘just-in-time’ refresher [16,17]. Caregivers in any remote, resource-constrained environment, whether on Earth or in space, may encounter medical emergencies with which they are unfamiliar or unprepared to manage. A library of just-in-time training scripts and real-time procedural guidance using AR or VR technology may allow the provider to save lives in situations where appropriate expertise and evacuation to a higher level of care are not available. A remote provider may use VR technology as an initial learning tool or rapid refresher on the technical skills for performing a non-urgent examination or procedure. However, AR technology can be used in urgent scenarios in which an examination or procedure must rapidly be performed. The remote provider may either utilize just-in-time training scripts allowing for the developed software to provide instruction or telementoring in which a provider receives live guidance from an expert in a remote location. As an example, a provider in a remote area on earth or in space may employ AR technology using either pre-uploaded software scripts or telementoring to perform a focused assessment with sonography for trauma examination to rapidly learn how to use an ultrasound machine and identify free intra-abdominal fluid or cardiac complications in the setting of trauma. Remote AR guidance has been reviewed and has demonstrated promising results. Bui et al. in a systematic review of 22 studies report that AR-dependent telementoring generally engendered positive feedback [31]. Eleven of the studies agreed that the AR system allowed for the effective performance of the procedure. Telementoring improved confidence and aided the trainee in diagnosing or performing the given task. Additionally, the rate of errors in performing a task reduced significantly from pre-test to post-test with telementoring [31].

Challenges and future directions

Simulation-based technology opens the door to an array of advancements in medical education. There is immense scalability with an opportunity to simulate a range of environments and situations within a single platform. Implementation of simulation-based teaching strategies is progressing in practice in resource-rich settings, but the opportunities for this technology in low-resource settings are, yet, largely untapped. A novice with access to a simple smart device can download an interactive programme to acquire a basic understanding of a medical operation. However, a novice can also acquire step-by-step training and pre-procedural rehearsal with immersive technology involving extended reality. Therefore, any trainee with access to this technology may obtain extensive instruction prior to exposure to a live patient in the absence of sufficient instructors and resources. An important limitation to consider is the cost and access to the technology and platforms. However, technology has advanced so that a simple smartphone or similar smart device may run these platforms, and wearables including lenses and VR devices are becoming progressively advanced as well as less expensive. The cost of high-end VR hardware including laptop and headset is approximately $4,000 and basic headsets including the Oculus Quest 2 can be purchased for $300, which may be used continuously throughout the lifetime of the product [32]. The cost of software for the VR set-up will vary by provider and quality of the product, but the software is commonly significantly less expensive than a physical simulation setup, regardless of the provider [32]. Although high-fidelity simulators are becoming increasingly advanced at a fraction of the cost, there are still challenges to replicating all the variables a trainee would encounter during a patient interaction [8]. Widespread deployment of the technology may still be limited by multiple factors outside of cost and generalizability. The climates in low-resource settings may increase failure rates and require air conditioning to maintain the equipment in working conditions, particularly in humid regions. Software bugs requiring rebooting of the software and interfering with the time-efficient use of the platforms as well as identifying trained technicians to maintain the upkeep of the equipment may be of concern. Security concerns may also play a role as there is a risk of loss or theft of valuable equipment. Moreover, as medical education trends towards a competency-based framework, a challenge may be deciphering at what level of competency in didactics and simulation-based models a trainee must possess to train on a live patient [8].

Conclusions

NASA has been at the forefront of simulation-based technology in its effort to train astronauts for autonomous operations and deliver remote guidance. We are seeing tremendous advancements to use this technology to transform medical education. Simulation-based technology offers an immersive learning experience that may augment current training methods. It creates a learner-centred environment in which novices can practice without the concern for harm to a live patient and allows for the standardization of feedback through objective measures. Regardless of the location of implementation, reductions in medical errors and improved confidence and efficiency in trainees may be observed. The cost–benefit is significant as the equipment may be used without limit throughout its lifetime, and software can be adapted to new and/or evolving conditions or scenarios. Additionally, simulation-based technology may have the highest benefit in resource-limited areas that lack educational resources including medical equipment and medical experts. Simulation-based technology can offer initial procedural training for a novice when no expert is available on-site. Furthermore, it allows any individual (novice to advanced) to acquire just-in-time training or refreshers for any task, in any environment. Nevertheless, the technology does have some limitations. The software should faithfully simulate the task, and the trainee must have access to and a means of operating and maintaining the technology. As technology advances, simulation-based training may instil a new standard of safety and competence and become an integral part of medical training across the globe.