Background

One of the recurring necessities in simulation instructional design is the need to create educational experiences that are appropriately contextualized to participants based on their background, level of training and current clinical roles. Such instructional design elements can include characteristics of the simulated patient, environment or both. Additionally, such design elements may be characterized as intrinsic or extrinsic. Intrinsic factors are those that are inherent and unmodifiable in task complexity, such as a more advanced skill requirement or varied background or experience levels of learners. Extrinsic factors are those related to differences in content presentation (e.g.external auditory or visual stimuli, differences in conditions of observation, physical obstacles, etc.). Given that different groups of participants will have different needs, simulation instructors may be called upon to adjust or amend scenarios to most closely align their content with learning objectives that are relevant to groups with differing backgrounds and experience levels.

In 2018, a scientific statement from the American Heart Association summarized the current state of resuscitation education, including important knowledge gaps and research priorities [1]. In a section of this statement dedicated to contextual learning, evidence summaries were presented for maximizing the relevance of education to clinical practice, as well as the role of stress and cognitive load. Simulation scenarios with increased levels of complexity can be construed as an element of instructional design that addresses both of these constructs. Both of these topics were identified as areas in need of further research.

For the present systematic review, we sought to examine whether simulation scenarios administered with varied levels of complexity led to differences in educational outcomes among participants.

Methods

The review was commissioned by the Society for Simulation in Healthcare as part of a global initiative to derive best practice guidelines in relation to the design and delivery of simulation activities. It was planned, conducted and reported in adherence with PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) standards of quality for reporting meta-analyses (see PRISMA diagram, Figure 1) [2]. Because this is a systematic review of published studies, no review board approval was required.

Figure 1:

PRISMA diagram.

Results

The search identified 15,991 articles; after removal of duplicates, 15,348 titles and abstracts were screened. Of these, 15,195 were excluded, leaving 149 full-text articles to be screened for eligibility (Figure 1). In total, 21 studies were identified for inclusion [11–31]. Three RCTs [17,21,23], 5 randomized crossover trials [11,12,15,22,25], and 13 observational studies [13,14,16,18–20,24,26–31] comprised the final set of included studies. Six studies were conducted in the United States [13–16,27,30]; six studies in Canada [11,17,18,21–23]; eight studies in Europe [12,19,20,24–26,28,31] and one in Asia [29]. Nine studies were conducted with physicians [11,18,20,21,25,27,28,30,31]; six were conducted with medical students [13–15,17,24,29]. Two studies involved prehospital providers [12,22] and two involved nursing students [16,26]. The remaining studies involved multiple provider groups [19,23]. There were no included studies with laypeople as subjects.

Study design details are reported in Table 1; there was marked heterogeneity between studies with regard to the study design, the selected interventions to assess the skills, knowledge and affective domains of participants, and the manner in which outcomes were reported.

Table 1:

Data extraction from included studies.

Study	Study design	Population and setting	Number of participants	Description of control group	Description of intervention group	Outcome measures (S = skill; K = knowledge; A = affective)	Results
Bensouda, 2018	Randomized crossover trial	Junior doctors, Canada	25 per group (50 total)	Simulated neonatal intubation with single observer (condition A)	Audience of 5 observers (condition B)	S (time to intubation)	Time to intubation: Condition A: 31 s (28–34 s); Condition B: 35 s (32–38 s); p = 0.07
Bjorshol, 2017	Randomized crossover trial	Paramedic teams, Norway	19 teams	Simulated field setting, no additional people present	Distracting bystander (non-native language speaking, ongoing added comments to increase pressure)	S (CPR quality, time to critical interventions); A (self-reported workload (NASA TLX))	CPR quality: no significant difference; Time to defibrillation: shorter in intervention group (121 s vs. 209 s; p < 0.001); NASA TLX: increase in scores for mental demands, time pressure, frustration, and effort
DeMaria, 2016	Observational study	Medical students, USA	26 (13 per group)	ACLS scenario with patient survival	ACLS scenario with patient death	S (clinical performance in ACLS scenario at 6 months); K (ACLS written test at 6 months); A (impact of event scale at 6 months)	Skill: Clinical: no difference (75 vs. 83%; p = 0.18); Knowledge: no difference (80 vs.78%; p = 0.89); Impact of Event Score: no difference
Evans, 2017	Observational study	4th year medical students, USA	16	Suturing of cadaver laceration undisturbed	Suturing cadaver laceration with mock page during procedure	S (time suturing; GRS score (rating instrument for suturing performance))	Total time suturing with page: 104 s (vs. 90 s); p < 0.01 GRS score 16.4 with page (vs. 15.6)
Gil, 2020	Randomized crossover trial	Medical students, USA	31 (16 nonpreferred music first, 15 preferred first)	Preferred genre of music playing	Least preferred genre of music playing	S (score on DaVinci simulator during tasks)	Higher scores with preferred music (77.4 vs. 72.2, p = 0.025)
Guhde, 2011	Observational study	Third year nursing students, USA	134	Simple adult medical scenario (single problem)	Complex scenario	A (survey on thinking, assessment, and satisfaction	No significant differences in mean Likert scale scores
Haji, 2016	RCT	Medical students, Canada	38 (19 per group)	Simple scenario during skill acquisition (followed by very complex scenario during transfer phase)	Complex scenario during skill acquisition (followed by very complex scenario during transfer phase)	S (global rating scale for LP performance); S (GRS for communication skills during transfer phase); S (# of LP attempts); S (# of sterility breaches); S (cognitive load; reaction time difference and signal detection rate to secondary stimulus)	Skill acquisition phase: simple group GRS LP scores higher than complex group (p = 0.002); fewer needle passes (0.002); fewer sterility breaches (p < 0.001). Reaction time and cognitive load declined in simple but not complex (p < 0.01). Transfer phase: no diff in LP performance or needle passes; fewer sterility breaches in simple group (p = 0.02); GRS worse from retention to transfer in simple group(p = 0.001) (no diff in complex); no diff in cog load measures
Harvey, 2012	Observational study	Emergency Medicine and surgical residents (both doctors), Canada	13	Low stress trauma simulation	High stress trauma simulation (distractors included, less VS stability)	S (checklist; global rating forms for clinical performance; completion of H&P form); S (ANTS scale); A (measures of stress; STAI; cognitive appraisal)	Checklist: HS 43%; LS 48% (p < 0.05); GRS: HS 59%, LS 61% (NS); ANTS HS 67%, LS 70% (NS); Hx form HS 60%, LS 69% (p < 0.05); STAI scores: End of scenario HS 39.8 LS 31.4 (p < 0.05); 30 min after scneario HS 35 LS 29 (p < 0.05); no diff in HR; change in cortisol: HS +1.56 nm/L vs. LS-1 (p < 0.05)
Hughes, 2022	Observational study	Surgeons and medical students, UK	10 students, 10 surgeons	Laparoscopy trainer peg threading task (baseline)	Same task with visual overlay; same task with audio distraction	S (time to task completion, distance of instrument motion, instrument smoothness); A (self-reported pressure/distraction)	Time: No change in time to completion for either distraction in either group; Smoothness: No diff in instrument smoothness in students, 37% decrease in smoothness in surgeons with audio (p = 0.05); Distance: no differences; Self-report: no differences
Kryillos, 2017	RCT	Medical specialty doctors, Quebec.	26 (13 residents, 13 attendings)	Simulated surgical tasks (without music)	Same surgical tasks with music playing	S (anti-tremor parameters, Capsulorhexis task)	No significant differences
Kyaw Tun, 2012	Observational study	Junior doctors, UK	10 experts, 10 novices	Laceration scenario (easy)	Same scenario with difficult patient	S (laceration repair rated by validated instruments (OSATS-TSC [10 pt], OSATS-GRS [5 pt] and DOPS [6 pt])	Simple scenario: no differences in any instrument. Complex scenario: experts better than novices (OSATS-TSC: 1 point (95% CI 0–6 pts); OSATS-GRS (1 pt, 95% CI 0.4–1.9); DOPS: 1.2 pt (0.3–2.2)
Leblanc, 2012	Randomized crossover trial	Paramedics, Canada	22	Low stress resuscitation scenario	High stress resuscitation scenario	S: global clinical score A: self-reported anxiety	S: clinical score worse in high stress scenario (14/21 vs. 12/21, p < 0.05) A: increased in high stress case (p<0.05)
Lizotte, 2017	RCT	Medical students and Senior doctors, Canada	42	Low stress neonatal resuscitation (cardiac arrest with ROSC)	High stress neonatal resuscitation (cardiac arrest with nonsurvival)	S: global score (NRP megacode) A: self-reported stress	No significant differences
Macdougall, 2013	Observational study	Medical students, UK	40	High realism simulation stations designed to generate increased stress	NA	A (self-reported knowledge and confidence)	No significant differences pre- and post-sessions
Marjanovic, 2018	Randomized crossover trial	Medical specialty Doctors, France	26	Intubation scenario	Same scenario with limited physical space and patient deterioration	S (intubation success); A (NASA TLX, self-reported stress)	S: 96% success in basic vs. 15% in difficult scenario A: reported stress and NASA TLX higher in stress scenario (p < 0.001)
Martin-Conty, 2020	Observational study	Nursing students, Spain	59	CPR in normal temperature	CPR in Hot (+40c) and Cold (-35c) conditions	S (CPR efficacy)	No significant difference
Matern, 2018	Observational study	Junior doctors, USA	125	Patient handover	Patient handover with either a looped recording of hospital noise, or looped recorded hospital noise with standardized pages going off	S (efficacy of patient handover)	The Control group showed the same level of prioritisation as the loop + pager group. They gave the least amount of opportunities to ask questions and had no issues being heard during handover (p ≤ 0.001)
Muhammad, 2019	Observational study	Consultant Neurosurgeons, Finland	2	Simulated vessel wall suturing procedure	Same simulated procedure with OR noise or meditation music	S (efficacy of suturing)	The number of repeated movement, which did not achieve the intended goal reduced in both surgeons when meditation music was played (p ≤ 0.05).
Poolton, 2011	Observational study	4th and 5th Medical students, Hong Kong.	30	Pegboard laparoscopy trainer	Pegboard laparoscopy trainer under time constraints (try to perform the task more quickly than their best time during the training phase)	A (self-reported stress)	Time pressure was considered to be significantly more stressful than control (p < 0.001)
Redmond, 2020	Observational study	Emergency Medicine Residents (doctors), USA	12	Neonatal resuscitations; 3 increasing levels of difficulty	NA	A (self-reported stress)	HR data: Low freq/hi freq ratio lowest in scenario 1 than scenario 2 (2.3 vs. 4.7, p = 0.04) and scenario 3 (2.3 vs. 4.6, p = 0.04); self-reported stress increased through 3 scenarios
Weigl, 2016	Observational study	Surgical specialty doctors, Germany	19	Vertebroplasty simulation	Vertebroplasty simulation with disruption	S (trocar position, # of X-rays); A (SURG-TLX score)	A: SURGTLX scores higher in difficult scenario (37 vs. 31, p < 0.01); association btw higher SURGTLX and poorer accuracy with trocar (p = 0.04); S: no differences

ACLS: Advanced Cardiac Life Support; ANTS: Anaesthesia Nontechnical Skills; CPR: cardiopulmonary resuscitation; DOPS: Direct Observation of Procedural Skills; GRS: global rating scale; LP: lumbar puncture; NA: not applicable; NASA TLX: NASA Task Load Index; NRP: Neonatal Resuscitation Program; OSATS-TSC: Modified Objective Structured Assessment of Technical Skills–Task Specific Checklist; OSATS-GRS: Objective Structured Assessment of Technical Skills – Global Rating Score; STAI: State-Trait Anxiety Inventory; SURGTLX: Surgery Task Load Index.

The risk-of-bias assessments (categorized as low, some concerns and serious) are summarized in Table 2. All included studies were determined to have at least a moderate risk of bias. The main issues, related to bias, identified with the studies related to inadequate blinding of participants and assessors, inadequate randomization, incomplete outcome reporting and unclear selection processes (both of participants and the relevance of the simulation for participants). The certainty of evidence was judged to be low for indicators of stress (downgraded for inconsistency and risk of bias) and very low for all other outcomes (downgraded for very serious risk of bias and inconsistency).

Table 2:

Risk of bias assessments.

Randomized controlled trials (RoB 2)
Author, year	Allocation generation		Allocation concealment		Blinding participants	Blinding assessors		Outcome complete	Outcome selective		Other		Overall
Bensouda, 2018	Some concerns		Low		Low	Some concerns		Low	Low		Low		Some concerns
Bjorshol, 2017	Some concerns		Low		Low	Some concerns		Low	Low		Some concerns		Some concerns
Gil, 2020	Some concerns		Low		Low	Low		Low	Low		Low		Some concerns
Haji, 2016	Some concerns		Low		Low	Low		Low	Low		Low		Some concerns
Kyrillos, 2017	Low		Low		Low	Some concerns		Some concerns	Low		Some concerns		Some concerns
Lizotte, 2017	Low		Low		Some concerns	Some concerns		Some concerns	Some concerns		Low		Serious
Marjanovic, 2018	Some concerns		Some concerns		Low	Some concerns		Some concerns	Low		Some concerns		Serious
Martin-Conty, 2020	Low		Some concerns		Some concerns	Some concerns		Low	Low		Low		Some concerns
Observational studies (ROBINS-I)
Author, year		Eligibility criteria		Exposure/outcome			Confounding			Follow-up		Overall
DeMaria, 2016		Moderate		Low			Low			Low		Moderate
Evans, 2017		Moderate		Moderate			High			Low		Serious
Harvey, 2012		Low		Moderate			High			Low		Serious
Hughes, 2022		Moderate		Low			Moderate			Low		Moderate
Kyaw Tun, 2012		Low		Moderate			High			Low		Serious
LeBlanc, 2012		Low		Low			Moderate			Low		Moderate
Macdougall, 2013		Low		Low			Moderate			Low		Moderate
Matern, 2018		Low		Low			Moderate			Low		Moderate
Muhammed, 2019		High		High			High			High		Serious
Poolton, 2011		Moderate		Moderate			Moderate			Low		Serious
Redmond, 2020		High		Moderate			Moderate			Low		Serious
Wiegl, 2016		Low		Moderate			Moderate			Low		Serious

Discussion

In this systematic review, we identified a group of studies where a simulation scenario is administered in at least two different fashions, where one of the scenarios is altered to increase the degree of complication, distraction, physical challenge or emotional context. Our goal was to identify differing degrees of complexity in scenarios; amongst our included studies, the ‘intervention’ of interest was markedly heterogeneous, and which of the instructional design changes truly resulted in increased ‘complexity may be difficult to infer. Nonetheless, many of the studies were able to demonstrate the ability to demonstrate distinct outcomes between the experimental groups based on the scenario change(s). Thus, although no specific definition for complexity is available in current literature, being able to challenge learners in a contextually appropriate way is an essential component of scenario design [5].

The overall quality of methodological rigour, including the selection of instruments to measure performance, reported in this systematic review, varied greatly. Therefore, it is not possible to make substantive statements that varying the level of complexity of simulation scenarios has a measurable effect on skill performance and the stress response of participants. Given the significant heterogeneity of the studies, it is not possible to precisely determine what combination of instructional design elements contributes to scenario complexity the most, nor is it possible to extrapolate the impact of such changes on clinical performance and/or patient care delivery in typical practice. Despite this limitation, the evidence suggests that instructors can create a higher level of complexity by using design elements seen among these studies.

There is a recurring tension in this body of literature where the validity of a simulation scenario from the perspective of the participant is not always apparent or reported. There are frequent instances where participants have no experience of undertaking a skill or procedure prior to the simulation scenario. Therefore, the stores of prior knowledge that participants can apply within this scenario are limited, and this can compromise their perceived agency and their ability to perform to the expected standard.

The majority of simulation scenarios were designed by those who are considered experts. The role of an expert is not well defined in the literature (only one article does this) [17]. The process for determining a relevant stimulus for each scenario (task complexity, auditory stimuli, visual stimuli, environmental conditions, perceived stressors, etc.) was a subjective process made by ‘experts’ in relation to their area of practice. Very few studies used literature or available data to determine the most appropriate stimulus for a scenario. It was noted that authors would share the outcomes from their simulation scenario, but the context for the data collection (the simulation itself) was described in little or no detail. It was noted that only three articles described the preparation of participants (prior to attending the simulation and immediately before this activity) and interventions to promote the well-being of participants during the simulation [12,14,16]. This does create an inherent challenge for healthcare individuals or teams who experience similar challenges and wish to replicate these experimental conditions, as the exact content of this simulation has not been declared.

While not a specific outcome of interest for the present review, several of the included studies reported changes in measurable physiologic parameters (e.g. heart rate, blood pressure, salivary cortisol or dihydroepiandosterone [DHEA] levels) as a proxy measure of stress. Bensouda et al.and DeMaria et al. both found a significantly greater increase in heart rate in the more difficult scenarios in their trials [11,13]. Harvey et al. and LeBlanc et al. demonstrated significantly greater salivary cortisol levels with more difficult scenarios [18,22]; DeMaria et al. found no difference in cortisol or DHEA level between groups [13]. Similar to the earlier discussion about the relationship between scenario complexity and educational impact, the relationship between physiologic stress and improved educational effectiveness remains elusive. Given the growing number of studies using these indices to quantify physiologic changes, future research should take advantage of these reproducible outcomes as the links between stress and learning are explored further.

In 2018, a task force assembled by the American Heart Association released a Scientific Statement on resuscitation educational science [1]. In that statement, evidence on cognitive load and stress on participants in the context of resuscitation education was presented, but no specific recommendations could be made based on the limited data and mixed reported findings of published studies. The results of our review are more consistent (albeit not uniform) in demonstrating that increased complexity of scenarios has a measurable impact on performance. Redmond et al., in a report on measured stress in neonatal resuscitation scenarios, stated that ‘Stress can be a tool, or stress can be a toxin’ [30]. It is intuitive that any given participant will have her/his own ‘sweet spot’ where a certain amount of challenge in a scenario enhances engagement and learning outcomes without being too stressful or taxing to negatively impact psychological safety; how to best determine that in a participant remains elusive. Several recent studies in both nursing and physician trainees have yielded preliminary evidence about the overlap and/or contrast between stress and learning outcomes, using both quantitative and qualitative methods [30,32,33]; further investigation is needed on how to determine and adapt curriculum design to capitalize on these factors.

Several limitations to the findings of this review should be acknowledged. As described earlier, the studies included in this review demonstrated marked heterogeneity, making summative conclusions about the impact of scenario complexity on educational outcomes nearly impossible. As is frequently the case in educational research, outcomes in the form of global assessment of skill were done using an array of instruments, some but not all of which have been determined to be psychometrically robust, biasing the overall results by worsening inconsistency. The development of broadly reproducible and applicable assessment tools in future studies is highly recommended to enable comparative studies and meta-analysis. This recommendation is aligned with the work of Sevdalis et al., who advocated the adoption of validated reporting frameworks, such as the STrengthening the Reporting of OBservational studies in Epidemiology and the Consolidated Standards of Reporting Trials, to create a consistent reporting structure, which would address the noted poor reporting of simulation-based publications. At the time, this authorship group was the editor of the three largest simulation journals in the world: (1) Sevdalis: British Medical Journal for Simulation and Technology Enhanced Learning (BMJSTEL), (2) Nestel: Advances in Simulation (AiS) and (3) Gaba: Journal of Simulation in Healthcare (JiSH). This authorship group endorsed the utilization of these frameworks to improve the rigorous reporting of scientific publications and to enhance the current level of influence and impact that the simulation community has within healthcare education [34]. This systematic review has demonstrated the frequent use of randomized trials within the simulation community. The increase in utilization of controlled and experimental conditions is encouraging, as this has the potential to influence the quality of data captured and the reliability of the outcomes reported. In the majority of randomized trials, there was no significant difference reported between the control and the intervention groups. Constructive debate within the simulation community is required to determine whether this methodological approach is ‘a good fit’ for simulation-based research activities or whether other approaches are more appropriate (such as qualitative methodologies and mixed methods research) to enhance the influence that this community has within healthcare education.

Based on the noted significant heterogeneity of the studies in terms of intervention, assessment(s) and outcomes, this review recommends the development of processes that increase the standardization of reporting the outcomes of simulation scenarios within the literature. To enhance the standardized reporting of simulation scenarios within the literature, it is recommended that journal editors consider requesting the inclusion of a description of the simulation scenario as an appendix for each manuscript. This appendix could be a pro forma that journal editors could endorse and would include a brief overview of the scenario. If two scenarios are included in a research study (in this systematic review, control and intervention scenarios were used frequently), then an overview of both scenarios should be included, and clear distinctions made between both scenarios. The duration of the scenario should be stated along with the interventions taken to ensure the well-being of participants throughout the simulation. This systematic review noted an increased use of validated assessment tools such as OSATS, DOPS and Mini-CEX to objectively measure performance demand. The NASA-TLX and the State Trait Anxiety Inventory and Trait Anxiety questionnaires were utilized to subjectively measure performance demand [35]. To improve the standardized reporting of simulation-based research, it is recommended that the objectives and the measurement instrument be stated in this pro forma to allow other simulationists to recreate these conditions and to capture comparable data. The rationale for selecting participants and the relevance of the scenario to this population should be stated (in this systematic review, it was not always apparent why a certain population had been recruited). Finally, a statement regarding how participants were supported to translate new learning into practice should conclude this pro forma document (Figure 2). This appendix would ensure a more standardized approach to the reporting of simulation scenarios, which has potential benefits of increasing the sharing of best practices and the generalizability of simulation-based research in an organic manner within the global simulation community.

Figure 2:

Simulation scenario pro forma.

Almost all of the included studies reported outcomes of individual subjects as a reaction to a training exercise, with a few studies reporting on changes in performance based on knowledge or skill acquisition. This observation is aligned with the seminal publication by Issenberg et al in 2005 and is something the global simulation community needs to address [36]. The simulation scenario pro forma described in the discussion might be helpful in addressing these issues. In addition, the results of this review highlight several important knowledge gaps. All of the included studies were performed at single institutions; only one enrolled subject with varied backgrounds in health care. None of the studies included in the review examined the time requirement or cost of implementing the interventions used to adjust scenario complexity or complexity, nor did any of them examine how best to train instructors to appropriately conduct and/or adjust scenarios to maximize the positive impact of scenario complexity. None of the studies included input from participants in the design of the simulation scenarios or the interventions, which might elicit an increased demand during this activity; the role of co-creation (where participants and experts develop scenarios together) as a means of determining beneficial scenario complexity, which correlates with participants’ stage of training and lived experience, requires further research [28]. The interplay of these factors requires further research to better delineate how simulation curricula can accommodate participants with diverse learning styles and levels of capability in a beneficial and developmental manner.

Conclusions

This systematic review supports the contention that scenario complexity may be a useful component of instructional design in simulation education to enhance outcomes. Future studies should determine how to use differences in scenario complexity to optimize participant engagement and durability of learning objectives. Future research should also clarify the ideal use of stress measurement (physiologic and/or psychologic) to guide curriculum design using scenario complexity.