Before beginning the practical work of a SR, a researcher should consider four crucial questions. First, is a review of the literature on this topic necessary and will it be of value? To address this, researcher(s) must decide if the question being asked is one of importance currently [29], will valuably guide future research, and whether there is a need for evidence to support decision-making and practice. Further, researchers must determine whether the findings will be novel [29]. This can be achieved by conducting simple literature searches (e.g. via Google Scholar) to identify any relevant published reviews, then considering their methodological quality, completeness and recency. Where a good-quality SR has been conducted in the previous 5–8 years, it is unlikely that a new review will add significant value except where there have been substantive developments or research activity.

Second, is a SR the right review methodology for achieving the aims and addressing the research question(s)? Confusion over what constitutes a SR, and a mischaracterisation of reviews as such, is pervasive [20]. Other forms of evidence synthesis (see Table 1) have emerged and grown in popularity [19], and researchers should determine whether an alternative review methodology might be more suitable. SR is considered appropriate when focused on: collating international evidence, establishing current practice or differences in practice, determining priorities for future research, exploring conflicting results and delivering clear guidance to support decision-making [20]. The Right Review tool [30] may be helpful for offering direction on appropriate review design.

Third, what type of SR is appropriate in this instance? Types of SRs likely to be relevant to HS researchers include: •

Effectiveness reviews: assessing the benefits and/or harms of an intervention or practice [19,20];

Table 2 introduces two example SRs (an effectiveness review and a measurement review) of HS research. These reviews have not been conducted but are offered as examples to support readers in understanding and implementing the guidance offered herein.

Fourth, does the researcher(s) have access to the resources (e.g. time, people and expertise) available to enable completion of a methodologically rigorous review? Research [31] suggests completing and publishing a review requires, on average, 67.3 weeks (not including the time required to assemble the research team and develop the review protocol) and involves a mean of five researchers. At a minimum, a SR team should include two members to enable adherence to best practice throughout (e.g. duplicate data extraction, critical appraisal), and the team must possess the requisite subject matter knowledge to allow completion of the review to a high standard. Further, relevant experts should be consulted for specific SR processes should they fall outside the remit of expertise of the core group; for example, having a research librarian review the search strategy, a statistician advise on data analysis plan, etc. It is crucial that consideration of resources occurs at an early stage of the process, as this will avoid the non-completion of reviews or the poor conduct of important HS reviews.

As in any research project, turning the identified problem into a targeted research question is fundamental to a SR’s success [43]. A clearly defined review question promotes clarity of thought, requiring the researcher to explicate the SR’s intended purpose [44]. Further, it supports confirmation of review methodology, given that different approaches are suited to different types of questions (see Table 1) and guides the conduct of all subsequent steps of the SR (e.g. development of eligibility criteria, determination of data extraction items [44]).

A good SR question is appropriate, meaningful and capable of being posed as a single sentence addressing a primary research question, although a well-defined hierarchy of review questions is also acceptable [43]. Given the diverse nature of research foci and methodologies across healthcare fields, a great number of frameworks exist to support HS researchers in formulating research questions, with suitability dependent on the type of SR being conducted and nature of the research literature being examined. These frameworks can support the consideration of diverse questions, beyond intervention effectiveness to understanding how the application of simulation for pedagogical or non-pedagogical purposes is experienced or elucidating the nuances of how, when and for whom/where the outcomes of its application differ. Some examples include: •

Effectiveness SRs: PICO [33] (Population, Intervention, Comparison, Outcome) is the most used mnemonic, sometimes modified to ‘PICOS’ to include study design [45] or ‘PICOT’ to specify a relevant timeframe [46];

For example, in our effectiveness SR, the researcher(s) should consider and define the four PICO components: the Participant(s) (i.e. Paediatric trainees), the Intervention(s) (i.e. simulation-based education; this could be further limited, for example, to specific simulation modalities or by purpose of application, depending on the focus of the review), the Comparison(s) (i.e. teaching as usual or no teaching) and the Outcome(s) (i.e. performance accuracy) [50]. Table 2 also illustrates the application of PICO within an example measurement/psychometric review. The use of such frameworks assists in ensuring an appropriately targeted review question and is associated with more complex search strategies and precise search results [51]. When a review question has been developed, the FINER criteria [32] may be applied to confirm it is: Feasible (i.e. of manageable scope); Interesting (i.e. will yield an answer important for research and practice); Novel (i.e. addresses a knowledge gap); Ethical (i.e. requires acceptable engagement with human subjects – typically not applicable to SRs) and Relevant (i.e. to knowledge, policy or future research).

Researchers must also consider and establish the scope of the SR (i.e. how broad or narrow its focus should be [52]) when defining their review question. Broader reviews synthesise a greater volume of evidence [18], allowing for generalisability and consistency of findings across a wider range of settings and populations [52]. However, if the proffered question is too broad (e.g. ‘Is simulation beneficial?’), then the researcher may struggle to arrive at a manageable number of studies for inclusion and generating actionable recommendations will be challenging [50]. Reviews addressing narrower questions may be more feasible [18], and the increased homogeneity of included studies may support offering focused recommendations [52]. However, the question must not be too tightly bounded or there may be insufficient relevant literature to inform useful guidance [50] (e.g. ‘What is the efficacy of using simulation to teach paediatric lumbar puncture to advanced nurse practitioners in Ireland?’). When establishing the optimal scope for their SR, researchers should consider the intended end-users’ needs and available resources [52] and engage relevant stakeholders in such discussions.

When a review question is final, researchers must use it to establish the eligibility criteria for including/excluding studies in the SR. Inclusion criteria are characteristics (e.g. demographic, clinical, methodological and/or geographic) that studies must possess to be included [53], while exclusion criteria describe studies that are related to some degree but will not contribute useful research data and are thereby ineligible for inclusion. These criteria should flow directly from the review question(s) and be specified a priori [54]. For each PICO/mnemonic component, the researcher(s) should determine what relevant information they wish to focus on. For instance, for ‘Population’, what participants are they interested in? For ‘Intervention’, is there a modality, purpose or type of HS intervention that should be specified? For ‘Outcome’, what type of data or measurements will be of use? Table 2 outlines relevant eligibility criteria for the example SRs. These criteria are crucial for guiding the development of search terms and decision-making during study selection.

Developing a protocol at the outset of the SR process ensures that all researchers have a shared understanding of plans, that they can be feasibly implemented and that the work constitutes good practice. When preparing a protocol, researchers should comply with the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) guidelines [36], which requires: •

Rationale: research-based justification of the need for the review and its evidence;

Protocol development should be viewed as a ‘trialling’ process, allowing researchers to pilot plans for each step and clarify any ambiguity identified. For instance, the planned search strategy should be trialled within electronic databases to ensure the resulting returns are manageable and relevant. When researchers are satisfied that the plans are clear, relevant and achievable, a final protocol should be agreed.

The protocol should be made accessible online before formal searches have been initiated [55]. Research suggests that SRs with prospectively registered protocols have greater methodological rigour [56] and are more likely to be published in higher impact factor journals [57]. Indeed, the PRISMA guidelines require a priori protocol registration [42]. Perceived barriers may include costs, concerns that others may ‘take’ the idea or excessive workload [58]. However, SR protocols can be made available online with minimal effort and often at no cost [59], via: •

The International Prospective Register of Systematic Reviews (PROSPERO [60]);

Registration requires little work beyond that inherent in developing the protocol [55] and may save time by ensuring a clear plan is in place, thereby reducing the opportunity for errors and miscommunications [58]. Instances of ‘stealing’ of SR plans are low [65], and protocol registration may be considered a means of ‘staking a claim’ on a topic [58]. Additionally, researchers may consider publishing their protocol in a relevant journal as an article for peer review, ideally before data collection has commenced. A number of HS-specific journals (e.g. International Journal of Healthcare Simulation) accept review protocols as an article type. Publication of protocols may yield great exposure and engagement, as well as furthering transparency, reproducibility and accountability.

Devoting sufficient time and effort to the initial protocol will help ensure the review can be completed as planned. However, in some instances, a need to revise the protocol may arise. For example, data on a particular outcome may not be presented within included studies and a protocol may need updating to omit this data [55]. Many review registries permit records to be updated, allowing researchers to clearly document and justify changes made [59]. Any amendments to the review protocol should be carefully considered and described in the review manuscript [42].

Once a protocol is agreed, the process of identifying studies for inclusion can begin. Typically, a SR search strategy will comprise electronic database and reference list screening but may be supplemented with additional search tactics. When composing a search strategy, researchers should carefully consider available resources, the nature of the review, where eligible studies are likely to be published and the time and effort required to enact any search tactic as compared to the likelihood of it yielding additional relevant studies. Researchers should consult Atkinson and colleagues’ [38] reporting standards for literature searches to ensure search quality, transparency and replicability.

Electronic database searches constitute the core work of searching within a SR. Databases available to researchers will vary by institution, and their appropriateness will vary depending on the review’s focus. We recommend including three to five electronic databases, in line with research suggesting most relevant articles are found within a limited number of databases [66]. Where smaller numbers of studies are expected, searching more databases may be reasonable [66]. The following databases are likely to be useful within SRs of HS [66,67]: EMBASE, MEDLINE, Web of Science, Scopus, CINAHL, PsycINFO and Education Resources Information Center (ERIC).

Following database selection, researchers must develop a set of search terms for each. This involves two stages: initial development and agreement of search terms (typically for the MEDLINE database), and the adaptation of these for application in other databases. There is significant work involved in developing search terms, which need to be sufficiently broad to capture all potentially relevant records but precise enough that the number of returns can be screened within available resources. For initial development, it is crucial to identify the different aspects of the review question to be reflected in the search to build ‘sets’ of search terms for each (Table 2 illustrates the appropriate development of sets for our example reviews). To build the initial MEDLINE search strategy, begin the population of each ‘set’ with Medical Subject Headings (MeSH terms) via the use of MeSH Browser [68] (which returns relevant MeSH terms when keywords are entered). Following this, augment subject headings within each set with free-text keywords (i.e. relevant words or phrases that describe the concept/issue/topic/intervention). These can be generated based on researchers’ knowledge but should be supplemented with additional keywords from the search strategies of other related reviews and the consideration of title, abstracts and keywords used within known relevant articles.

Typically, search terms within sets are combined using the ‘OR’ Boolean operator (the instruction of a database to find articles using this term OR that term OR this term etc.), while the sets themselves are combined using the ‘AND’ Boolean operator (the instruction of a database to find articles that use at least one term from Set 1 AND at least one term from Set 2). When a full search strategy is available, researchers should carefully consider the application of any search filters (e.g. language, peer-review status), as their use substantially impacts the returns generated and requires justification. It is important to ensure that the same search filter can be applied across each database.

Finally, researchers will need to add wildcards or truncation to their search strategy as appropriate. This involves adding a term or symbols to individual search terms to allow identification of different forms of those words (e.g. entering ‘simulation’ will return records using that term only, entering ‘simulat*’ will return records using simulations, simulates, simulated, etc.). This allows researchers to capture plurals (e.g. measurement and measurements), alternate spellings (e.g. behaviour and behavior) and potentially different word endings (e.g. mannequin and mannekin). Wildcards also support researchers in searching within specific parts of an article (e.g. ti,ab. entered on MEDLINE will implement searches for the relevant term only in articles’ title or abstract) and searching for words only when used in proximity (e.g. ‘high adj1 fidelity’, ‘task adj1 trainer’). The symbols for enacting truncation and wildcards differ across databases and researchers should use the search guidance available within databases to ensure an equivalent search. HS researchers may find search ‘hedges’ [69] – pre-defined comprehensive combinations of search terms for a concept/topic created by an expert searcher, typically for a single database, useful to save time or narrow the scope of the search.

Best practice suggests engaging a research librarian in, at minimum, review of the initial search terms, with discernible positive effects of librarian involvement evidenced [70]. The Peer Review of Electronic Search Strategies (PRESS) [37] checklist can be applied by researchers to critically appraise their search strategy. In advance of formally conducting the search, it may also be useful to first identify a ‘seed set’ of articles (i.e. a sample set of 4–6 articles that meet inclusion criteria and would be expected to be captured by the search strategy); these can then be used to test the thoroughness of the database searches, ensuring that search terms and processes would capture such relevant articles. Once an initial search strategy has been agreed, researchers must adapt it for use within the remaining databases. While the MeSH browser yields subject headings existing in MEDLINE and PubMed, other databases will have their own, tailored to their specific focus. Researchers can identify these terms within each database’s relevant thesaurus or library.

Several issues may arise during the process of developing the search strategy. An excessive number of returns will require researchers to reconsider keywords, explore an additional ‘set’ of search terms, and/or incorporate the use of additional wildcards (e.g. proximity operators). Conversely, few returns may necessitate broadening the search through removal of a ‘set’ of terms, introducing additional keywords, removing search filters and/or revising use of truncation or wildcards. Once the electronic database search strategy has been finalised for each database, its development is complete. The PRISMA 2020 statement [42] requires presentation of a full search strategy for each database, typically within the supplementary materials. Researchers should ensure that the search strategy for each database is saved, allowing searches to be enacted quickly and consistently and facilitating the updating of searches later. Updating searches prior to completion of the SR may be considered to ensure the most recent publications are included. There is no clear guidance on acceptable time lapse since completion of searching, although some journals require searches to have been completed within the previous six months.

Once plans for electronic searches have been agreed, researchers must decide on the use of any supplemental search tactics to identify additional relevant studies missed by electronic searches and to ensure the comprehensiveness of their overall search strategy. Table 3 provides an overview of supplemental search tactics and guidance on employing each. While the target within a SR is identifying all relevant studies, it is recognised that there will be instances where relevant articles are not identified, which is not necessarily an indicator of a poor-quality search. Where researchers have adequately planned and carefully implemented a search strategy, the likelihood is that the resulting review and its conclusions will be useful and well-founded despite this [38].

Once a researcher has retrieved relevant records from the electronic databases, the process of screening these and selecting studies for inclusion begins, comprising de-duplication, initial screening and second-stage screening.

Searching multiple databases inevitably results in the retrieval of duplicate publications (e.g. different databases returning the same publication, or different versions of the same study such as a conference abstract and a full text article) [72]. Each study should only be considered once when screening; therefore, researchers should use an electronic reference management system (e.g. EndNote, ProCite) to facilitate the identification and removal of duplicates (i.e. de-duplication) [18,72,73]. De-duplication lessens workload [72], ensuring reviewers avoid wasting time screening the same citation multiple times.

During the initial screening of remaining records’ titles and abstracts, researcher(s) will evaluate each through application of the pre-defined eligibility criteria. Two researchers screening records independently is ideal to ensure transparency and minimise bias [18,42] and increases the number of relevant studies identified [74]. Each researcher screens each record independently and determines whether there is enough information in the title and abstract to include or exclude or whether it is potentially relevant and requires further review. Next, researchers compare assessments; records should be excluded if both reviewers agree, and records should be moved to the next stage (i.e. full-text screening) if both reviewers indicate eligibility or potential eligibility. Records with differing judgements should be reviewed together, and consensus achieved via discussion [74]. Those articles identified as relevant, or potentially relevant, should be obtained for second-stage screening [73]. In practice, engaging two researchers in study screening may not be feasible and will require consideration of capacity and resources available. Using a second reviewer during full-text screening only may be a way to reduce resources, while still limiting bias [74].

Second-stage screening involves examination of the full text of records retained from title and abstract screening, with involvement of two researchers strongly recommended [74]. Each researcher should scrutinise the entire text of each record to confirm/disconfirm relevance, guided by the eligibility criteria, and decide whether to ‘include’ or ‘exclude’ each article. For each excluded article, a justification should be documented [72]. When independent screening is complete, the researchers’ logs should be compared in the same manner described previously. The involvement of a third researcher, or wider research team, is important for resolving any remaining disagreements or inconsistencies. Ultimately, a final list of included studies from electronic database searches will have been identified [18]. The use of supplemental search tactics (e.g. reference list reviews) can be completed at this point (see Table 3).

Adequate documentation of screening (i.e. a record of the count of records identified, screened, included, excluded and reasons) is essential [72] for completing the PRISMA flowchart [18]. We direct readers to Rethlefen and Page [75] for further useful guidance on tracking records throughout the SR process.

When the studies for inclusion have been finalised, data extraction can begin. Data extraction refers to the process of recording relevant information from studies for consideration, refinement and analysis [76]. The data extraction process comprises selecting data collection tools, constructing data collection forms and abstracting, managing and archiving data [77].

Data extraction can be conducted using either paper forms, electronic forms (e.g. Microsoft Word), spreadsheet software (e.g. Microsoft Excel) or commercial data systems (e.g. Covidence) that allow online form building, data entry by several users, data sharing and efficient data management [77]. Electronic forms are more efficient and less error-prone than paper forms, while data systems require greater financial investment and training [77,78]. However, no single data-extraction tool is best for all SRs in all circumstances. Therefore, researchers should carefully consider the volume, nature and anticipated complexity of the data and resource availability, before selecting a tool [78]. We centre our subsequent guidance on using an electronic form via Microsoft Word, which is generalisable to other tools.

The data extraction form should be easy to use, standardised and allow for the structured collation of data to sufficiently represent the study [77], minimising the need to return to the source [79]. Careful development, pilot testing and application are crucial, and should involve iterative input from all researchers. First, researcher(s) should prepare outlines of the tables and figures for the SR (e.g. table of study characteristics, tables/figures offering a summary of the findings) to support determination of the data required [77,79]. Next, data extraction variables/elements should be logically grouped (see Table 2 for illustration of data extraction for example reviews). Subsequently, the form can be presented using a word processor, in a table format capturing all studies or using individual forms that can later be tabulated. A codebook should also be generated, containing relevant definitions and details on how to capture and code data for each variable. Providing guidance on the form itself supports quality and consistency across researchers [77]. Finally, the form should be piloted among multiple researchers, extracting data from at least three articles [77], with modifications made as necessary to ensure agreement. The ‘seed set’ of articles developed within Step 4 for trialling the search strategy, could also be used within the pilot data extraction.

It is imperative that researchers (especially SR novices) receive training to familiarise them with the data extraction process and clarify any uncertainties. Data extraction errors are frequent [80]; therefore, two researchers should be involved [4,77], which has been found to result in fewer errors [81]. Data extractors from complementary disciplines (e.g. a methodologist and subject-matter expert) [77,79] may valuably support one another. Each researcher should extract data for all studies independently, then compare their records to identify discrepancies. Discrepancies can be resolved through discussion, with re-engagement with the source document as necessary. If consensus cannot be achieved, a third researcher may adjudicate. A record of the presence and resolution of disagreements, and raw copies of original individual data extraction forms or files (this will differ based on whether the researchers have opted to extract data from all studies into one table or used individual forms), should be retained [79]. Ultimately, files should be merged into one final single document that offers a complete and agreed record of data extraction.

To facilitate the later synthesis of extracted data, researcher(s) may have to perform a degree of coding when initial data extraction is completed, introducing homogeneity within the data extraction records [82]. This can be achieved via content analysis, which involves researchers developing categories/codes to capture the extracted data either de novo (i.e. inductive coding [83]) or through the application of existing frameworks to standardise the data (i.e. deductive coding [83]). Such coding allows researchers to establish the frequency and consistency of findings, methods or study characteristics and helps to determine what is ‘key’ across studies or how they reflect existing theory or understanding.

Given the implications of SR for practice and research, consideration of trustworthiness via critical appraisal of included studies is core to conducting a high-quality SR [42]. Researchers will encounter two relevant concepts: •

Risk of bias: how methodological limitations or flaws within design, conduct or analysis may compromise resulting data [84]; and

Focusing on risk of bias is recommended [84], particularly within effectiveness reviews. Several tools and corresponding guidance on their use, have been developed to support researchers in formally appraising studies. Our non-systematic examination of recent HS-related SRs revealed a great number of tools have been used to support critical appraisal, including: •

ROB-2 (Cochrane’s risk-of-bias tool for randomised trials) [85];

When selecting a tool, researchers should consider the likely study designs, and the experience within the research team. Considering critical appraisal tools used in similar SRs, consulting colleagues and experts in evidence synthesis, and examining other review articles that have used a tool under consideration may support decision-making.

Typically, two researchers will independently complete critical appraisal with discussion to agree final ratings, though this process is sometimes conducted in tandem, with researchers working together to finalise judgements for each item and study. Numerous challenges exist within the appraisal process: considerable time is required [89], inter-rater agreement is variable even with well-established tools [89], and performance is likely impacted by familiarity with the tool, between co-raters and with critical appraisal [89]. Accordingly, the tool’s user-friendliness, sufficient allocation of time, researcher experience and initial training, review and piloting of the tool are crucial considerations to ensure the process is completed effectively.

When the data extraction record is finalised, the process of data synthesis commences. SRs in HS frequently emphasise diversity across included studies (e.g. heterogeneity of outcome measures [27,28]) which is likely to render synthesis via meta-analysis (see Table 1) challenging. Accordingly, within both effectiveness reviews and other review types, a range of alternative approaches are often more suitable. We focus our guidance herein on the process of narrative synthesis, which we expect to be suited to most reviews of the HS literature, though Table 4 details alternative tactics for synthesising data in SRs that can also be applied.

Narrative synthesis is a textual approach to collating findings, involving the use of text to ‘tell the story’ of the findings and learning [16]. Narrative synthesis is inherently subjective to an extent [90], and it is important that researchers pursue a rigorous and transparent approach to reduce potential bias. Popay et al. [16] offer comprehensive guidance on the conduct and reporting of narrative synthesis, defining four main elements. First, developing a theory about why the intervention works and for whom. This element is applicable to effectiveness SRs and potentially others such as experiential or qualitative SRs. In addressing this, researchers consider the rationale for the intervention outlined in studies seeking to identify the pathways through which an intervention has its effects [91]. Researchers focusing on the effects of a HS intervention will need to organise the results of included studies to enable description of emerging patterns in terms of the direction and size of effects [16]. Second, researchers must develop a preliminary synthesis of findings of included studies. This involves determining how best to bring together, organise and describe the results from included studies. Ryan [91] offers valuable guidance on varied ways to achieve this: •

Textual descriptions summarising features for each study;

Third, researchers must explore relationships in the data within and between studies. Researchers should explore two key relationships: those between characteristics of individual studies and their findings, and those between the findings of different studies [16]. To achieve this, the researcher should examine the extracted data with a view to looking at the relationships between study results and key aspects of the studies and comparing these relationships across studies. At this point, the researcher should consider whether variability in study design, populations, interventions or outcomes is likely to explain these differences, in addition to the effect of heterogeneity [91]. The following approaches to explore the relationships between and within studies can be used [91]: •

Visual tools (e.g. forest plots, graphs) can help to organise and present results. These should supplement, rather than replace, a textual description of results.

Finally, researchers should appraise the rigour of the synthesis. This comprises examining the quality or risk of bias of included studies (see Step 7). Researchers should also consider the trustworthiness of their processes for collating evidence, such as critical reflection on strengths and weaknesses of the methods used to narratively synthesise included studies [91]. When reporting on the narrative synthesis of studies, researchers should adhere to the Synthesis Without Meta-analysis (SWiM) guideline [92] to ensure a clear description of their procedures.

The remaining work of the SR consists of preparing a written manuscript. There is significant value in identifying a target journal prior to commencing the write-up, as this will guide certain elements (e.g. structure, word count etc.). Considering other SRs published in the target journal may also support authors in preparing a suitable manuscript. A SR manuscript will include the following sections: Introduction, Methods, Results and Discussion along with an Abstract summarising the conduct and core messages of the SR.

Drawing conclusions in a SR typically involves considering the implications for practice, policy and future research. When providing implications for theory, practice or policy, researchers should address the review question and consider the intended end-users’ needs. These should be supported by the evidence synthesis (i.e. they should not be speculative), with acknowledgement of the strength and limitations of the evidence [45,97]. In stating recommendations for future research, researchers should avoid vague statements (e.g. ‘more research is needed’) and identify specific areas where evidence is lacking and warrants prioritisation [45]. The PICO domains may be useful here to inform aspects that could be addressed more effectively in the future [97]. Further, the domains of the GRADE [98] certainty framework can support researchers in interpreting their evidence and drawing conclusions about future research and practice.

Researchers must carefully consult and adhere to the PRISMA 2020 checklist [42], which offers detailed instruction for writing each manuscript component and for appropriately detailing funding, conflicts of interest and author contributions. This guidance includes 12 items that dictate good practice in the presentation of an Abstract, the preparation of which is often most appropriate when the main manuscript text has been completed. Writing within the confines of limited word counts is challenging, and the considered use of tables and figures, as well as supplementary materials (e.g. for detailed study-by-study summaries, full search strategies), is required to ensure that the SR’s data is presented comprehensively and effectively [99].