Step 1: establish the necessity and type of review to be conducted and consider resources required

Before beginning the practical work of a SR, a researcher should consider four crucial questions. First, is a review of the literature on this topic necessary and will it be of value? To address this, researcher(s) must decide if the question being asked is one of importance currently [29], will valuably guide future research, and whether there is a need for evidence to support decision-making and practice. Further, researchers must determine whether the findings will be novel [29]. This can be achieved by conducting simple literature searches (e.g. via Google Scholar) to identify any relevant published reviews, then considering their methodological quality, completeness and recency. Where a good-quality SR has been conducted in the previous 5–8 years, it is unlikely that a new review will add significant value except where there have been substantive developments or research activity.

Second, is a SR the right review methodology for achieving the aims and addressing the research question(s)? Confusion over what constitutes a SR, and a mischaracterisation of reviews as such, is pervasive [20]. Other forms of evidence synthesis (see Table 1) have emerged and grown in popularity [19], and researchers should determine whether an alternative review methodology might be more suitable. SR is considered appropriate when focused on: collating international evidence, establishing current practice or differences in practice, determining priorities for future research, exploring conflicting results and delivering clear guidance to support decision-making [20]. The Right Review tool [30] may be helpful for offering direction on appropriate review design.

Third, what type of SR is appropriate in this instance? Types of SRs likely to be relevant to HS researchers include:

•Effectiveness reviews: assessing the benefits and/or harms of an intervention or practice [19,20];

•Experiential or qualitative reviews: improving understanding of interventions, their implementation or impact and/or exploring stakeholder interpretation of an intervention/event/experience [19,20];

•Costs/economic evaluation reviews: assessing the costs and resource usage of an intervention/service/experience/process/procedure [20];

•Measurement/psychometric reviews: exploring the characteristics of measurement instruments and clarifying best practice in assessing a construct [19,20];

•Expert opinion/policy reviews: synthesising documents describing guidelines, policies or protocols, sometimes alongside relevant original research, to clarify best practice [20]; and

•Methodology reviews: considering aspects of study design or conduct with a focus on research methods and/or quality [20].

Table 2 introduces two example SRs (an effectiveness review and a measurement review) of HS research. These reviews have not been conducted but are offered as examples to support readers in understanding and implementing the guidance offered herein.

Table 2:

Summary of key guidance, and example healthcare simulation systematic reviews illustrating each step of the SR process

	Key recommendations	Example review 1 (effectiveness review)	Example review 2 (measurement review)
Articulate an appropriate review question	• Apply appropriate framework to guide development of a clear, targeted review question. • If posing multiple review questions, clarify the hierarchy or prioritisation of these. • Apply FINER criteria [32] to confirm review question appropriateness. • Once review question is final, develop detailed eligibility criteria. • Establish appropriate scope of SR, with consideration or involvement of intended end-users and their needs.	Research question: what is the efficacy of using simulation to teach paediatric lumbar puncture among paediatric trainees? PICO framework for intervention reviews [33]: Population: paediatric trainees Intervention: simulation-based training targeting performance of paediatric lumbar puncture Comparator: training as usual or no training Outcome: performance accuracy, successful/traumatic lumbar puncture in clinical setting Inclusion criteria: Studies that: are published in a peer-reviewed journal; Report original research; Employ the use of simulation education for training LP procedure*; Focus on training paediatric trainees; Report outcome data on the impact of the training with the use of at least one formal measurement tool. *These criteria could be further limited to specific simulation modalities Exclusion criteria: Non-empirical studies (e.g. editorials, commentaries); Studies using simulation for undergraduate education; Studies that recruit a wider population of HCPs, where it is not possible to extract the data for paediatric trainees in isolation; Studies that do not report outcome data.	Research question: how has the impact of in-situ simulation to identify latent safety threats been measured in the research literature? PICO framework for measurement/psychometric reviews [34,35]: Population: healthcare professionals participating in hospital-based high-fidelity in-situ simulation(s) focused on identifying latent safety threats Instrument: quantitative and/or qualitative measures applied Construct: impact of in-situ simulation focused on identifying latent safety threats Outcomes: characteristics of measures (e.g. number of items, validity, reliability, usability) Inclusion criteria: Studies that: are published in a peer-reviewed journal; Report original research; Assess the impact of at least one session of high-fidelity simulation in a healthcare setting implemented with the intention of identifying latent safety threats using at least one explicit measurement tool; Provide detail on at least psychometric characteristic of the applied measure (e.g. reliability, validity). Exclusion criteria: Non-empirical studies (e.g. editorials, commentaries); Studies conducted in a simulation laboratory setting; Studies using in-situ simulation without a reported focus on identifying latent safety threats; Studies that do not implement a formal measurement tool; Studies that do not provide detail on any psychometric characteristics of the measure used.
Develop a review protocol	• Assemble an appropriate review team. • Develop detailed protocol in adherence with the PRISMA-P guidelines [36]. • Pilot the work of key steps to confirm clarity, feasibility and appropriateness of plans. • Prospectively register the final SR protocol online within an appropriate repository. • Update protocol if a clear need arises and detail changes made within SR manuscript.
Identify relevant literature	• Develop overall search strategy and appropriate search terms for individual electronic databases, supported by engagement with relevant literature, piloting of searches and the advice of a research librarian. • Apply the PRESS checklist [37] to evaluate plans. • Adhere to Atkinson [38] and Colleagues’ reporting standards for literature searches. • Search three to five electronic databases. • Consider the use of supplemental search tactics (particularly reference list screening) based on resources available and anticipated body of research. • Update searches if there is a significant interlude between searches and manuscript submission.	Search 5 electronic databases (MEDLINE, CINAHL, ERIC, PsycInfo, Web of Science) and supplement with reference list screening and forwards citation chasing (based on anticipated reasonable size and spread of relevant literature) Search terms to be organised within three sets: Terms focused on Healthcare Simulation (e.g. subject headings such as ‘Patient Simulation’ OR ‘Simulation Training’ combined with free-text entries such as ‘Simulat*’ OR ‘task adj1 trainer*1’) Terms focused on Paediatrics (e.g. subject headings such as ‘Pediatrics’ combined with free-text entries such as ‘p?ediatric*1’ OR ‘neonat*’) Terms focused on Lumbar Puncture (e.g. subject headings such as ‘Spinal Puncture’ combined with free-text entries such as ‘lumbar adj1 puncture*1’ OR ‘spinal adj1 tap*1’)	Search 3 electronic databases (MEDLINE, CINAHL, Web of Science; based on anticipated high volume of relevant studies and specific focus on in-situ simulation). Search terms to be organised within two sets: Terms focused on In-situ simulation (e.g. subject headings such as ‘Patient Simulation’ OR ‘High Fidelity Simulation Training’ combined with free-text entries such as ‘high adj1 fidelity’ OR ‘in adj1 situ adj1 simulation*1’) Terms focused on identifying latent safety threats (e.g. Subject headings such as ‘Patient Safety’ combined with free text entries such as ‘latent adj1 (safety OR error* OR hazard*)’ Terms focused on assessment of outcomes or impact (e.g. subject headings such as ‘Outcome Assessment, Health Care’ combined with free-text entries such as ‘outcome*1’ OR ‘measure*’)
Screen and select studies	• Use electronic reference management system to identify and remove duplicate citations. • Conduct initial screening (i.e. title and abstract screening), ideally with involvement of, and discussion between, two researchers. • Conduct second-stage screening (i.e. full-text screening), with involvement of, and discussion between, two or more researchers as necessary. • Agree final list of studies for inclusion from electronic databases searches and implement any supplemental search tactics at this stage. • Maintain careful documentation of the work at each stage of the process (e.g. reasons for exclusion of studies during full-text review).
Extract data and finalise a record of studies	• Select appropriate data extraction tool. • Develop appropriate data extraction form and accompanying codebook/guide. • Devote time for training researchers in use of the data extraction form. • Pilot the form on a sample of included studies with revision as necessary. • Two researchers extract data independently from each study. • Researchers compare files, resolve discrepancies, and agree a final record of data extraction. • Code extracted data via content analysis as necessary to facilitate synthesis.	Data extraction items to include: • Author, year • Country the study was conducted in • Study design • Setting • Number and level of participants • Intervention characteristics (e.g. learning outcomes, simulator type, additional learning opportunities and resources, facilitators involved, duration and timing) • Outcome measures employed (e.g. confidence/knowledge questionnaire, accuracy and timing of skill performance) • Impact of training recorded	Data extraction items to include: • Author, year • Country the study was conducted in • Study design and methodology • Setting • Number, level and profession of participants • Measure characteristics (e.g. number of items, validity, reliability, usability any identified barriers and facilitators to its’ use) • Findings
Critically appraise included studies	• Devote sufficient time to identifying a critical appraisal tool appropriate for application with the types of studies that will be included that can be effectively implemented by researchers. • Allow time for training and familiarisation with the selected tool. • Ensure novice reviewers are appropriately supported in applying the tool. • Involve two researchers in completing the critical appraisal, working independently or in tandem, with discussion to agree final judgements.	A relevant tool should have the capacity to consider non-randomised interventional research, and/or research evaluating educational interventions. Researchers might consider applying tools such as: • MERSQI [39] • the Risk-of-Bias in Non-randomized Studies of Interventions tool [40]	A relevant tool should have the capacity to consider non-randomised interventional research and might usefully support consideration of the psychometric characteristics of measurement tools. Researchers might consider applying tools such as: • QuADs tool [41] • The COSMIN Risk-of-Bias tool to assess the quality of studies on measurement properties [34]
Synthesis and write-up	• Consider and select appropriate synthesis approach. • Complete synthesis of data to bring together results of included studies, supplementing textual descriptions with visual tools in the form of tables and figures, use of counting or statistics as helpful in summarising the data, subgroup analysis, thematic or content analysis or development of conceptual models as useful to ‘tell the story’ of the data. • Identify target journal, noting available guidance on word count presentation etc., and examine SRs previously published within it. • Write up manuscript according to PRISMA guidelines [42]. • Make use of appendices or supplementary material as possible to be comprehensive in presentation of the SR’s processes and data.

* Abbreviations: HCPs: Healthcare Professionals; MERSQI: Medical Education Research Study Quality Instrument; PRESS: Peer Review of Electronic Search Strategies; PRISMA: Preferred Reporting Items for Systematic reviews and Meta-Analyses; PRISMA-P: Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols; QuADs: Quality Assessment with Diverse Studies.

Fourth, does the researcher(s) have access to the resources (e.g. time, people and expertise) available to enable completion of a methodologically rigorous review? Research [31] suggests completing and publishing a review requires, on average, 67.3 weeks (not including the time required to assemble the research team and develop the review protocol) and involves a mean of five researchers. At a minimum, a SR team should include two members to enable adherence to best practice throughout (e.g. duplicate data extraction, critical appraisal), and the team must possess the requisite subject matter knowledge to allow completion of the review to a high standard. Further, relevant experts should be consulted for specific SR processes should they fall outside the remit of expertise of the core group; for example, having a research librarian review the search strategy, a statistician advise on data analysis plan, etc. It is crucial that consideration of resources occurs at an early stage of the process, as this will avoid the non-completion of reviews or the poor conduct of important HS reviews.

Step 2: articulate an appropriate review question and determine eligibility criteria

As in any research project, turning the identified problem into a targeted research question is fundamental to a SR’s success [43]. A clearly defined review question promotes clarity of thought, requiring the researcher to explicate the SR’s intended purpose [44]. Further, it supports confirmation of review methodology, given that different approaches are suited to different types of questions (see Table 1) and guides the conduct of all subsequent steps of the SR (e.g. development of eligibility criteria, determination of data extraction items [44]).

A good SR question is appropriate, meaningful and capable of being posed as a single sentence addressing a primary research question, although a well-defined hierarchy of review questions is also acceptable [43]. Given the diverse nature of research foci and methodologies across healthcare fields, a great number of frameworks exist to support HS researchers in formulating research questions, with suitability dependent on the type of SR being conducted and nature of the research literature being examined. These frameworks can support the consideration of diverse questions, beyond intervention effectiveness to understanding how the application of simulation for pedagogical or non-pedagogical purposes is experienced or elucidating the nuances of how, when and for whom/where the outcomes of its application differ. Some examples include:

•Effectiveness SRs: PICO [33] (Population, Intervention, Comparison, Outcome) is the most used mnemonic, sometimes modified to ‘PICOS’ to include study design [45] or ‘PICOT’ to specify a relevant timeframe [46];

•Experiential/qualitative SRs: SPIDER [47] (Sample, Phenomenon of Interest, Design, Evaluation, Research type);

•Costs/economic evaluation SRs: PICOC [48] (Population, Intervention, Comparator/s, Outcomes, Context);

•Measurement/psychometric SRs: PICO [34,35] (Population, Instrument, Construct, Outcomes);

•Expert opinion/policy SRs: ECLIPSE [49] (Expectation, Client group, Location, Impact, Professionals, Service); and

•Methodology SRs: SDMO [4] (Study type, Data type, Method type, Outcome).

For example, in our effectiveness SR, the researcher(s) should consider and define the four PICO components: the Participant(s) (i.e. Paediatric trainees), the Intervention(s) (i.e. simulation-based education; this could be further limited, for example, to specific simulation modalities or by purpose of application, depending on the focus of the review), the Comparison(s) (i.e. teaching as usual or no teaching) and the Outcome(s) (i.e. performance accuracy) [50]. Table 2 also illustrates the application of PICO within an example measurement/psychometric review. The use of such frameworks assists in ensuring an appropriately targeted review question and is associated with more complex search strategies and precise search results [51]. When a review question has been developed, the FINER criteria [32] may be applied to confirm it is: Feasible (i.e. of manageable scope); Interesting (i.e. will yield an answer important for research and practice); Novel (i.e. addresses a knowledge gap); Ethical (i.e. requires acceptable engagement with human subjects – typically not applicable to SRs) and Relevant (i.e. to knowledge, policy or future research).

Researchers must also consider and establish the scope of the SR (i.e. how broad or narrow its focus should be [52]) when defining their review question. Broader reviews synthesise a greater volume of evidence [18], allowing for generalisability and consistency of findings across a wider range of settings and populations [52]. However, if the proffered question is too broad (e.g. ‘Is simulation beneficial?’), then the researcher may struggle to arrive at a manageable number of studies for inclusion and generating actionable recommendations will be challenging [50]. Reviews addressing narrower questions may be more feasible [18], and the increased homogeneity of included studies may support offering focused recommendations [52]. However, the question must not be too tightly bounded or there may be insufficient relevant literature to inform useful guidance [50] (e.g. ‘What is the efficacy of using simulation to teach paediatric lumbar puncture to advanced nurse practitioners in Ireland?’). When establishing the optimal scope for their SR, researchers should consider the intended end-users’ needs and available resources [52] and engage relevant stakeholders in such discussions.

When a review question is final, researchers must use it to establish the eligibility criteria for including/excluding studies in the SR. Inclusion criteria are characteristics (e.g. demographic, clinical, methodological and/or geographic) that studies must possess to be included [53], while exclusion criteria describe studies that are related to some degree but will not contribute useful research data and are thereby ineligible for inclusion. These criteria should flow directly from the review question(s) and be specified a priori [54]. For each PICO/mnemonic component, the researcher(s) should determine what relevant information they wish to focus on. For instance, for ‘Population’, what participants are they interested in? For ‘Intervention’, is there a modality, purpose or type of HS intervention that should be specified? For ‘Outcome’, what type of data or measurements will be of use? Table 2 outlines relevant eligibility criteria for the example SRs. These criteria are crucial for guiding the development of search terms and decision-making during study selection.

Step 3: prepare, register and potentially update a review protocol

Developing a protocol at the outset of the SR process ensures that all researchers have a shared understanding of plans, that they can be feasibly implemented and that the work constitutes good practice. When preparing a protocol, researchers should comply with the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) guidelines [36], which requires:

•Rationale: research-based justification of the need for the review and its evidence;

•Objectives: the review question(s) that have been developed;

•Eligibility criteria: guidelines on the characteristics of the studies that will be included, with decision-making justified as appropriate;

•Information sources: each search tactic (including where, when and how) that will be used to retrieve relevant articles;

•Data management: how records and data will be handled throughout including any software to be used;

•Selection process: how, and how many, researchers will work to identify, consider, and select studies for inclusion;

•Data extraction: how, and what, data will be taken from studies and by who;

•Critical appraisal: how methodological rigour of studies will be assessed, and resultant data will be used; and

•Synthesis: plans for bringing the data together to create a useful summary and to answer the review question.

Protocol development should be viewed as a ‘trialling’ process, allowing researchers to pilot plans for each step and clarify any ambiguity identified. For instance, the planned search strategy should be trialled within electronic databases to ensure the resulting returns are manageable and relevant. When researchers are satisfied that the plans are clear, relevant and achievable, a final protocol should be agreed.

The protocol should be made accessible online before formal searches have been initiated [55]. Research suggests that SRs with prospectively registered protocols have greater methodological rigour [56] and are more likely to be published in higher impact factor journals [57]. Indeed, the PRISMA guidelines require a priori protocol registration [42]. Perceived barriers may include costs, concerns that others may ‘take’ the idea or excessive workload [58]. However, SR protocols can be made available online with minimal effort and often at no cost [59], via:

•The International Prospective Register of Systematic Reviews (PROSPERO [60]);

•Research Registry [61];

•INPLASY [62];

•OSF Registries [63]; or

•Protocols.io [64].

Registration requires little work beyond that inherent in developing the protocol [55] and may save time by ensuring a clear plan is in place, thereby reducing the opportunity for errors and miscommunications [58]. Instances of ‘stealing’ of SR plans are low [65], and protocol registration may be considered a means of ‘staking a claim’ on a topic [58]. Additionally, researchers may consider publishing their protocol in a relevant journal as an article for peer review, ideally before data collection has commenced. A number of HS-specific journals (e.g. International Journal of Healthcare Simulation) accept review protocols as an article type. Publication of protocols may yield great exposure and engagement, as well as furthering transparency, reproducibility and accountability.

Devoting sufficient time and effort to the initial protocol will help ensure the review can be completed as planned. However, in some instances, a need to revise the protocol may arise. For example, data on a particular outcome may not be presented within included studies and a protocol may need updating to omit this data [55]. Many review registries permit records to be updated, allowing researchers to clearly document and justify changes made [59]. Any amendments to the review protocol should be carefully considered and described in the review manuscript [42].

Step 4: identify relevant literature

Once a protocol is agreed, the process of identifying studies for inclusion can begin. Typically, a SR search strategy will comprise electronic database and reference list screening but may be supplemented with additional search tactics. When composing a search strategy, researchers should carefully consider available resources, the nature of the review, where eligible studies are likely to be published and the time and effort required to enact any search tactic as compared to the likelihood of it yielding additional relevant studies. Researchers should consult Atkinson and colleagues’ [38] reporting standards for literature searches to ensure search quality, transparency and replicability.

Electronic database searches constitute the core work of searching within a SR. Databases available to researchers will vary by institution, and their appropriateness will vary depending on the review’s focus. We recommend including three to five electronic databases, in line with research suggesting most relevant articles are found within a limited number of databases [66]. Where smaller numbers of studies are expected, searching more databases may be reasonable [66]. The following databases are likely to be useful within SRs of HS [66,67]: EMBASE, MEDLINE, Web of Science, Scopus, CINAHL, PsycINFO and Education Resources Information Center (ERIC).

Following database selection, researchers must develop a set of search terms for each. This involves two stages: initial development and agreement of search terms (typically for the MEDLINE database), and the adaptation of these for application in other databases. There is significant work involved in developing search terms, which need to be sufficiently broad to capture all potentially relevant records but precise enough that the number of returns can be screened within available resources. For initial development, it is crucial to identify the different aspects of the review question to be reflected in the search to build ‘sets’ of search terms for each (Table 2 illustrates the appropriate development of sets for our example reviews). To build the initial MEDLINE search strategy, begin the population of each ‘set’ with Medical Subject Headings (MeSH terms) via the use of MeSH Browser [68] (which returns relevant MeSH terms when keywords are entered). Following this, augment subject headings within each set with free-text keywords (i.e. relevant words or phrases that describe the concept/issue/topic/intervention). These can be generated based on researchers’ knowledge but should be supplemented with additional keywords from the search strategies of other related reviews and the consideration of title, abstracts and keywords used within known relevant articles.

Typically, search terms within sets are combined using the ‘OR’ Boolean operator (the instruction of a database to find articles using this term OR that term OR this term etc.), while the sets themselves are combined using the ‘AND’ Boolean operator (the instruction of a database to find articles that use at least one term from Set 1 AND at least one term from Set 2). When a full search strategy is available, researchers should carefully consider the application of any search filters (e.g. language, peer-review status), as their use substantially impacts the returns generated and requires justification. It is important to ensure that the same search filter can be applied across each database.

Finally, researchers will need to add wildcards or truncation to their search strategy as appropriate. This involves adding a term or symbols to individual search terms to allow identification of different forms of those words (e.g. entering ‘simulation’ will return records using that term only, entering ‘simulat*’ will return records using simulations, simulates, simulated, etc.). This allows researchers to capture plurals (e.g. measurement and measurements), alternate spellings (e.g. behaviour and behavior) and potentially different word endings (e.g. mannequin and mannekin). Wildcards also support researchers in searching within specific parts of an article (e.g. ti,ab. entered on MEDLINE will implement searches for the relevant term only in articles’ title or abstract) and searching for words only when used in proximity (e.g. ‘high adj1 fidelity’, ‘task adj1 trainer’). The symbols for enacting truncation and wildcards differ across databases and researchers should use the search guidance available within databases to ensure an equivalent search. HS researchers may find search ‘hedges’ [69] – pre-defined comprehensive combinations of search terms for a concept/topic created by an expert searcher, typically for a single database, useful to save time or narrow the scope of the search.

Best practice suggests engaging a research librarian in, at minimum, review of the initial search terms, with discernible positive effects of librarian involvement evidenced [70]. The Peer Review of Electronic Search Strategies (PRESS) [37] checklist can be applied by researchers to critically appraise their search strategy. In advance of formally conducting the search, it may also be useful to first identify a ‘seed set’ of articles (i.e. a sample set of 4–6 articles that meet inclusion criteria and would be expected to be captured by the search strategy); these can then be used to test the thoroughness of the database searches, ensuring that search terms and processes would capture such relevant articles. Once an initial search strategy has been agreed, researchers must adapt it for use within the remaining databases. While the MeSH browser yields subject headings existing in MEDLINE and PubMed, other databases will have their own, tailored to their specific focus. Researchers can identify these terms within each database’s relevant thesaurus or library.

Several issues may arise during the process of developing the search strategy. An excessive number of returns will require researchers to reconsider keywords, explore an additional ‘set’ of search terms, and/or incorporate the use of additional wildcards (e.g. proximity operators). Conversely, few returns may necessitate broadening the search through removal of a ‘set’ of terms, introducing additional keywords, removing search filters and/or revising use of truncation or wildcards. Once the electronic database search strategy has been finalised for each database, its development is complete. The PRISMA 2020 statement [42] requires presentation of a full search strategy for each database, typically within the supplementary materials. Researchers should ensure that the search strategy for each database is saved, allowing searches to be enacted quickly and consistently and facilitating the updating of searches later. Updating searches prior to completion of the SR may be considered to ensure the most recent publications are included. There is no clear guidance on acceptable time lapse since completion of searching, although some journals require searches to have been completed within the previous six months.

Once plans for electronic searches have been agreed, researchers must decide on the use of any supplemental search tactics to identify additional relevant studies missed by electronic searches and to ensure the comprehensiveness of their overall search strategy. Table 3 provides an overview of supplemental search tactics and guidance on employing each. While the target within a SR is identifying all relevant studies, it is recognised that there will be instances where relevant articles are not identified, which is not necessarily an indicator of a poor-quality search. Where researchers have adequately planned and carefully implemented a search strategy, the likelihood is that the resulting review and its conclusions will be useful and well-founded despite this [38].

Table 3:

Supplemental search tactics for consideration based on resources available and anticipated body of research

Search tactic	Description and guidance
Reference list screening (RLS)	RLS is also referred to as ancestry searching or backwards citation chasing. It involves manually screening the reference lists of all studies identified for inclusion through the electronic database searches and is typically recommended within systematic reviews. The rationale for this process is that ‘like cites like’ and that studies relevant for inclusion may be expected to cite other articles that have a similar focus/aim/intervention. RLS may also take the form of examining the reference lists of related review articles that have been identified. RLS will generally yield a small number of additional returns; identification of many additional articles should be considered a cause for concern as it suggests that the electronic searches were potentially inadequate. Review of the new articles will support the researchers in identifying search terms that were missing and should prompt consideration of whether searches should be re-run to ensure all relevant studies have been included.
Direct contact searching	Involves contacting researchers working in the field to identify relevant unpublished research, or new and/or ongoing research articles. Individuals contacted are usually the authors of studies already identified as eligible for inclusion or active researchers in the focal area. Direct contact searching can be disadvantageous for several reasons including its time-consuming nature, difficulty identifying current contact details for target researchers, a low response rate and the need for repeated contacts, potentially low-quality returns in the form of unpublished data and its suitability only for relatively recent research [71]. Direct contact searching is most likely to be useful within intervention-focused reviews as a means of reducing the likelihood of publication bias impacting results. However, in a field like HS with well-indexed journals, the resources and time required to enact direct contact searching are unlikely to result in a meaningful yield.
Forwards citation tracking	Also referred to as citation chasing. The process involves reviewing articles that have cited studies which have been identified as suitable for inclusion – the converse of RLS – and may be particularly useful within reviews of rapidly evolving research areas. Platforms that support the examination of citations of an individual study include Web of Science, Scopus and Google Scholar. Both RLS and forwards citation tracking can typically be completed by one researcher, and the number of records examined, number of full-texts screened and number of articles ultimately included from these efforts should be recorded.
Grey literature searches	Involves the use of web searches (e.g. Google Scholar) or the use of databases focused on grey literature (e.g. ProQuest) to identify relevant non-peer-reviewed materials, often within intervention-focused reviews as an attempt to avoid the over-estimation of intervention effects. Examples may include study protocols for ongoing studies, student theses, conference abstracts or technical reports. Challenges include a lack of best practice guidance, difficulty conducting replicable web searches, materials of low methodological/reporting quality and the resource intensive nature of searches. However, prestigious evidence synthesis organisations (e.g. Cochrane, Campbell Collaboration) recommend its use.
Hand searching	Involves manually reviewing the records published within specific journals considered relevant to the topic. Researchers may consider this appropriate when most relevant outputs are published within a core set of journals or where core journals are not indexed within the electronic databases used. However, it should be noted that although HS-specific journals exist, many of the most influential simulation articles have been published in other outlets, suggesting a limited utility of hand searching within HS-focused reviews. A low precision, and lengthy process, have been identified as deterrents to its use.

Note: The guidance within this table has been informed by the work of Cooper et al. [71].

Step 5: screen and select studies

Once a researcher has retrieved relevant records from the electronic databases, the process of screening these and selecting studies for inclusion begins, comprising de-duplication, initial screening and second-stage screening.

Searching multiple databases inevitably results in the retrieval of duplicate publications (e.g. different databases returning the same publication, or different versions of the same study such as a conference abstract and a full text article) [72]. Each study should only be considered once when screening; therefore, researchers should use an electronic reference management system (e.g. EndNote, ProCite) to facilitate the identification and removal of duplicates (i.e. de-duplication) [18,72,73]. De-duplication lessens workload [72], ensuring reviewers avoid wasting time screening the same citation multiple times.

During the initial screening of remaining records’ titles and abstracts, researcher(s) will evaluate each through application of the pre-defined eligibility criteria. Two researchers screening records independently is ideal to ensure transparency and minimise bias [18,42] and increases the number of relevant studies identified [74]. Each researcher screens each record independently and determines whether there is enough information in the title and abstract to include or exclude or whether it is potentially relevant and requires further review. Next, researchers compare assessments; records should be excluded if both reviewers agree, and records should be moved to the next stage (i.e. full-text screening) if both reviewers indicate eligibility or potential eligibility. Records with differing judgements should be reviewed together, and consensus achieved via discussion [74]. Those articles identified as relevant, or potentially relevant, should be obtained for second-stage screening [73]. In practice, engaging two researchers in study screening may not be feasible and will require consideration of capacity and resources available. Using a second reviewer during full-text screening only may be a way to reduce resources, while still limiting bias [74].

Second-stage screening involves examination of the full text of records retained from title and abstract screening, with involvement of two researchers strongly recommended [74]. Each researcher should scrutinise the entire text of each record to confirm/disconfirm relevance, guided by the eligibility criteria, and decide whether to ‘include’ or ‘exclude’ each article. For each excluded article, a justification should be documented [72]. When independent screening is complete, the researchers’ logs should be compared in the same manner described previously. The involvement of a third researcher, or wider research team, is important for resolving any remaining disagreements or inconsistencies. Ultimately, a final list of included studies from electronic database searches will have been identified [18]. The use of supplemental search tactics (e.g. reference list reviews) can be completed at this point (see Table 3).

Adequate documentation of screening (i.e. a record of the count of records identified, screened, included, excluded and reasons) is essential [72] for completing the PRISMA flowchart [18]. We direct readers to Rethlefen and Page [75] for further useful guidance on tracking records throughout the SR process.

Step 6: extract data and finalise a record of studies

When the studies for inclusion have been finalised, data extraction can begin. Data extraction refers to the process of recording relevant information from studies for consideration, refinement and analysis [76]. The data extraction process comprises selecting data collection tools, constructing data collection forms and abstracting, managing and archiving data [77].

Data extraction can be conducted using either paper forms, electronic forms (e.g. Microsoft Word), spreadsheet software (e.g. Microsoft Excel) or commercial data systems (e.g. Covidence) that allow online form building, data entry by several users, data sharing and efficient data management [77]. Electronic forms are more efficient and less error-prone than paper forms, while data systems require greater financial investment and training [77,78]. However, no single data-extraction tool is best for all SRs in all circumstances. Therefore, researchers should carefully consider the volume, nature and anticipated complexity of the data and resource availability, before selecting a tool [78]. We centre our subsequent guidance on using an electronic form via Microsoft Word, which is generalisable to other tools.

The data extraction form should be easy to use, standardised and allow for the structured collation of data to sufficiently represent the study [77], minimising the need to return to the source [79]. Careful development, pilot testing and application are crucial, and should involve iterative input from all researchers. First, researcher(s) should prepare outlines of the tables and figures for the SR (e.g. table of study characteristics, tables/figures offering a summary of the findings) to support determination of the data required [77,79]. Next, data extraction variables/elements should be logically grouped (see Table 2 for illustration of data extraction for example reviews). Subsequently, the form can be presented using a word processor, in a table format capturing all studies or using individual forms that can later be tabulated. A codebook should also be generated, containing relevant definitions and details on how to capture and code data for each variable. Providing guidance on the form itself supports quality and consistency across researchers [77]. Finally, the form should be piloted among multiple researchers, extracting data from at least three articles [77], with modifications made as necessary to ensure agreement. The ‘seed set’ of articles developed within Step 4 for trialling the search strategy, could also be used within the pilot data extraction.

It is imperative that researchers (especially SR novices) receive training to familiarise them with the data extraction process and clarify any uncertainties. Data extraction errors are frequent [80]; therefore, two researchers should be involved [4,77], which has been found to result in fewer errors [81]. Data extractors from complementary disciplines (e.g. a methodologist and subject-matter expert) [77,79] may valuably support one another. Each researcher should extract data for all studies independently, then compare their records to identify discrepancies. Discrepancies can be resolved through discussion, with re-engagement with the source document as necessary. If consensus cannot be achieved, a third researcher may adjudicate. A record of the presence and resolution of disagreements, and raw copies of original individual data extraction forms or files (this will differ based on whether the researchers have opted to extract data from all studies into one table or used individual forms), should be retained [79]. Ultimately, files should be merged into one final single document that offers a complete and agreed record of data extraction.

To facilitate the later synthesis of extracted data, researcher(s) may have to perform a degree of coding when initial data extraction is completed, introducing homogeneity within the data extraction records [82]. This can be achieved via content analysis, which involves researchers developing categories/codes to capture the extracted data either de novo (i.e. inductive coding [83]) or through the application of existing frameworks to standardise the data (i.e. deductive coding [83]). Such coding allows researchers to establish the frequency and consistency of findings, methods or study characteristics and helps to determine what is ‘key’ across studies or how they reflect existing theory or understanding.

Step 7: critically appraise studies

Given the implications of SR for practice and research, consideration of trustworthiness via critical appraisal of included studies is core to conducting a high-quality SR [42]. Researchers will encounter two relevant concepts:

•Risk of bias: how methodological limitations or flaws within design, conduct or analysis may compromise resulting data [84]; and

•Study/methodological quality/rigour: how closely studies resemble best practice in conduct and design [84].

Focusing on risk of bias is recommended [84], particularly within effectiveness reviews. Several tools and corresponding guidance on their use, have been developed to support researchers in formally appraising studies. Our non-systematic examination of recent HS-related SRs revealed a great number of tools have been used to support critical appraisal, including:

•ROB-2 (Cochrane’s risk-of-bias tool for randomised trials) [85];

•Mixed Methods Assessment Tool (MMAT) [86];

•COSMIN Risk-of-Bias tool to assess the quality of studies on measurement properties [34];

•JBI’s range of critical appraisal tools [87];

•Critical Appraisal Skills Programme (CASP) checklists [88];

•Medical Education Research Study Quality Instrument (MERSQI) [39]; and

•Risk of Bias in Non-randomized Studies of Interventions tool [40].

When selecting a tool, researchers should consider the likely study designs, and the experience within the research team. Considering critical appraisal tools used in similar SRs, consulting colleagues and experts in evidence synthesis, and examining other review articles that have used a tool under consideration may support decision-making.

Typically, two researchers will independently complete critical appraisal with discussion to agree final ratings, though this process is sometimes conducted in tandem, with researchers working together to finalise judgements for each item and study. Numerous challenges exist within the appraisal process: considerable time is required [89], inter-rater agreement is variable even with well-established tools [89], and performance is likely impacted by familiarity with the tool, between co-raters and with critical appraisal [89]. Accordingly, the tool’s user-friendliness, sufficient allocation of time, researcher experience and initial training, review and piloting of the tool are crucial considerations to ensure the process is completed effectively.

Step 8: synthesise and write-up

When the data extraction record is finalised, the process of data synthesis commences. SRs in HS frequently emphasise diversity across included studies (e.g. heterogeneity of outcome measures [27,28]) which is likely to render synthesis via meta-analysis (see Table 1) challenging. Accordingly, within both effectiveness reviews and other review types, a range of alternative approaches are often more suitable. We focus our guidance herein on the process of narrative synthesis, which we expect to be suited to most reviews of the HS literature, though Table 4 details alternative tactics for synthesising data in SRs that can also be applied.

Table 4:

Supplemental or alternative tactics for consideration based on focus of review and nature of anticipated body of research

Synthesis tactic	Description and guidance
Summarising effect estimates	involves using descriptive statistics (e.g. using median, interquartile range, range) to provide information on the range and distribution of observed effects and is typically used to synthesise results in cases where it is difficult to undertake a meta-analysis [93]. McKenzie and Brennan [93] provide guidance on how best to complement reporting of the summary statistics with visual displays. Limitations include not accounting for differences in the relative sizes of the studies, and lack of evaluation of the performance of these statistics applied in the context of summarising effect estimates [93].
Combining p-values	(i.e. significance levels) of each study involves conducting a statistical test to determine whether there is evidence that the intervention is effective in at least one study. These can be used to synthesise results when included studies report minimal information beyond p-values and direction of effect, results of non-parametric tests or results of heterogeneous outcomes and statistical tests [93]. Several methods exist, of which the most commonly applied is Fisher’s method [94] which combines the p-values from statistical tests across studies using a formula. Researchers should consult guidance from Cooper et al. [94] on applying Fisher’s method. Limitations include the lack of information on the magnitude of effects, and difficulty in interpreting the test results when statistically significant [93].
Vote counting based on direction of effect	involves comparing the number of effects showing benefit to the number of effects showing harm for a particular outcome based on the observed direction of effect alone (regardless of statistical significance), thereby creating a standardised binary metric [93,95]. This can be used to synthesise results when only direction of effect is reported, or there is inconsistency in the effect measures or data reported across studies [93]. Conventional forms of vote counting based on the count on the statistical significance and direction of the effect estimates, which has considerable limitations and should be avoided [93,96].

Note: The guidance within this table has been informed by the work of McKenzie and Brennan [93].

Narrative synthesis is a textual approach to collating findings, involving the use of text to ‘tell the story’ of the findings and learning [16]. Narrative synthesis is inherently subjective to an extent [90], and it is important that researchers pursue a rigorous and transparent approach to reduce potential bias. Popay et al. [16] offer comprehensive guidance on the conduct and reporting of narrative synthesis, defining four main elements. First, developing a theory about why the intervention works and for whom. This element is applicable to effectiveness SRs and potentially others such as experiential or qualitative SRs. In addressing this, researchers consider the rationale for the intervention outlined in studies seeking to identify the pathways through which an intervention has its effects [91]. Researchers focusing on the effects of a HS intervention will need to organise the results of included studies to enable description of emerging patterns in terms of the direction and size of effects [16]. Second, researchers must develop a preliminary synthesis of findings of included studies. This involves determining how best to bring together, organise and describe the results from included studies. Ryan [91] offers valuable guidance on varied ways to achieve this:

•Textual descriptions summarising features for each study;

•Grouping the studies (e.g. by intervention, population groups, study design, etc.);

•Tabulating results across the included studies to identify patterns;

•Transforming the data (e.g. into a common statistical or descriptive format; see description of coding in Step 6);

•Using vote counting to provide an initial description of results – typically acceptable only as an interim step in assessing data;

•Translating data using thematic or content analysis to identify recurrent concepts or themes.

Third, researchers must explore relationships in the data within and between studies. Researchers should explore two key relationships: those between characteristics of individual studies and their findings, and those between the findings of different studies [16]. To achieve this, the researcher should examine the extracted data with a view to looking at the relationships between study results and key aspects of the studies and comparing these relationships across studies. At this point, the researcher should consider whether variability in study design, populations, interventions or outcomes is likely to explain these differences, in addition to the effect of heterogeneity [91]. The following approaches to explore the relationships between and within studies can be used [91]:

•Visual tools (e.g. forest plots, graphs) can help to organise and present results. These should supplement, rather than replace, a textual description of results.

•Subgroup analysis can be used to explore the impact of potential effect modifiers on intervention effects.

•Developing conceptual models (e.g. idea webbing, concept mapping).

Finally, researchers should appraise the rigour of the synthesis. This comprises examining the quality or risk of bias of included studies (see Step 7). Researchers should also consider the trustworthiness of their processes for collating evidence, such as critical reflection on strengths and weaknesses of the methods used to narratively synthesise included studies [91]. When reporting on the narrative synthesis of studies, researchers should adhere to the Synthesis Without Meta-analysis (SWiM) guideline [92] to ensure a clear description of their procedures.

The remaining work of the SR consists of preparing a written manuscript. There is significant value in identifying a target journal prior to commencing the write-up, as this will guide certain elements (e.g. structure, word count etc.). Considering other SRs published in the target journal may also support authors in preparing a suitable manuscript. A SR manuscript will include the following sections: Introduction, Methods, Results and Discussion along with an Abstract summarising the conduct and core messages of the SR.

Drawing conclusions in a SR typically involves considering the implications for practice, policy and future research. When providing implications for theory, practice or policy, researchers should address the review question and consider the intended end-users’ needs. These should be supported by the evidence synthesis (i.e. they should not be speculative), with acknowledgement of the strength and limitations of the evidence [45,97]. In stating recommendations for future research, researchers should avoid vague statements (e.g. ‘more research is needed’) and identify specific areas where evidence is lacking and warrants prioritisation [45]. The PICO domains may be useful here to inform aspects that could be addressed more effectively in the future [97]. Further, the domains of the GRADE [98] certainty framework can support researchers in interpreting their evidence and drawing conclusions about future research and practice.

Researchers must carefully consult and adhere to the PRISMA 2020 checklist [42], which offers detailed instruction for writing each manuscript component and for appropriately detailing funding, conflicts of interest and author contributions. This guidance includes 12 items that dictate good practice in the presentation of an Abstract, the preparation of which is often most appropriate when the main manuscript text has been completed. Writing within the confines of limited word counts is challenging, and the considered use of tables and figures, as well as supplementary materials (e.g. for detailed study-by-study summaries, full search strategies), is required to ensure that the SR’s data is presented comprehensively and effectively [99].