Evidence-to-decision frameworks: a review and analysis to inform decision-making for environmental health interventions

Abstract

Background

Evidence-to-decision (EtD) frameworks provide a structured and transparent approach for groups of experts to use when formulating recommendations or making decisions. While extensively used for clinical and public health recommendations, EtD frameworks are not in widespread use in environmental health. This review sought to identify, compare and contrast key EtD frameworks for decisions on interventions used in clinical medicine, public health or environmental health. This information can be used to develop an EtD framework suitable for formulating recommendations for interventions in environmental health.

Methods

We identified a convenience sample of EtD frameworks used by a range of organizations. We searched Medline for systematic reviews of frameworks. We summarized the decision criteria in the selected frameworks and reviews in a qualitative manner.

Findings

Fourteen organizations provided 18 EtD frameworks; most frameworks focused on clinical medicine or public health interventions; four focused on environmental health and three on economic considerations. Harms of interventions were examined in all frameworks and benefits in all but one. Other criteria included certainty of the body of evidence (15 frameworks), resource considerations (15), feasibility (13), equity (12), values (11), acceptability (11), and human rights (2). There was variation in how specific criteria were defined. The five identified systematic reviews reported a similar spectrum of EtD criteria.

Interpretation

The EtD frameworks examined encompassed similar criteria, with tailoring to specific audience needs. Existing frameworks are a useful starting point for development of one tailored to decision-making in environmental health.

Funder

JPB Foundation.

Introduction

It is widely recognized that environmental pollution is an important determinant of health and that interventions, in particular policy recommendations, are a key means by which population health can be improved. The formulation of trustworthy and impactful recommendations and policies on environmental health interventions is a complex task, which requires the identification of all relevant data and evidence, their critical appraisal and synthesis and translation into a recommendation or policy. The processes and methods for hazard identification and risk assessment of environmental substances based on systematic reviews of the evidence have advanced considerably in the last decade [1,2,3,4]. These approaches have been largely based on methods developed for clinical medicine [5], which have been expanded to encompass public health interventions [6], diagnostic test accuracy and impact [7], coverage decisions [8, 9], and health technology assessments (HTA) [10], among others.

The final step in the process of formulating evidence-informed recommendations and policies involves the translation of data and evidence on various decision factors into an explicit recommendation or policy. Evidence-to-decision (EtD) frameworks provide a structured and transparent approach for groups of technical experts or policy-makers to accomplish this step [6, 11]. These frameworks include explicit criteria which the group considers individually and in aggregate focused on the relative benefits and harms, as well as other considerations. Such frameworks can facilitate: i) consideration of all relevant criteria in the decision-making process; ii) examination of the pros and cons of each intervention option; iii) presentation of relevant evidence for each criterion; iv) identification of the reasons for any disagreement within the expert group; v) transparent reporting of the decision-making process; and vi) crafting of the rationale statement for each recommendation. Populated EtD frameworks can also facilitate implementation by assisting the end-user in understanding how and why specific recommendations were made, and by providing data and evidence on each decision criterion which may facilitate local adoption or adaptation [12].

In environmental health, once hazards are identified and risks assessed, organizations may want to examine mitigating and prevention interventions and make recommendations and policies based on systematic reviews of research evidence and other data and information. An EtD framework suitable for environmental health interventions will facilitate this process. However, such frameworks have not been widely used in this field.

The objectives of this review were to identify, compare and contrast key EtD frameworks for interventions in clinical medicine, public health and environmental health and to summarize the main decision criteria across these frameworks. The identification of these criteria will inform the development of an EtD framework for use in formulating recommendations regarding mitigating and prevention interventions related to exposure to harmful substances in the environment.

Methods

In order to identify relevant, existing EtD frameworks, we took two approaches: a search of the peer-reviewed literature for systematic reviews of EtD frameworks, and identification of frameworks used by a range of organizations which make decisions or formulate recommendations in clinical medicine, or public or environmental health.

Scope

The focus of this paper is on the substantive criteria for decision-making with respect to interventions, including both normative criteria (what should be done) and feasibility criteria (what can be done) [13]. Criteria related to the process of decision-making are beyond the scope of this work.

There are an increasing number of organizations which use systematic and transparent approaches to synthesize evidence in environmental health for hazard identification [14], risk assessment [15], and for the synthesis of the benefits and harms of interventions [16]. These organizations were not included in the current analysis because they do not make recommendations on interventions to mitigate the effects of harmful exposures.

Bibliographic database search and article screening

To identify relevant EtD frameworks, we searched Medline via PubMed for systematic reviews of EtD frameworks, templates or tools published in English and indexed in the 10-year period up to 7 January 2021. The complete search strategy is found in Annex 1. We also solicited the advice of guideline development experts for any additional potential citations. Publications were included if they were systematic reviews of frameworks, tools or templates for formulating decisions or recommendations, or for priority setting related to the use of interventions (including diagnostic tests) in clinical medicine, public health, or environmental health. The included frameworks focused on population-level interventions; decision tools for the provider-patient interaction and patient decision aids were excluded. Frameworks which focused exclusively on economic considerations were also excluded. The setting for both decision-making and implementation of recommendations was not restricted.

Two persons (SLN and MTA) screened the titles and abstracts, and full-text versions were retrieved for studies which potentially fulfilled inclusion criteria. Consensus was achieved between the two reviewers for final inclusion in the review.

Search for EtD frameworks used by key organizations

In addition to the systematic review described above, we examined a broad range of organizations including clinical and public health guideline development groups in academia and the private sector, healthcare provider professional organizations, governmental agencies and international organizations. Given the purpose of this review, we focused particularly on organizations that work in the field of environmental health. Organizations were identified based on author knowledge of prominent guidelines in public health and clinical medicine, by snowballing (reviewing the origin of or basis for identified frameworks), and by conferring with a broad international network of guideline developers based on our experiences and contacts.

We used this pragmatic approach for several reasons. First, because of the dominance of the GRADE (Grading of Recommendations Assessment, Development and Evaluation) EtD framework over the last 15 years [5, 11, 17], many organizations either use GRADE or a modification thereof. Thus, a more exhaustive search was unlikely to yield novel or unique frameworks. Second, because the methods used by guideline development organizations are infrequently published in the peer-reviewed literature, an extensive hand search of the grey literature including organizational web-sites would have been required, which was infeasible.

Data extraction and synthesis

For each systematic review identified in our search, a single author (SLN) extracted key information describing the focus of the review, time period searched, and the main findings, including an overview of the criteria identified across the included frameworks. We did not extract data on each individual framework included in each review.

For the EtD frameworks used by key organizations, we extracted data on organizational characteristics, the methods used to develop the framework, the funders and any declared interests of the developers of the framework, how evidence should be used to inform each criterion in the framework, quality assessment of individual studies and of the body of evidence, the specific EtD criteria, and the types of conclusions or recommendations formulated. One author extracted these data (SLN) and a second checked them (MTA); disagreements were discussed and consensus reached. Data were extracted into a template in Excel (Microsoft Corporation, Redmond WA, USA).

For the key organizations, the primary decision criteria for each EtD framework were extracted from the main published reports, and specifically from the identified template or list of criteria. Criteria which are mentioned only in the narrative text accompanying a framework were not extracted unless the text suggested that they were consistently applied in decision making.

Data were summarized in a narrative, qualitative manner. Given that the included frameworks were a convenience sample, descriptive statistics and statistical comparisons are not meaningful. Quality assessment of the frameworks was not performed as there is no standard for such an assessment.

Role of the funding source

The study funder (JPB Foundation) had no role in the study design; data collection, analysis or interpretation; in writing the report; or in the decision to publish.

Results

Identified systematic reviews of EtD frameworks

The bibliographic database search for systematic reviews of EtD frameworks yielded 399 citations, of which four fulfilled inclusion criteria [7, 10, 18, 19] (Table 1, Annex 2 and 3). One additional study which fulfilled inclusion criteria was identified by the co-authors [8]. Each of these reviews included a cohort of EtD frameworks which the authors had systematically identified, with a particular focus: multicriteria decision analysis in HTA [10], vaccine adoption into national programs [18], diagnostic tests [7], decision making in local low-income settings [19], and frameworks for informing health system coverage decisions [8]. Each review summarized the main EtD criteria identified across their included, individual frameworks, and these main criteria are presented in Table 1. All five reviews included the balance of benefits and harms and consideration of resource use or cost-effectiveness. Only the review by Mustafa and colleagues [7] included an assessment of certainty of evidence. Considerations of equity, acceptability and feasibility were included in only two reviews [10, 18]. The EtD criteria outlined in these five reviews generally corresponded to the main criteria in the EtD frameworks of the selected key organizations (see Results section 2).

Table 1 Systematic reviews of evidence-to-decision frameworks: review characteristics and key findings. Decision criteria are the main criteria that the review authors identified across the frameworks which were included in their review

EtD frameworks used by key organizations

Fourteen organizations that use EtD frameworks for recommendation formulation or decision-making were examined in detail (Table 2). One organization, the GRADE Working Group, is an informal network of individuals from a broad range of organizations, including academic institutions, national and international guideline development agencies, and healthcare provider organizations, among others [42]. The GRADE Working Group does not, itself, publish guidelines, but rather develops processes and methods for use by other organizations which develop guidelines. The other 13 organizations develop guidelines for specific audiences and with a clearly defined scope or set of topics: five are agencies of national governments [25, 31, 33, 35, 39], two are related to the World Health Organization (WHO) [37, 38], one is a U.S. State agency (California Environmental Protection Agency (CalEPA) [22]) and the remainder are non-governmental organizations or academic groups [2, 21, 24, 27, 28].

Table 2 Evidence-to-decision frameworks: Characteristics of the organizations that developed the frameworks

Two organizations focus primarily on clinical care: the Scottish Intercollegiate Guideline Network (SIGN) [34] and the U.S. Preventive Service Task Force (USPSTF) [36]. The Guide to Community Preventive Services (GCPS) focuses on interventions aimed at groups, communities or health systems [26] and WHO and WHO-INTEGRATE primarily on public health interventions [37, 38]. Three organizations are oriented to health technologies, with a prominent focus on economic evaluations and resource considerations [24, 27, 28]. The UK National Institutes for Health and Care Excellence (NICE) examines a broad range of clinical, public health, and social interventions [32]. The Advisory Committee on Immunization Practices (ACIP) focuses exclusively on vaccine recommendations for U.S. populations [20]. Breast Cancer Prevention Partners (BCPP) [21], CalEPA [22, 23], and the Navigation Guide [2, 16, 40] focus on the human health effects of hazardous substances in the environment.

Development process and expertise

For the organizations that developed de novo EtD frameworks and described the process for developing them, all used an iterative approach based on an examination of other organizations, with input from experts in guideline methods and evidence synthesis (Table 2). The most comprehensive approach was taken by Rehfuess and colleagues [38] in development of the WHO-INTEGRATE framework. Their approach included development of a theoretical framework, a review of WHO basic documents, a systematic review of EtD criteria, input from a range of stakeholders, and a thematic analysis to identify key domains [41].

It was difficult to discern the expertise of contributors to framework development; most groups appeared to consist mainly of academic, generalist guideline methodologists with either clinical or public health experience. Social scientists led work on WHO-INTEGRATE [38] and contributed to the GRADE public health framework [6]. Eight of the 14 organizations reported who funded the development of the EtD framework [2, 11, 21, 24, 27, 34, 37, 38] and five reported declarations of interest among framework developers [2, 11, 20, 24, 38]; the remaining organizations did not provide this information.

Decision criteria

Eighteen frameworks were examined in detail: one from each of the 14 organizations, except for CalEPA and GRADE, where two [22, 23] and four [6, 9, 11, 17] unique frameworks were examined, respectively. There were significant commonalities across frameworks in the criteria for formulating recommendations (Table 3 and Fig. 1). Unsurprisingly, all included consideration of benefits, except the CalEPA framework for pesticides [22] which examines risks (of environmental exposures) and not benefits. All frameworks included an assessment of harms of the intervention under consideration. Fifteen of the 18 frameworks included some assessment of certainty or quality of the body of evidence across outcomes in the decision-making process: BCPP, one of the CalEPA frameworks, and ISPOR (International Society for Pharmacoeconomics and Outcomes Research) did not [21, 22, 28]. Some measure of costs, resource use or cost-effectiveness was found in all frameworks except BCPP, USPSTF and the GCPS [21, 26, 36]. Other decision criteria were variably included: feasibility (13 frameworks), equity (12), values (11), and acceptability (11). Only two frameworks included human rights: WHO [37] and WHO-INTEGRATE frameworks [41].

Table 3 Criteria included in evidence-to-decision frameworks

Fig. 1

Evidence-to-decision criteria for each key organization. Legend. Across the top of the figure, broad categories of evidence-to-decision criteria are presented. For each listed organization, the cell is shaded if their evidence-to-decision framework encompasses one or more criteria within a category.

“Priority of the problem” was explicitly included in the EtD frameworks based on GRADE [20, 34, 37], as well as the BCPP [21], EVIDEM (Evidence and Values Impact on DEcision Making) [24], ISPOR [28,29,30] and Navigation Guide frameworks [16]. However, in the background materials for several other frameworks, the burden of the disease was mentioned as an important consideration when prioritizing an intervention for development of a guideline or policy.

The meaning of the criterion “values and preferences” varied over time and across publications. This phrase encompasses two different constructs: the relative value that persons affected by the recommendations place on the outcomes of the intervention and the preferences such persons have regarding the intervention options. The GRADE Working Group initially combined these two constructs [5] with a definition encompassing both [43]. The 2016 update of the GRADE EtD framework introduced two separate criteria: variability of the value affected persons place on the main outcomes, and acceptability of the intervention [11] to various stakeholders. Other key organizations include a decision criterion related to values using verbiage closely aligned with GRADE [16] and focused exclusively on the relative value of outcomes [32, 34], or they include the concept but with a somewhat different meaning [20, 22, 28].

Evidence used to inform each criterion when organizations make decisions

All of the frameworks report that the EtD criteria should be informed by evidence obtained from a variety of sources, with an emphasis on systematic reviews of research evidence (Table 2). Twelve of the organizations recommend an assessment of the quality (validity or certainty) of the body of evidence for important outcomes as part of the evidence review, and most recommend the GRADE system with or without modifications. Only BCPP [21] and one of the CalEPA guidelines [22], do not recommend such an assessment. The approaches used by the GCPS [26] and USPSTF [36] differ somewhat from GRADE in their assessment of the “strength of evidence” and “certainty of net benefit”, respectively. The EtD frameworks with an economic focus describe varied approaches to assessing quality of the body of evidence [24, 27, 28].

Nomenclature for recommendations

Nine of the included organizations provide specific guidance on the nomenclature for the types of recommendations formulated based on the EtD criteria (Table 2). The most common categorization of recommendations among those nine organizations was two levels of strength both for and against a recommendation, i.e., four categories [11, 34, 37]. GRADE has extensive documents on this issue and uses the terms “strong” and “weak or conditional “[5]. The USPSTF has five categories of recommendations [34], while ACIP [37] and the GCPS [35] have three. The economic-focused frameworks refer to “value” in various ways [24, 27, 30]. NICE uses the wording of recommendations to reflect the strength of the evidence (e.g., offer, advise, consider) rather than standardized terms to represent the strength of the recommendation [32].

Five organizations provide explicit guidance on the situations where recommendations cannot be formulated due to insufficient evidence or a close balance between benefits and harms (Table 2) [11, 20, 26, 32, 36]. The USPSTF provides the most detailed guidance on this situation [44]. Ten of the organizations suggest including knowledge or research gaps with the recommendations, with particular emphasis in ACIP, NICE, GCPS and the USPSTF.

Specific evidence-to-decision frameworks

GRADE includes EtD frameworks for four different purposes: i) clinical recommendations, individual perspective; ii) clinical recommendations, population perspective; iii) coverage decisions; and iv) health system and public health recommendations/decisions. (There is a fifth GRADE EtD framework for diagnostic, screening and other tests but this was not included in this review of interventions considered most relevant for application to environmental health [11].) These frameworks are all very similar, all with 12 criteria covering the same concepts, with some variation in verbiage, tailored to the different audiences (Table 3). There is more emphasis on resource considerations, equity, acceptability and feasibility for health systems and public health decisions than for individual patient clinical recommendations [6, 9].

ACIP’s EtD framework is derived directly from GRADE [20], while NICE [31] and SIGN’s [34] EtD criteria closely resemble those of GRADE. WHO uses the GRADE EtD framework which was current when their guidance was published in 2014 [37].

The WHO-INTEGRATE EtD framework version 1.0 [38, 41], first published in 2019 (Table 2), was developed in response to a perceived need to take a complexity perspective into account, to develop a conceptual framework to underpin the EtD, and to incorporate public health and WHO-specific values when developing WHO guidelines. WHO-INTEGRATE includes six broad criteria [38] (Table 3), with quality of evidence applied to all criteria. Each criterion includes sub-criteria: for example, the main criterion “Health equity, equality and non-discrimination” [38] includes impact on health equality and/or health equity, distribution of benefits and harms, affordability, and accessibility.

The frameworks that focus on interventions in environmental health were generally less well developed and lacked specificity compared with the clinical and public health frameworks [2, 21, 22] with the exception of a recently published guide to alternatives analysis for chemicals in consumer products [23].

Additional details on each of the included frameworks are provided in Additional file 1.

Discussion

A number of EtD frameworks have been developed in a variety of clinical, and public and environmental health disciplines, and there is significant overlap in the criteria used to inform recommendations or decisions across these frameworks. Benefits and harms are almost universally included, and an assessment of certainty or quality of the body of evidence and some measure of resource use were included in most frameworks examined. Other decision criteria such as values, equity, feasibility, and acceptability were variably included, while only two frameworks encompassed human rights. There was variation across frameworks in terminology, definitions and presentation of EtD criteria. The five systematic reviews of EtD frameworks describe decision criteria that were similar to those presented in the frameworks from key organizations.

The 18 frameworks identified for 14 key organizations focused mainly on clinical medicine or public health interventions. The GRADE framework is the most well developed, with extensive information and guidance on its methods. A number of the key organizations examined have adopted GRADE, often with minor modifications [16, 20, 32, 34, 37]. Three of the four frameworks related to environmental health lacked detail on the specific criteria for decision-making [2, 21, 22]. Frameworks originating in the HTA realm unsurprisingly focus on economic considerations [24, 27, 28], while also including criteria similar to those encompassed by other frameworks.

General aspects of evidence-to-decision frameworks

The process and methods for developing EtD frameworks were often superficial and poorly reported. Only WHO-INTEGRATE [41] and the GRADE coverage [8] frameworks were developed following a systematic review of existing frameworks or potential decision criteria. Only WHO-INTEGRATE describes a conceptual framework [38] underpinning the EtD criteria. The lack of a conceptual framework represents a significant weakness in most frameworks: key considerations may be missed and long-standing criteria may be perpetuated without adequate scrutiny.

GRADE has dominated guideline methods in health care and public health over the last 15 years. This has led to productive collaborations, standardization of processes and methods, and the development of detailed methods and guidance. On the other hand, there are instances where intellectual dominance by a single group – i.e., monopolies of knowledge – can suppress innovation and slow down development processes [45].

EtD frameworks largely focus on, and are optimal for, single-component interventions with simple linear pathways from intervention to outcomes. In addition, most EtD frameworks do not consider the context in which the intervention is delivered. With the exception of WHO-INTEGRATE, EtD frameworks do not explicitly or even implicitly incorporate a complexity perspective including the inter-relationship between the intervention and the context or environment in which it is delivered. Only the GCPS, the USPSTF, and WHO-INTEGRATE recommend the use of visual depictions of the relationships among the intervention components: such approaches facilitate examination of multiple key questions linked across the causal pathway for intervention effects, as well as the consideration of contextual factors which may be important for decision making.

Guidance on if and when to identify, synthesize and integrate evidence on criteria other than benefits and harms of the intervention into the decision-making process was provided only for WHO-INTEGRATE [38]. In our experience, most guideline development processes focus almost exclusively on benefits and harms in the evidence review and in discussions. Recently, however, guideline groups are examining qualitative evidence on acceptability, feasibility, and other decision criteria. Guideline groups continue to be uncertain as to how to incorporate EtD criteria such as equity, into recommendations in a meaningful fashion.

Ideally, EtD considerations are discussed early in the guideline development process, and strategic decisions are made as to the types of information and evidence that will be needed to inform key considerations. It may not be necessary, feasible, or even possible, to systematically examine evidence on all EtD considerations. The participation of a range of stakeholders, including persons and communities potentially affected by the intervention, is important both early in the planning phase and later when decisions are made. Their perspectives and experiences with issues such as acceptability and feasibility, for example, are critical for formulating credible and impactful recommendations or policies.

Evidence-to-decision criteria across frameworks

It was difficult to compare specific EtD criteria across frameworks due to significant variation in approaches to lumping and splitting criteria across frameworks, as well as in the terminology and definitions used. While most frameworks presented broad categories of criteria (e.g., “equity”), only WHO-INTEGRATE [38] provides detailed sub-criteria to facilitate understanding and decision making. Some main criteria included several constructs, which can complicate the decision-making process and reporting of the rationale for the recommendation or decision. For example, WHO-INTEGRATE’s “Human rights and sociocultural acceptability” includes both human rights and what other frameworks refer to as “preferences” or “acceptability”.

There is also variability in how criteria are defined and whether operational definitions or guidance are provided. The criterion “values and preferences” is particularly problematic and guideline developers continue to use the phrase “values and preferences” with unclear and variable meaning.

The GRADE criterion of “priority of the problem” is also problematic. It was initially included in the GRADE framework to facilitate prioritization across interventions, such as at the national or sub-national levels. However, in most guideline development scenarios, the problem on which interventions and comparators are focused has already been determined to be of high priority. Thus, examination of the burden of disease and other priority considerations is irrelevant at the stage of recommendation formulation.

The criteria related to resource use vary considerably, and may include cost, affordability, infrastructure needs, personnel training, and measures of economic efficiency (e.g., cost-benefit and cost-effectiveness). This variability is due in part to the different perspectives of the sponsors of the EtD framework or of the target audiences for the organization’s products. For example, the USPSTF is prohibited by national legislation from considering cost or cost-effectiveness in their recommendations. On the other hand, for the UK’s NICE, cost-effectiveness is a critical part of their decisions. There is also variability in the extent and specificity of the guidance on how to incorporate resource use into decision-making. While well-developed for NICE [32], and reasonably so for GRADE [46], guidance is almost completely lacking for WHO [37] and WHO-INTEGRATE [38].

The criterion of “equity” is poorly described and little operational guidance is provided across the frameworks except for WHO-INTEGRATE which provides several sub-criteria for consideration under the main criterion of “health equity, equality and non-discrimination” [38].

Group decision making

Group decision making when formulating recommendations in guidelines is rarely a simple, linear process. While EtD frameworks support the normative aspect of decision making in terms of how expert panels “should” or “ought to” develop recommendations by providing a structure for decision making [47,48,49], many factors affect how groups make decisions [47]. These include: i) situational or contextual factors (e.g., time pressure, social context, gender bias, political pressures); ii) individual characteristics of the decision maker (e.g., role on the expert panels, cultural and professional background, race/ethnicity, methodological expertise) [50]; and iii) individual panel member’s emotions and experiences (e.g., personal experiences with a disease). How these various factors contribute to decision making and recommendation formulation in guidelines is unknown [47], but their potential effects must be kept in mind and made explicit to the extent possible.

While acknowledging the complex process of formulating recommendations in guidelines, EtD frameworks are a valuable tool and have led to vast improvements over “free-for-all” meetings where decision making criteria were selected in an ad hoc manner, and it was often unclear which considerations actually contributed to final decisions and their relative weight, what evidence was examined, and how and why the final recommendation or decision was arrived at. EtD frameworks can be combined with any one of a number of approaches for achieving group consensus on the direction, strength, and wording of a recommendation or decision, including formal approaches such as the Delphi approach and the Nominal Group Technique, or voting if consensus cannot be reached.

Applying clinical and public health frameworks to environmental health

Applying approaches for clinical medicine interventions to other scientific fields, including environmental health, is challenging. There are important differences both in the assessment of the quality (certainty) of the body of evidence, and in translating evidence to recommendations or decisions. Woodruff and colleagues [16] note that the GRADE system and other evidence-based medicine approaches have limitations in terms of applicability to questions and decision making in environmental health. The reasons include: i) the need to combine human, animal and (sometimes) mechanistic evidence; ii) the paucity of randomized controlled trials (RCTs) and other types of experimental studies in humans due to ethical considerations; and iii) differences in the decision-making context (e.g., weighing the benefits and harms is different for clinical interventions than for unintended exposures to substances in the environment) [16]. Nonetheless, the constructs of the GRADE and WHO-INTEGRATE frameworks likely apply to environmental health interventions.

Limitations of this analysis

The approach taken for this review and analysis has a number of limitations. The methods for identifying existing frameworks were limited: the systematic review included only reviews and focused only on English language literature accessible through PubMed. Furthermore, many EtD frameworks are not published in the peer reviewed literature: rather, they are found on organizational web-sites and/or in organizational procedure manuals. For the review of key organizations, only a convenience sample was examined and EtD frameworks with important additional considerations or novel approaches may have been overlooked. However, it is unlikely that this approach missed key or significantly different frameworks in view of the dominance of GRADE and our consultation with experts on EtD frameworks.

Our analysis also has limitations related to the nature of the available data. Published information on the methods for developing frameworks and for operationalizing them was generally sparse, except for GRADE and WHO-INTEGRATE. For some frameworks it was difficult to identify exactly which criteria were routinely included in the decision-making process. While the publication might present a list or table of criteria, additional criteria might be mentioned in the adjacent text. However, the degree to which the guideline group or other decision maker addressed these additional criteria was often unclear.

Conclusions and next steps

EtD frameworks are an extremely useful tool for recommendation formulation and decision making in healthcare and other scientific fields. They enable decisions to be made based on research evidence and with explicit consideration of a range of constructs, and they facilitate clear articulation of the rationale for decisions.

The principles underpinning evidence-informed, transparent and impactful decision making are the same across scientific fields, including environmental health. The GRADE EtD framework, along with WHO-INTEGRATE with its focus on contextual issues and inter-relationships, provide a useful starting point for consideration for decision-making for interventions in environmental health. Significant modifications will be needed, however, given the nature of the evidence base and the complex context in which environmental health interventions are designed, implemented, regulated and evaluated. The process for developing an EtD framework for environmental health interventions requires a broad range of experts, including not only environmental health scientists, but evidence synthesis and guideline methodologists, public health generalists, human rights and ethics experts, social scientists, and economists, among others. An iterative development process will be needed: a draft framework should be pilot-tested, revised and evaluated. The framework’s utility and impact on decision making, and the quality and impact of the resultant recommendations need careful evaluation, with the results used to develop future iterations.

Appendix

Annex 1. Search strategy and inclusion criteria

((“Policy Making”[Mesh]) OR (“Decision Making”[Mesh])) AND (((framework[Title/Abstract]) OR (frameworks[Title/Abstract])) OR ((template[Title/Abstract]) OR (templates[Title/Abstract]) OR (tool[Title/Abstract]) OR (tools[Title/Abstract]))).

Filters: English language, systematic review, last 10 years.

The search was executed 7 January 2021 in the PubMed search engine for Medline.

Annex 2. PRISMA flow diagram

Annex 3. Results of the search: full-text review

The search of PubMed performed on 7 Jan 2021. 399 citations were identified, 25 of which were reviewed in full text: these are listed below with the reason for exclusion.

Abbreviations: E, excluded citation; E2D, evidence-to-decision; I, included citation; LMICs, low- and middle-income countries; NA (not applicable because the publication fulfilled inclusion criteria)

Citation	Title	Include/exclude	Rationale for exclusion
Baltussen 2019 [1]	Multicriteria decision analysis to support health technology assessment agencies: Benefits, limitations, and the way forward	I	NA
Boivin 2018 [2]	Patient and public engagement in research and health system decision making: A systematic review of evaluation tools	E	Review of tools for evaluating patient and public engagement, not for decision making
Burchett 2012 [3]	National decision-making on adopting new vaccines: a systematic review	I	NA
Carroll 2013 [4]	Best fit framework synthesis: refining the method	E	Describes “best fit” method for qualitative synthesis; no EtD frameworks
Carvalho 2018 [5]	Capturing budget impact considerations within economic evaluations: A systematic review of economic evaluations of rotavirus vaccine in low- and middle-income countries and a proposed assessment framework	E	Checklist for budget impact analyses
Clark 2014 [6]	Discrete choice experiments in health economics: A review of the literature	E	Review of discrete chose experiments; no examination of EtD frameworks
Clark 2018 [7]	Measuring trade-offs in nephrology: A systematic review of discrete choice experiments and conjoint analysis studies	E	Review of discrete choice experiments in nephrology
Clarke 2016 [8]	The application of theories of the policy process to obesity prevention: a systematic review and meta-synthesis	E	Review of studies where theory underpins the policy development process or adoption of policies. Focus is on explaining the policies and not on what factors should be considered in decision-making.
Clifford 2017 [9]	What information is used in treatment decision aids? A systematic review of the types of evidence populating health decision aids	E	Focus on patient decision aids
Dodd 2019 [10]	Investigating the process of evidence-informed health policymaking in Bangladesh: a systematic review	E	Reviews factors linked to policy uptake or prioritization and how and why evidence is used; presents explanatory and descriptive information and not frameworks for normative decisions
Gentil 2015 [11]	A systematic review of socio-economic assessments in support of coastal zone management (1992–2011)	E	Describes concepts used for decision-making; presents descriptive information and not normative frameworks
Kolasa 2015 [12]	Pricing and reimbursement frameworks in Central Eastern Europe: a decision tool to support choices	E	Review revealed 33 pricing and reimbursement schemes; led to framework with two economic questions. Exclude as exclusively economic focus.
Kristensen 2019 [13]	Identifying the need for good practices in health technology assessment: Summary of the ISPOR HTA Council Working Group Report on Good Practices in HTA	E	Neither this paper or the full report (https://www.ispor.org/member-groups/councils-roundtables/health-technology-assessment-council) provides a review of EtD frameworks (referred to as “contextualization”).
Lam 2018 [14]	Decision support tools for regenerative medicine: Systematic review	E	Examines decision support tools for product development and manufacturing of cell and gene therapies in regenerative medicine; does not focus on clinical medicine, public health or environmental health
LaRocca 2012 [15]	The effectiveness of knowledge translation strategies used in public health: A systematic review	E	Review of evaluations of knowledge translation strategies and not the strategies themselves
Laurans 2013 [16]	Use of ecosystem services economic valuation for decision making: Questioning a literature blindspot	E	Focuses on the types and uses of Ecosystem Services economic Valuation (ESV), not on the criteria for decision making
Liverani 2013 [17]	Political and institutional influences on the use of evidence in public health policy. A systematic review	E	Examines factors that impact the use of evidence in policy-making
Mayer 2017 [18]	Costing evidence for health care decision making in Austria: A systematic review	E	Examines economic analyses and costing methods used in Austria; does not present EtD frameworks
Mustafa 2017 [19]	Decision making about healthcare-related tests and diagnostic test strategies. Paper 3: a systematic review shows limitations in most tools designed to assess quality and develop recommendations	I	NA
Sullivan 2015 [42]	What guidance are economists given on how to present economic evaluations for policymakers? A systematic review	E	Review of reporting guidance for economic analyses; does not deal directly with EtD concepts
Trapero-Bertram 2019 [33]	What attributes should be included in a discrete choice experiment related to health technologies? A systematic literature review	E	Review of attributes for discrete choice experiments; not a review of EtD frameworks
Tsoi 2013 [35]	Harmonization of reimbursement and regulatory approval processes: a systematic review of international experiences	E	Review of approaches to harmonize reimbursement and regulatory approval processes
Tsoi 2015 [39]	Systematic narrative review of decision frameworks to select the appropriate modelling approaches for health economic evaluations	E	Examines frameworks for selecting economic modelling approaches
Votruba 2018 [25]	A systematic review of frameworks for the interrelationships of mental health evidence and policy in low- and middle-income countries	E	Review of theories and frameworks explaining use of evidence to formulate policies in mental health in LMICs; does not review frameworks for decision-making
Wickremasinghe 2016 [31]	District decision-making for health in low-income settings: A systematic literature review	I	NA
From other sources/ad hoc
Domrumel 2020 [37]	Organizational aspect in healthcare decision-making: a literature review	E	Systematic review of individual criteria for decision-making (not a review of frameworks or sets of criteria)
Morgan 2018 [38]	Decision-making frameworks and considerations for informing coverage decisions for healthcare interventions: a critical interpretive synthesis.	I	NA

References for the full-text review

1. Baltussen R, Marsh K, Thokala P, et al. Multicriteria Decision Analysis to Support Health Technology Assessment Agencies: Benefits, Limitations, and the Way Forward. Value Health. 2019;22(11):1283–1288.

2. Boivin A, L’Espérance A, Gauvin FP, et al. Patient and public engagement in research and health system decision making: A systematic review of evaluation tools. Health Expect. 2018;21(6):1075–1084.

3. Burchett HE, Mounier-Jack S, Griffiths UK, Mills AJ. National decision-making on adopting new vaccines: a systematic review. Health Policy Plan. 2012;27 Suppl 2:ii62–76.

4. Carroll C, Booth A, Leaviss J, Rick J. “Best fit” framework synthesis: refining the method. BMC Med Res Methodol. 2013;13:37.

5. Carvalho N, Jit M, Cox S, Yoong J, Hutubessy RCW. Capturing Budget Impact Considerations Within Economic Evaluations: A Systematic Review of Economic Evaluations of Rotavirus Vaccine in Low- and Middle-Income Countries and a Proposed Assessment Framework. Pharmacoeconomics. 2018;36(1):79–90.

6. Clark MD, Determann D, Petrou S, Moro D, de Bekker-Grob EW. Discrete choice experiments in health economics: a review of the literature. Pharmacoeconomics. 2014;32(9):883–902.

7. Clark MD, Szczepura A, Gumber A, Howard K, Moro D, Morton RL. Measuring trade-offs in nephrology: a systematic review of discrete choice experiments and conjoint analysis studies. Nephrol Dial Transplant. 2018;33(2):348–355.

8. Clarke B, Swinburn B, Sacks G. The application of theories of the policy process to obesity prevention: a systematic review and meta-synthesis. BMC Public Health. 2016;16(1):1084.

9. Clifford AM, Ryan J, Walsh C, McCurtin A. What information is used in treatment decision aids? A systematic review of the types of evidence populating health decision aids. BMC Med Inform Decis Mak. 2017;17(1):22.

10. Dodd M, Ivers R, Zwi AB, Rahman A, Jagnoor J. Investigating the process of evidence-informed health policymaking in Bangladesh: a systematic review. Health Policy Plan. 2019;34(6):469–478.

11. Le Gentil E, Mongruel R. A systematic review of socio-economic assessments in support of coastal zone management (1992–2011). J Environ Manage. 2015;149:85–96.

12. Kolasa K, Kalo Z, Hornby E. Pricing and reimbursement frameworks in Central Eastern Europe: a decision tool to support choices. Expert Rev Pharmacoecon Outcomes Res. 2015;15(1):145–155.

13. Kristensen FB, Husereau D, Huić M, et al. Identifying the Need for Good Practices in Health Technology Assessment: Summary of the ISPOR HTA Council Working Group Report on Good Practices in HTA. Value Health. 2019;22(1):13–20.

14. Lam C, Meinert E, Alturkistani A, et al. Decision Support Tools for Regenerative Medicine: Systematic Review. J Med Internet Res. 2018;20(12):e12448.

15. LaRocca R, Yost J, Dobbins M, Ciliska D, Butt M. The effectiveness of knowledge translation strategies used in public health: a systematic review. BMC Public Health. 2012;12:751.

16. Laurans Y, Rankovic A, Billé R, Pirard R, Mermet L. Use of ecosystem services economic valuation for decision making: questioning a literature blindspot. J Environ Manage. 2013;119:208–219.

17. Liverani M, Hawkins B, Parkhurst JO. Political and institutional influences on the use of evidence in public health policy. A systematic review. PLoS One. 2013;8(10):e77404.

18. Mayer S, Kiss N, Łaszewska A, Simon J. Costing evidence for health care decision-making in Austria: A systematic review. PLoS One. 2017;12(8):e0183116.

19. Mustafa RA, Wiercioch W, Falavigna M, et al. Decision making about healthcare-related tests and diagnostic test strategies. Paper 3: a systematic review shows limitations in most tools designed to assess quality and develop recommendations. J Clin Epi. 2017;92:29–37.

20. Sullivan SM, Wells G, Coyle D. What Guidance are Economists Given on How to Present Economic Evaluations for Policymakers? A Systematic Review. Value Health. 2015;18(6):915–924.

21. Trapero-Bertran M, Rodríguez-Martín B, López-Bastida J. What attributes should be included in a discrete choice experiment related to health technologies? A systematic literature review. PloS one. 2019;14(7):e0219905.

22. Tsoi B, Masucci L, Campbell K, Drummond M, O’Reilly D, Goeree R. Harmonization of reimbursement and regulatory approval processes: a systematic review of international experiences. Expert Rev Pharmacoecon Outcomes Res. 2013;13(4):497–511.

23. Tsoi B, O’Reilly D, Jegathisawaran J, Tarride JE, Blackhouse G, Goeree R. Systematic narrative review of decision frameworks to select the appropriate modelling approaches for health economic evaluations. BMC Res Notes. 2015;8:244.

24. Votruba N, Ziemann A, Grant J, Thornicroft G. A systematic review of frameworks for the interrelationships of mental health evidence and policy in low- and middle-income countries. Health Res Policy Syst. 2018;16(1):85.

25. Wickremasinghe D, Hashmi IE, Schellenberg J, Avan BI. District decision-making for health in low-income settings: a systematic literature review. Health Policy and Planning. 2016;31(suppl_2):ii12-ii24.

26. Dubromel A, Duvinage-Vonesch MA, Geffroy L, Dussart C. Organizational aspect in healthcare decision-making: a literature review. J Mark Access Health Policy. 2020;8(1):1810905.

27. Morgan RL, Kelley L, Guyatt GH, Johnson A, Lavis JN. Decision-making frameworks and considerations for informing coverage decisions for healthcare interventions: a critical interpretive synthesis. J Clin Epi. 2018;94:143–150.