- Open Access
Evidence-to-decision frameworks: a review and analysis to inform decision-making for environmental health interventions
Environmental Health volume 20, Article number: 124 (2021)
Evidence-to-decision (EtD) frameworks provide a structured and transparent approach for groups of experts to use when formulating recommendations or making decisions. While extensively used for clinical and public health recommendations, EtD frameworks are not in widespread use in environmental health. This review sought to identify, compare and contrast key EtD frameworks for decisions on interventions used in clinical medicine, public health or environmental health. This information can be used to develop an EtD framework suitable for formulating recommendations for interventions in environmental health.
We identified a convenience sample of EtD frameworks used by a range of organizations. We searched Medline for systematic reviews of frameworks. We summarized the decision criteria in the selected frameworks and reviews in a qualitative manner.
Fourteen organizations provided 18 EtD frameworks; most frameworks focused on clinical medicine or public health interventions; four focused on environmental health and three on economic considerations. Harms of interventions were examined in all frameworks and benefits in all but one. Other criteria included certainty of the body of evidence (15 frameworks), resource considerations (15), feasibility (13), equity (12), values (11), acceptability (11), and human rights (2). There was variation in how specific criteria were defined. The five identified systematic reviews reported a similar spectrum of EtD criteria.
The EtD frameworks examined encompassed similar criteria, with tailoring to specific audience needs. Existing frameworks are a useful starting point for development of one tailored to decision-making in environmental health.
It is widely recognized that environmental pollution is an important determinant of health and that interventions, in particular policy recommendations, are a key means by which population health can be improved. The formulation of trustworthy and impactful recommendations and policies on environmental health interventions is a complex task, which requires the identification of all relevant data and evidence, their critical appraisal and synthesis and translation into a recommendation or policy. The processes and methods for hazard identification and risk assessment of environmental substances based on systematic reviews of the evidence have advanced considerably in the last decade [1,2,3,4]. These approaches have been largely based on methods developed for clinical medicine , which have been expanded to encompass public health interventions , diagnostic test accuracy and impact , coverage decisions [8, 9], and health technology assessments (HTA) , among others.
The final step in the process of formulating evidence-informed recommendations and policies involves the translation of data and evidence on various decision factors into an explicit recommendation or policy. Evidence-to-decision (EtD) frameworks provide a structured and transparent approach for groups of technical experts or policy-makers to accomplish this step [6, 11]. These frameworks include explicit criteria which the group considers individually and in aggregate focused on the relative benefits and harms, as well as other considerations. Such frameworks can facilitate: i) consideration of all relevant criteria in the decision-making process; ii) examination of the pros and cons of each intervention option; iii) presentation of relevant evidence for each criterion; iv) identification of the reasons for any disagreement within the expert group; v) transparent reporting of the decision-making process; and vi) crafting of the rationale statement for each recommendation. Populated EtD frameworks can also facilitate implementation by assisting the end-user in understanding how and why specific recommendations were made, and by providing data and evidence on each decision criterion which may facilitate local adoption or adaptation .
In environmental health, once hazards are identified and risks assessed, organizations may want to examine mitigating and prevention interventions and make recommendations and policies based on systematic reviews of research evidence and other data and information. An EtD framework suitable for environmental health interventions will facilitate this process. However, such frameworks have not been widely used in this field.
The objectives of this review were to identify, compare and contrast key EtD frameworks for interventions in clinical medicine, public health and environmental health and to summarize the main decision criteria across these frameworks. The identification of these criteria will inform the development of an EtD framework for use in formulating recommendations regarding mitigating and prevention interventions related to exposure to harmful substances in the environment.
In order to identify relevant, existing EtD frameworks, we took two approaches: a search of the peer-reviewed literature for systematic reviews of EtD frameworks, and identification of frameworks used by a range of organizations which make decisions or formulate recommendations in clinical medicine, or public or environmental health.
The focus of this paper is on the substantive criteria for decision-making with respect to interventions, including both normative criteria (what should be done) and feasibility criteria (what can be done) . Criteria related to the process of decision-making are beyond the scope of this work.
There are an increasing number of organizations which use systematic and transparent approaches to synthesize evidence in environmental health for hazard identification , risk assessment , and for the synthesis of the benefits and harms of interventions . These organizations were not included in the current analysis because they do not make recommendations on interventions to mitigate the effects of harmful exposures.
Bibliographic database search and article screening
To identify relevant EtD frameworks, we searched Medline via PubMed for systematic reviews of EtD frameworks, templates or tools published in English and indexed in the 10-year period up to 7 January 2021. The complete search strategy is found in Annex 1. We also solicited the advice of guideline development experts for any additional potential citations. Publications were included if they were systematic reviews of frameworks, tools or templates for formulating decisions or recommendations, or for priority setting related to the use of interventions (including diagnostic tests) in clinical medicine, public health, or environmental health. The included frameworks focused on population-level interventions; decision tools for the provider-patient interaction and patient decision aids were excluded. Frameworks which focused exclusively on economic considerations were also excluded. The setting for both decision-making and implementation of recommendations was not restricted.
Two persons (SLN and MTA) screened the titles and abstracts, and full-text versions were retrieved for studies which potentially fulfilled inclusion criteria. Consensus was achieved between the two reviewers for final inclusion in the review.
Search for EtD frameworks used by key organizations
In addition to the systematic review described above, we examined a broad range of organizations including clinical and public health guideline development groups in academia and the private sector, healthcare provider professional organizations, governmental agencies and international organizations. Given the purpose of this review, we focused particularly on organizations that work in the field of environmental health. Organizations were identified based on author knowledge of prominent guidelines in public health and clinical medicine, by snowballing (reviewing the origin of or basis for identified frameworks), and by conferring with a broad international network of guideline developers based on our experiences and contacts.
We used this pragmatic approach for several reasons. First, because of the dominance of the GRADE (Grading of Recommendations Assessment, Development and Evaluation) EtD framework over the last 15 years [5, 11, 17], many organizations either use GRADE or a modification thereof. Thus, a more exhaustive search was unlikely to yield novel or unique frameworks. Second, because the methods used by guideline development organizations are infrequently published in the peer-reviewed literature, an extensive hand search of the grey literature including organizational web-sites would have been required, which was infeasible.
Data extraction and synthesis
For each systematic review identified in our search, a single author (SLN) extracted key information describing the focus of the review, time period searched, and the main findings, including an overview of the criteria identified across the included frameworks. We did not extract data on each individual framework included in each review.
For the EtD frameworks used by key organizations, we extracted data on organizational characteristics, the methods used to develop the framework, the funders and any declared interests of the developers of the framework, how evidence should be used to inform each criterion in the framework, quality assessment of individual studies and of the body of evidence, the specific EtD criteria, and the types of conclusions or recommendations formulated. One author extracted these data (SLN) and a second checked them (MTA); disagreements were discussed and consensus reached. Data were extracted into a template in Excel (Microsoft Corporation, Redmond WA, USA).
For the key organizations, the primary decision criteria for each EtD framework were extracted from the main published reports, and specifically from the identified template or list of criteria. Criteria which are mentioned only in the narrative text accompanying a framework were not extracted unless the text suggested that they were consistently applied in decision making.
Data were summarized in a narrative, qualitative manner. Given that the included frameworks were a convenience sample, descriptive statistics and statistical comparisons are not meaningful. Quality assessment of the frameworks was not performed as there is no standard for such an assessment.
Role of the funding source
The study funder (JPB Foundation) had no role in the study design; data collection, analysis or interpretation; in writing the report; or in the decision to publish.
Identified systematic reviews of EtD frameworks
The bibliographic database search for systematic reviews of EtD frameworks yielded 399 citations, of which four fulfilled inclusion criteria [7, 10, 18, 19] (Table 1, Annex 2 and 3). One additional study which fulfilled inclusion criteria was identified by the co-authors . Each of these reviews included a cohort of EtD frameworks which the authors had systematically identified, with a particular focus: multicriteria decision analysis in HTA , vaccine adoption into national programs , diagnostic tests , decision making in local low-income settings , and frameworks for informing health system coverage decisions . Each review summarized the main EtD criteria identified across their included, individual frameworks, and these main criteria are presented in Table 1. All five reviews included the balance of benefits and harms and consideration of resource use or cost-effectiveness. Only the review by Mustafa and colleagues  included an assessment of certainty of evidence. Considerations of equity, acceptability and feasibility were included in only two reviews [10, 18]. The EtD criteria outlined in these five reviews generally corresponded to the main criteria in the EtD frameworks of the selected key organizations (see Results section 2).
EtD frameworks used by key organizations
Fourteen organizations that use EtD frameworks for recommendation formulation or decision-making were examined in detail (Table 2). One organization, the GRADE Working Group, is an informal network of individuals from a broad range of organizations, including academic institutions, national and international guideline development agencies, and healthcare provider organizations, among others . The GRADE Working Group does not, itself, publish guidelines, but rather develops processes and methods for use by other organizations which develop guidelines. The other 13 organizations develop guidelines for specific audiences and with a clearly defined scope or set of topics: five are agencies of national governments [25, 31, 33, 35, 39], two are related to the World Health Organization (WHO) [37, 38], one is a U.S. State agency (California Environmental Protection Agency (CalEPA) ) and the remainder are non-governmental organizations or academic groups [2, 21, 24, 27, 28].
Two organizations focus primarily on clinical care: the Scottish Intercollegiate Guideline Network (SIGN)  and the U.S. Preventive Service Task Force (USPSTF) . The Guide to Community Preventive Services (GCPS) focuses on interventions aimed at groups, communities or health systems  and WHO and WHO-INTEGRATE primarily on public health interventions [37, 38]. Three organizations are oriented to health technologies, with a prominent focus on economic evaluations and resource considerations [24, 27, 28]. The UK National Institutes for Health and Care Excellence (NICE) examines a broad range of clinical, public health, and social interventions . The Advisory Committee on Immunization Practices (ACIP) focuses exclusively on vaccine recommendations for U.S. populations . Breast Cancer Prevention Partners (BCPP) , CalEPA [22, 23], and the Navigation Guide [2, 16, 40] focus on the human health effects of hazardous substances in the environment.
Development process and expertise
For the organizations that developed de novo EtD frameworks and described the process for developing them, all used an iterative approach based on an examination of other organizations, with input from experts in guideline methods and evidence synthesis (Table 2). The most comprehensive approach was taken by Rehfuess and colleagues  in development of the WHO-INTEGRATE framework. Their approach included development of a theoretical framework, a review of WHO basic documents, a systematic review of EtD criteria, input from a range of stakeholders, and a thematic analysis to identify key domains .
It was difficult to discern the expertise of contributors to framework development; most groups appeared to consist mainly of academic, generalist guideline methodologists with either clinical or public health experience. Social scientists led work on WHO-INTEGRATE  and contributed to the GRADE public health framework . Eight of the 14 organizations reported who funded the development of the EtD framework [2, 11, 21, 24, 27, 34, 37, 38] and five reported declarations of interest among framework developers [2, 11, 20, 24, 38]; the remaining organizations did not provide this information.
Eighteen frameworks were examined in detail: one from each of the 14 organizations, except for CalEPA and GRADE, where two [22, 23] and four [6, 9, 11, 17] unique frameworks were examined, respectively. There were significant commonalities across frameworks in the criteria for formulating recommendations (Table 3 and Fig. 1). Unsurprisingly, all included consideration of benefits, except the CalEPA framework for pesticides  which examines risks (of environmental exposures) and not benefits. All frameworks included an assessment of harms of the intervention under consideration. Fifteen of the 18 frameworks included some assessment of certainty or quality of the body of evidence across outcomes in the decision-making process: BCPP, one of the CalEPA frameworks, and ISPOR (International Society for Pharmacoeconomics and Outcomes Research) did not [21, 22, 28]. Some measure of costs, resource use or cost-effectiveness was found in all frameworks except BCPP, USPSTF and the GCPS [21, 26, 36]. Other decision criteria were variably included: feasibility (13 frameworks), equity (12), values (11), and acceptability (11). Only two frameworks included human rights: WHO  and WHO-INTEGRATE frameworks .
“Priority of the problem” was explicitly included in the EtD frameworks based on GRADE [20, 34, 37], as well as the BCPP , EVIDEM (Evidence and Values Impact on DEcision Making) , ISPOR [28,29,30] and Navigation Guide frameworks . However, in the background materials for several other frameworks, the burden of the disease was mentioned as an important consideration when prioritizing an intervention for development of a guideline or policy.
The meaning of the criterion “values and preferences” varied over time and across publications. This phrase encompasses two different constructs: the relative value that persons affected by the recommendations place on the outcomes of the intervention and the preferences such persons have regarding the intervention options. The GRADE Working Group initially combined these two constructs  with a definition encompassing both . The 2016 update of the GRADE EtD framework introduced two separate criteria: variability of the value affected persons place on the main outcomes, and acceptability of the intervention  to various stakeholders. Other key organizations include a decision criterion related to values using verbiage closely aligned with GRADE  and focused exclusively on the relative value of outcomes [32, 34], or they include the concept but with a somewhat different meaning [20, 22, 28].
Evidence used to inform each criterion when organizations make decisions
All of the frameworks report that the EtD criteria should be informed by evidence obtained from a variety of sources, with an emphasis on systematic reviews of research evidence (Table 2). Twelve of the organizations recommend an assessment of the quality (validity or certainty) of the body of evidence for important outcomes as part of the evidence review, and most recommend the GRADE system with or without modifications. Only BCPP  and one of the CalEPA guidelines , do not recommend such an assessment. The approaches used by the GCPS  and USPSTF  differ somewhat from GRADE in their assessment of the “strength of evidence” and “certainty of net benefit”, respectively. The EtD frameworks with an economic focus describe varied approaches to assessing quality of the body of evidence [24, 27, 28].
Nomenclature for recommendations
Nine of the included organizations provide specific guidance on the nomenclature for the types of recommendations formulated based on the EtD criteria (Table 2). The most common categorization of recommendations among those nine organizations was two levels of strength both for and against a recommendation, i.e., four categories [11, 34, 37]. GRADE has extensive documents on this issue and uses the terms “strong” and “weak or conditional “. The USPSTF has five categories of recommendations , while ACIP  and the GCPS  have three. The economic-focused frameworks refer to “value” in various ways [24, 27, 30]. NICE uses the wording of recommendations to reflect the strength of the evidence (e.g., offer, advise, consider) rather than standardized terms to represent the strength of the recommendation .
Five organizations provide explicit guidance on the situations where recommendations cannot be formulated due to insufficient evidence or a close balance between benefits and harms (Table 2) [11, 20, 26, 32, 36]. The USPSTF provides the most detailed guidance on this situation . Ten of the organizations suggest including knowledge or research gaps with the recommendations, with particular emphasis in ACIP, NICE, GCPS and the USPSTF.
Specific evidence-to-decision frameworks
GRADE includes EtD frameworks for four different purposes: i) clinical recommendations, individual perspective; ii) clinical recommendations, population perspective; iii) coverage decisions; and iv) health system and public health recommendations/decisions. (There is a fifth GRADE EtD framework for diagnostic, screening and other tests but this was not included in this review of interventions considered most relevant for application to environmental health .) These frameworks are all very similar, all with 12 criteria covering the same concepts, with some variation in verbiage, tailored to the different audiences (Table 3). There is more emphasis on resource considerations, equity, acceptability and feasibility for health systems and public health decisions than for individual patient clinical recommendations [6, 9].
ACIP’s EtD framework is derived directly from GRADE , while NICE  and SIGN’s  EtD criteria closely resemble those of GRADE. WHO uses the GRADE EtD framework which was current when their guidance was published in 2014 .
The WHO-INTEGRATE EtD framework version 1.0 [38, 41], first published in 2019 (Table 2), was developed in response to a perceived need to take a complexity perspective into account, to develop a conceptual framework to underpin the EtD, and to incorporate public health and WHO-specific values when developing WHO guidelines. WHO-INTEGRATE includes six broad criteria  (Table 3), with quality of evidence applied to all criteria. Each criterion includes sub-criteria: for example, the main criterion “Health equity, equality and non-discrimination”  includes impact on health equality and/or health equity, distribution of benefits and harms, affordability, and accessibility.
The frameworks that focus on interventions in environmental health were generally less well developed and lacked specificity compared with the clinical and public health frameworks [2, 21, 22] with the exception of a recently published guide to alternatives analysis for chemicals in consumer products .
Additional details on each of the included frameworks are provided in Additional file 1.
A number of EtD frameworks have been developed in a variety of clinical, and public and environmental health disciplines, and there is significant overlap in the criteria used to inform recommendations or decisions across these frameworks. Benefits and harms are almost universally included, and an assessment of certainty or quality of the body of evidence and some measure of resource use were included in most frameworks examined. Other decision criteria such as values, equity, feasibility, and acceptability were variably included, while only two frameworks encompassed human rights. There was variation across frameworks in terminology, definitions and presentation of EtD criteria. The five systematic reviews of EtD frameworks describe decision criteria that were similar to those presented in the frameworks from key organizations.
The 18 frameworks identified for 14 key organizations focused mainly on clinical medicine or public health interventions. The GRADE framework is the most well developed, with extensive information and guidance on its methods. A number of the key organizations examined have adopted GRADE, often with minor modifications [16, 20, 32, 34, 37]. Three of the four frameworks related to environmental health lacked detail on the specific criteria for decision-making [2, 21, 22]. Frameworks originating in the HTA realm unsurprisingly focus on economic considerations [24, 27, 28], while also including criteria similar to those encompassed by other frameworks.
General aspects of evidence-to-decision frameworks
The process and methods for developing EtD frameworks were often superficial and poorly reported. Only WHO-INTEGRATE  and the GRADE coverage  frameworks were developed following a systematic review of existing frameworks or potential decision criteria. Only WHO-INTEGRATE describes a conceptual framework  underpinning the EtD criteria. The lack of a conceptual framework represents a significant weakness in most frameworks: key considerations may be missed and long-standing criteria may be perpetuated without adequate scrutiny.
GRADE has dominated guideline methods in health care and public health over the last 15 years. This has led to productive collaborations, standardization of processes and methods, and the development of detailed methods and guidance. On the other hand, there are instances where intellectual dominance by a single group – i.e., monopolies of knowledge – can suppress innovation and slow down development processes .
EtD frameworks largely focus on, and are optimal for, single-component interventions with simple linear pathways from intervention to outcomes. In addition, most EtD frameworks do not consider the context in which the intervention is delivered. With the exception of WHO-INTEGRATE, EtD frameworks do not explicitly or even implicitly incorporate a complexity perspective including the inter-relationship between the intervention and the context or environment in which it is delivered. Only the GCPS, the USPSTF, and WHO-INTEGRATE recommend the use of visual depictions of the relationships among the intervention components: such approaches facilitate examination of multiple key questions linked across the causal pathway for intervention effects, as well as the consideration of contextual factors which may be important for decision making.
Guidance on if and when to identify, synthesize and integrate evidence on criteria other than benefits and harms of the intervention into the decision-making process was provided only for WHO-INTEGRATE . In our experience, most guideline development processes focus almost exclusively on benefits and harms in the evidence review and in discussions. Recently, however, guideline groups are examining qualitative evidence on acceptability, feasibility, and other decision criteria. Guideline groups continue to be uncertain as to how to incorporate EtD criteria such as equity, into recommendations in a meaningful fashion.
Ideally, EtD considerations are discussed early in the guideline development process, and strategic decisions are made as to the types of information and evidence that will be needed to inform key considerations. It may not be necessary, feasible, or even possible, to systematically examine evidence on all EtD considerations. The participation of a range of stakeholders, including persons and communities potentially affected by the intervention, is important both early in the planning phase and later when decisions are made. Their perspectives and experiences with issues such as acceptability and feasibility, for example, are critical for formulating credible and impactful recommendations or policies.
Evidence-to-decision criteria across frameworks
It was difficult to compare specific EtD criteria across frameworks due to significant variation in approaches to lumping and splitting criteria across frameworks, as well as in the terminology and definitions used. While most frameworks presented broad categories of criteria (e.g., “equity”), only WHO-INTEGRATE  provides detailed sub-criteria to facilitate understanding and decision making. Some main criteria included several constructs, which can complicate the decision-making process and reporting of the rationale for the recommendation or decision. For example, WHO-INTEGRATE’s “Human rights and sociocultural acceptability” includes both human rights and what other frameworks refer to as “preferences” or “acceptability”.
There is also variability in how criteria are defined and whether operational definitions or guidance are provided. The criterion “values and preferences” is particularly problematic and guideline developers continue to use the phrase “values and preferences” with unclear and variable meaning.
The GRADE criterion of “priority of the problem” is also problematic. It was initially included in the GRADE framework to facilitate prioritization across interventions, such as at the national or sub-national levels. However, in most guideline development scenarios, the problem on which interventions and comparators are focused has already been determined to be of high priority. Thus, examination of the burden of disease and other priority considerations is irrelevant at the stage of recommendation formulation.
The criteria related to resource use vary considerably, and may include cost, affordability, infrastructure needs, personnel training, and measures of economic efficiency (e.g., cost-benefit and cost-effectiveness). This variability is due in part to the different perspectives of the sponsors of the EtD framework or of the target audiences for the organization’s products. For example, the USPSTF is prohibited by national legislation from considering cost or cost-effectiveness in their recommendations. On the other hand, for the UK’s NICE, cost-effectiveness is a critical part of their decisions. There is also variability in the extent and specificity of the guidance on how to incorporate resource use into decision-making. While well-developed for NICE , and reasonably so for GRADE , guidance is almost completely lacking for WHO  and WHO-INTEGRATE .
The criterion of “equity” is poorly described and little operational guidance is provided across the frameworks except for WHO-INTEGRATE which provides several sub-criteria for consideration under the main criterion of “health equity, equality and non-discrimination” .
Group decision making
Group decision making when formulating recommendations in guidelines is rarely a simple, linear process. While EtD frameworks support the normative aspect of decision making in terms of how expert panels “should” or “ought to” develop recommendations by providing a structure for decision making [47,48,49], many factors affect how groups make decisions . These include: i) situational or contextual factors (e.g., time pressure, social context, gender bias, political pressures); ii) individual characteristics of the decision maker (e.g., role on the expert panels, cultural and professional background, race/ethnicity, methodological expertise) ; and iii) individual panel member’s emotions and experiences (e.g., personal experiences with a disease). How these various factors contribute to decision making and recommendation formulation in guidelines is unknown , but their potential effects must be kept in mind and made explicit to the extent possible.
While acknowledging the complex process of formulating recommendations in guidelines, EtD frameworks are a valuable tool and have led to vast improvements over “free-for-all” meetings where decision making criteria were selected in an ad hoc manner, and it was often unclear which considerations actually contributed to final decisions and their relative weight, what evidence was examined, and how and why the final recommendation or decision was arrived at. EtD frameworks can be combined with any one of a number of approaches for achieving group consensus on the direction, strength, and wording of a recommendation or decision, including formal approaches such as the Delphi approach and the Nominal Group Technique, or voting if consensus cannot be reached.
Applying clinical and public health frameworks to environmental health
Applying approaches for clinical medicine interventions to other scientific fields, including environmental health, is challenging. There are important differences both in the assessment of the quality (certainty) of the body of evidence, and in translating evidence to recommendations or decisions. Woodruff and colleagues  note that the GRADE system and other evidence-based medicine approaches have limitations in terms of applicability to questions and decision making in environmental health. The reasons include: i) the need to combine human, animal and (sometimes) mechanistic evidence; ii) the paucity of randomized controlled trials (RCTs) and other types of experimental studies in humans due to ethical considerations; and iii) differences in the decision-making context (e.g., weighing the benefits and harms is different for clinical interventions than for unintended exposures to substances in the environment) . Nonetheless, the constructs of the GRADE and WHO-INTEGRATE frameworks likely apply to environmental health interventions.
Limitations of this analysis
The approach taken for this review and analysis has a number of limitations. The methods for identifying existing frameworks were limited: the systematic review included only reviews and focused only on English language literature accessible through PubMed. Furthermore, many EtD frameworks are not published in the peer reviewed literature: rather, they are found on organizational web-sites and/or in organizational procedure manuals. For the review of key organizations, only a convenience sample was examined and EtD frameworks with important additional considerations or novel approaches may have been overlooked. However, it is unlikely that this approach missed key or significantly different frameworks in view of the dominance of GRADE and our consultation with experts on EtD frameworks.
Our analysis also has limitations related to the nature of the available data. Published information on the methods for developing frameworks and for operationalizing them was generally sparse, except for GRADE and WHO-INTEGRATE. For some frameworks it was difficult to identify exactly which criteria were routinely included in the decision-making process. While the publication might present a list or table of criteria, additional criteria might be mentioned in the adjacent text. However, the degree to which the guideline group or other decision maker addressed these additional criteria was often unclear.
Conclusions and next steps
EtD frameworks are an extremely useful tool for recommendation formulation and decision making in healthcare and other scientific fields. They enable decisions to be made based on research evidence and with explicit consideration of a range of constructs, and they facilitate clear articulation of the rationale for decisions.
The principles underpinning evidence-informed, transparent and impactful decision making are the same across scientific fields, including environmental health. The GRADE EtD framework, along with WHO-INTEGRATE with its focus on contextual issues and inter-relationships, provide a useful starting point for consideration for decision-making for interventions in environmental health. Significant modifications will be needed, however, given the nature of the evidence base and the complex context in which environmental health interventions are designed, implemented, regulated and evaluated. The process for developing an EtD framework for environmental health interventions requires a broad range of experts, including not only environmental health scientists, but evidence synthesis and guideline methodologists, public health generalists, human rights and ethics experts, social scientists, and economists, among others. An iterative development process will be needed: a draft framework should be pilot-tested, revised and evaluated. The framework’s utility and impact on decision making, and the quality and impact of the resultant recommendations need careful evaluation, with the results used to develop future iterations.
Availability of data and materials
All data generated or analysed during this study are included in this published article.
Advisory committee on immunization practices
Breast cancer prevention partners
California environmental protection agency
Evidence and values impact on decision making
Grading of recommendations, assessment, development and evaluation
Guide community preventive services
Institute for clinical and economic review
International society for pharmacoeconomics and outcomes research
National institutes for health and care excellence
World health organization
Rooney AA, Boyles AL, Wolfe MS, Bucher JR, Thayer KA. Systematic review and evidence integration for literature-based environmental health science assessments. Environ Health Perspect. 2014;122(7):711–8.
Woodruff TJ, Sutton P. Navigation Guide Work Group. An evidence-based medicine methodology to bridge the gap between clinical and environmental health sciences. Health Affairs. 2011;30(5):931–7.
International Agency for Research on Cancer. IARC Monographs on the Identification of Carcinogenic Hazards to Humans. Lyon, France; 2019.
National Academies of Sciences, Engineering, and Medicine. The Use of Systematic Review in EPA's Toxic Substances Control Act Risk Evaluations. Washington, DC; 2021.
Guyatt GH, Oxman AD, Kunz R, et al. Going from evidence to recommendations. BMJ. 2008;336(7652):1049–51.
Moberg J, Oxman AD, Rosenbaum S, et al. The GRADE Evidence to Decision (EtD) framework for health system and public health decisions. Health Res Policy Syst. 2018;16(1):45.
Mustafa RA, Wiercioch W, Falavigna M, et al. Decision making about healthcare-related tests and diagnostic test strategies. Paper 3: a systematic review shows limitations in most tools designed to assess quality and develop recommendations. J Clin Epi. 2017;92:29–37.
Morgan RL, Kelley L, Guyatt GH, Johnson A, Lavis JN. Decision-making frameworks and considerations for informing coverage decisions for healthcare interventions: a critical interpretive synthesis. J Clin Epi. 2018;94:143–50.
Parmelli E, Amato L, Oxman AD, et al. GRADE Evidence to Decision (EtD) framework for coverage decisions. Int J Technol Assess Health Care. 2017;33(2):176–82.
Baltussen R, Marsh K, Thokala P, et al. Multicriteria decision analysis to support health technology assessment agencies: benefits, limitations, and the way forward. Value Health. 2019;22(11):1283–8.
Alonso-Coello P, Schünemann HJ, Moberg J, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016;353:i2016.
Wang Z, Norris SL, Bero L. The advantages and limitations of guideline adaptation frameworks. Implement Sci. 2018;13(1).
Guindo LA, Wagner M, Baltussen R, et al. From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Effectiveness Resour Allocation. 2012;10(1):9.
Office of Health Assessment and Translation (OHAT), Division of the National Toxicology Program, National Institute of Environmental Health Sciences. Handbook for Conducting a Literature-Based Health Assessment Using OHAT Approach for Systematic Review and Evidence Integration. 2019.
U.S. Environmental Protection Agency (EPA), Office of Research and Development. Office of Research and Development Staff Handbook for Developing IRIS Assessments (Public Comment Draft, Nov 2020). Washington, D.C., USA; 2020.
Woodruff TJ, Sutton P. The Navigation Guide Systematic Review Methodology: A Rigorous and Transparent Method for Translating Environmental Health Science into Better Health Outcomes. Environ Health Perspect. 2014;122(10):1007–14.
Alonso-Coello P, Oxman AD, Moberg J, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines. BMJ. 2016;353:i2089.
Burchett HE, Mounier-Jack S, Griffiths UK, Mills AJ. National decision-making on adopting new vaccines: a systematic review. Health Policy Plan. 2012;27(Suppl 2):ii62–76.
Wickremasinghe D, Hashmi IE, Schellenberg J, Avan BI. District decision-making for health in low-income settings: a systematic literature review. Health Policy Plann. 2016;31(suppl_2):ii12–24.
Lee G, Carr W, Group AE-BRW, et al. Updated framework for development of evidence-based recommendations by the Advisory Committee on Immunization Practices. Morb Mortal Wkly Rep. 2018;67(45):1271.
Buermeyer N, Engel C, Nudelman J, Rasanayagam S, Sarantis H. Paths to Prevention: The California Breast Cancer Primary Prevention Plan. Breast Cancer Prevention Partners. https://www.bcpp.org/our-work/policy-projects/breast-cancer-plan/. Published 2020. Accessed 28 January, 2021.
California Department of Pesticide Regulation. A Guide to Pesticide Regulation in California 2017 Update. California Environmental Protection Agency; 2017.
California Department of Toxic Substances Control Safer Consumer Products Program. Safer Consumer Products: Alternative Analysis Guide Version 1.1. 2020.
Goetghebeur MM, Cellier MS. Can reflective multicriteria be the new paradigm for healthcare decision-making? The EVIDEM journey. Cost Effectiveness Resour Allocation. 2018;16(1):54.
The Community Guide. About The Community Guide. USA Department of Health & Human Services. https://www.thecommunityguide.org/about/about-community-guide. Accessed 31 July, 2020.
Briss PA, Zaza S, Pappaioanou M, et al. Developing an evidence-based Guide to Community Preventive Services—methods. Am J Prev Med. 2000;18(1):35–43.
Institute for Clinical and Economic Review. 2020-2023 Value Assessment Framework. Institute for Clinical and Economic Review; October 23 2020.
Neumann PJ, Willke RJ, Garrison LP Jr. A Health Economics Approach to US Value Assessment Frameworks-Introduction: An ISPOR Special Task Force Report . Value Health. 2018;21(2):119–23.
Lakdawalla DN, Doshi JA, Garrison LP Jr, Phelps CE, Basu A, Danzon PM. Defining elements of value in health care-a health economics approach: an ISPOR special task force report . Value Health. 2018;21(2):131–9.
Thokala P, Devlin N, Marsh K, et al. Multiple criteria decision analysis for health care decision making—an introduction: report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016;19(1):1–13.
National Institute for Health and Care Excellence. NICE: The National Institute for Health and Care Excellence. National Institute for Health and Care Excellence. https://www.nice.org.uk/. Accessed 1 August, 2020.
National Institute for Health and Care Excellence. Developing NICE guidelines: the manual. National Institute for Health and Care Excellence,; 31 October 2014.
Scottish Intercollegiate Guidelines Network. About us. Scottish Intercollegiate Guidelines Network,. https://www.sign.ac.uk/about-us/. Published 2020. Accessed August 1, 2020.
Scottish Intercollegiate Guidelines Network (SIGN). A guideline developer’s handbook. Edinburgh: SIGN; 2019. (SIGN publication no. 50). [November 2019]. Available from URL: http://www.sign.ac.uk.
US Preventive Services Task Force. Home. US Preventive Services Task Force. https://www.uspreventiveservicestaskforce.org/uspstf/. Accessed 1 Aug, 2020.
US Preventive Services Task Force. US Preventive Services Task Force Procedure Manual. 2015.
World Health Organization. WHO handbook for guideline development. 2nd ed. Geneva, Switzerland: World Health Organization; 2014.
Rehfuess EA, Stratil JM, Scheel IB, Portela A, Norris SL, Baltussen R. The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective. BMJ Glob Health. 2019;4(Suppl 1).
Ahmed FUS. Advisory Committee on Immunization Practices (ACIP) Handbook for Developing Evidence-based Recommendations Version 1.2. Atlanta, GA: Centers for Disease Control and Prevention; 2013.
Lam J, Koustas E, Sutton P, et al. The Navigation Guide - evidence-based medicine meets environmental health: integration of animal and human evidence for PFOA effects on fetal growth. Environ Health Perspect. 2014;122(10):1040–51.
Stratil JM, Baltussen R, Scheel I, Nacken A, Rehfuess EA. Development of the WHO-INTEGRATE evidence-to-decision framework: an overview of systematic reviews of decision criteria for health decision-making. Cost Effectiveness Resour Allocation. 2020;18(1):8.
The GRADE Working Group. GRADE home. The GRADE Working Group. https://www.gradeworkinggroup.org/. Accessed 1 August, 2020.
Andrews J, Schünemann H, Oxman A, et al. GRADE guidelines: 15. Going from evidence to recommendation - Determinants of a recommendation's direction and strength. J Clin Epi. 2013;66.
Petitti DB, Teutsch SM, Barton MB, et al. Update on the methods of the U.S. Preventive Services Task Force: insufficient evidence. Ann Intern Med. 2009;150(3):199–205.
Tol R. Regulating knowledge monopolies: the case of the IPCC. Climatic Change. 2011;108:827.
Guyatt GH, Oxman AD, Kunz R, et al. Incorporating considerations of resources use into grading recommendations. BMJ. 2008;336(7654):1170–3.
Li SA, Alexander PE, Reljic T, et al. Evidence to Decision framework provides a structured "roadmap" for making GRADE guidelines recommendations. J Clin Epi. 2018;104:103–12.
Decision making. Descriptive, normative, and prescriptive interactions: Cambridge University Press; 1988.
Djulbegovic B, Elqayam S. Many faces of rationality: Implications of the great rationality debate for clinical decision-making. J Eval Clin Pract. 2017;23(5):915–22.
Chartres N, Grundy Q, Parker L, Bero L. It’s Not Smooth Sailing : Bridging the Gap Between Methods and Content Expertise in Public Health Guideline Development. Int J Health Policy Manag. 2020;9:335–43.
Funding for this work was provided by the JPB Foundation (Grant 681).
Ethics approval and consent to participate
Consent for publication
Susan L Norris was contracted by the University of California, San Francisco to develop this review. She is a member of the GRADE Working Group and has published on GRADE; she is a former employee of the World Health Organization and in that role contributed to the WHO Handbook for Guideline Development (2nd edition 2014) and to the WHO-INTEGRATE framework. Tracey Woodruff led development of the Navigation Guide. All other authors have nothing to declare.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Annex 1. Search strategy and inclusion criteria
((“Policy Making”[Mesh]) OR (“Decision Making”[Mesh])) AND (((framework[Title/Abstract]) OR (frameworks[Title/Abstract])) OR ((template[Title/Abstract]) OR (templates[Title/Abstract]) OR (tool[Title/Abstract]) OR (tools[Title/Abstract]))).
Filters: English language, systematic review, last 10 years.
The search was executed 7 January 2021 in the PubMed search engine for Medline.
Annex 2. PRISMA flow diagram
Annex 3. Results of the search: full-text review
The search of PubMed performed on 7 Jan 2021. 399 citations were identified, 25 of which were reviewed in full text: these are listed below with the reason for exclusion.
Abbreviations: E, excluded citation; E2D, evidence-to-decision; I, included citation; LMICs, low- and middle-income countries; NA (not applicable because the publication fulfilled inclusion criteria)
|Citation||Title||Include/exclude||Rationale for exclusion|
|Baltussen 2019 ||Multicriteria decision analysis to support health technology assessment agencies: Benefits, limitations, and the way forward||I||NA|
|Boivin 2018 ||Patient and public engagement in research and health system decision making: A systematic review of evaluation tools||E||Review of tools for evaluating patient and public engagement, not for decision making|
|Burchett 2012 ||National decision-making on adopting new vaccines: a systematic review||I||NA|
|Carroll 2013 ||Best fit framework synthesis: refining the method||E||Describes “best fit” method for qualitative synthesis; no EtD frameworks|
|Carvalho 2018 ||Capturing budget impact considerations within economic evaluations: A systematic review of economic evaluations of rotavirus vaccine in low- and middle-income countries and a proposed assessment framework||E||Checklist for budget impact analyses|
|Clark 2014 ||Discrete choice experiments in health economics: A review of the literature||E||Review of discrete chose experiments; no examination of EtD frameworks|
|Clark 2018 ||Measuring trade-offs in nephrology: A systematic review of discrete choice experiments and conjoint analysis studies||E||Review of discrete choice experiments in nephrology|
|Clarke 2016 ||The application of theories of the policy process to obesity prevention: a systematic review and meta-synthesis||E||Review of studies where theory underpins the policy development process or adoption of policies. Focus is on explaining the policies and not on what factors should be considered in decision-making.|
|Clifford 2017 ||What information is used in treatment decision aids? A systematic review of the types of evidence populating health decision aids||E||Focus on patient decision aids|
|Dodd 2019 ||Investigating the process of evidence-informed health policymaking in Bangladesh: a systematic review||E||Reviews factors linked to policy uptake or prioritization and how and why evidence is used; presents explanatory and descriptive information and not frameworks for normative decisions|
|Gentil 2015 ||A systematic review of socio-economic assessments in support of coastal zone management (1992–2011)||E||Describes concepts used for decision-making; presents descriptive information and not normative frameworks|
|Kolasa 2015 ||Pricing and reimbursement frameworks in Central Eastern Europe: a decision tool to support choices||E||Review revealed 33 pricing and reimbursement schemes; led to framework with two economic questions. Exclude as exclusively economic focus.|
|Kristensen 2019 ||Identifying the need for good practices in health technology assessment: Summary of the ISPOR HTA Council Working Group Report on Good Practices in HTA||E||Neither this paper or the full report (https://www.ispor.org/member-groups/councils-roundtables/health-technology-assessment-council) provides a review of EtD frameworks (referred to as “contextualization”).|
|Lam 2018 ||Decision support tools for regenerative medicine: Systematic review||E||Examines decision support tools for product development and manufacturing of cell and gene therapies in regenerative medicine; does not focus on clinical medicine, public health or environmental health|
|LaRocca 2012 ||The effectiveness of knowledge translation strategies used in public health: A systematic review||E||Review of evaluations of knowledge translation strategies and not the strategies themselves|
|Laurans 2013 ||Use of ecosystem services economic valuation for decision making: Questioning a literature blindspot||E||Focuses on the types and uses of Ecosystem Services economic Valuation (ESV), not on the criteria for decision making|
|Liverani 2013 ||Political and institutional influences on the use of evidence in public health policy. A systematic review||E||Examines factors that impact the use of evidence in policy-making|
|Mayer 2017 ||Costing evidence for health care decision making in Austria: A systematic review||E||Examines economic analyses and costing methods used in Austria; does not present EtD frameworks|
|Mustafa 2017 ||Decision making about healthcare-related tests and diagnostic test strategies. Paper 3: a systematic review shows limitations in most tools designed to assess quality and develop recommendations||I||NA|
|Sullivan 2015 ||What guidance are economists given on how to present economic evaluations for policymakers? A systematic review||E||Review of reporting guidance for economic analyses; does not deal directly with EtD concepts|
|Trapero-Bertram 2019 ||What attributes should be included in a discrete choice experiment related to health technologies? A systematic literature review||E||Review of attributes for discrete choice experiments; not a review of EtD frameworks|
|Tsoi 2013 ||Harmonization of reimbursement and regulatory approval processes: a systematic review of international experiences||E||Review of approaches to harmonize reimbursement and regulatory approval processes|
|Tsoi 2015 ||Systematic narrative review of decision frameworks to select the appropriate modelling approaches for health economic evaluations||E||Examines frameworks for selecting economic modelling approaches|
|Votruba 2018 ||A systematic review of frameworks for the interrelationships of mental health evidence and policy in low- and middle-income countries||E||Review of theories and frameworks explaining use of evidence to formulate policies in mental health in LMICs; does not review frameworks for decision-making|
|Wickremasinghe 2016 ||District decision-making for health in low-income settings: A systematic literature review||I||NA|
|From other sources/ad hoc|
|Domrumel 2020 ||Organizational aspect in healthcare decision-making: a literature review||E||Systematic review of individual criteria for decision-making (not a review of frameworks or sets of criteria)|
|Morgan 2018 ||Decision-making frameworks and considerations for informing coverage decisions for healthcare interventions: a critical interpretive synthesis.||I||NA|
References for the full-text review
1. Baltussen R, Marsh K, Thokala P, et al. Multicriteria Decision Analysis to Support Health Technology Assessment Agencies: Benefits, Limitations, and the Way Forward. Value Health. 2019;22(11):1283–1288.
2. Boivin A, L’Espérance A, Gauvin FP, et al. Patient and public engagement in research and health system decision making: A systematic review of evaluation tools. Health Expect. 2018;21(6):1075–1084.
3. Burchett HE, Mounier-Jack S, Griffiths UK, Mills AJ. National decision-making on adopting new vaccines: a systematic review. Health Policy Plan. 2012;27 Suppl 2:ii62–76.
4. Carroll C, Booth A, Leaviss J, Rick J. “Best fit” framework synthesis: refining the method. BMC Med Res Methodol. 2013;13:37.
5. Carvalho N, Jit M, Cox S, Yoong J, Hutubessy RCW. Capturing Budget Impact Considerations Within Economic Evaluations: A Systematic Review of Economic Evaluations of Rotavirus Vaccine in Low- and Middle-Income Countries and a Proposed Assessment Framework. Pharmacoeconomics. 2018;36(1):79–90.
6. Clark MD, Determann D, Petrou S, Moro D, de Bekker-Grob EW. Discrete choice experiments in health economics: a review of the literature. Pharmacoeconomics. 2014;32(9):883–902.
7. Clark MD, Szczepura A, Gumber A, Howard K, Moro D, Morton RL. Measuring trade-offs in nephrology: a systematic review of discrete choice experiments and conjoint analysis studies. Nephrol Dial Transplant. 2018;33(2):348–355.
8. Clarke B, Swinburn B, Sacks G. The application of theories of the policy process to obesity prevention: a systematic review and meta-synthesis. BMC Public Health. 2016;16(1):1084.
9. Clifford AM, Ryan J, Walsh C, McCurtin A. What information is used in treatment decision aids? A systematic review of the types of evidence populating health decision aids. BMC Med Inform Decis Mak. 2017;17(1):22.
10. Dodd M, Ivers R, Zwi AB, Rahman A, Jagnoor J. Investigating the process of evidence-informed health policymaking in Bangladesh: a systematic review. Health Policy Plan. 2019;34(6):469–478.
11. Le Gentil E, Mongruel R. A systematic review of socio-economic assessments in support of coastal zone management (1992–2011). J Environ Manage. 2015;149:85–96.
12. Kolasa K, Kalo Z, Hornby E. Pricing and reimbursement frameworks in Central Eastern Europe: a decision tool to support choices. Expert Rev Pharmacoecon Outcomes Res. 2015;15(1):145–155.
13. Kristensen FB, Husereau D, Huić M, et al. Identifying the Need for Good Practices in Health Technology Assessment: Summary of the ISPOR HTA Council Working Group Report on Good Practices in HTA. Value Health. 2019;22(1):13–20.
14. Lam C, Meinert E, Alturkistani A, et al. Decision Support Tools for Regenerative Medicine: Systematic Review. J Med Internet Res. 2018;20(12):e12448.
15. LaRocca R, Yost J, Dobbins M, Ciliska D, Butt M. The effectiveness of knowledge translation strategies used in public health: a systematic review. BMC Public Health. 2012;12:751.
16. Laurans Y, Rankovic A, Billé R, Pirard R, Mermet L. Use of ecosystem services economic valuation for decision making: questioning a literature blindspot. J Environ Manage. 2013;119:208–219.
17. Liverani M, Hawkins B, Parkhurst JO. Political and institutional influences on the use of evidence in public health policy. A systematic review. PLoS One. 2013;8(10):e77404.
18. Mayer S, Kiss N, Łaszewska A, Simon J. Costing evidence for health care decision-making in Austria: A systematic review. PLoS One. 2017;12(8):e0183116.
19. Mustafa RA, Wiercioch W, Falavigna M, et al. Decision making about healthcare-related tests and diagnostic test strategies. Paper 3: a systematic review shows limitations in most tools designed to assess quality and develop recommendations. J Clin Epi. 2017;92:29–37.
20. Sullivan SM, Wells G, Coyle D. What Guidance are Economists Given on How to Present Economic Evaluations for Policymakers? A Systematic Review. Value Health. 2015;18(6):915–924.
21. Trapero-Bertran M, Rodríguez-Martín B, López-Bastida J. What attributes should be included in a discrete choice experiment related to health technologies? A systematic literature review. PloS one. 2019;14(7):e0219905.
22. Tsoi B, Masucci L, Campbell K, Drummond M, O’Reilly D, Goeree R. Harmonization of reimbursement and regulatory approval processes: a systematic review of international experiences. Expert Rev Pharmacoecon Outcomes Res. 2013;13(4):497–511.
23. Tsoi B, O’Reilly D, Jegathisawaran J, Tarride JE, Blackhouse G, Goeree R. Systematic narrative review of decision frameworks to select the appropriate modelling approaches for health economic evaluations. BMC Res Notes. 2015;8:244.
24. Votruba N, Ziemann A, Grant J, Thornicroft G. A systematic review of frameworks for the interrelationships of mental health evidence and policy in low- and middle-income countries. Health Res Policy Syst. 2018;16(1):85.
25. Wickremasinghe D, Hashmi IE, Schellenberg J, Avan BI. District decision-making for health in low-income settings: a systematic literature review. Health Policy and Planning. 2016;31(suppl_2):ii12-ii24.
26. Dubromel A, Duvinage-Vonesch MA, Geffroy L, Dussart C. Organizational aspect in healthcare decision-making: a literature review. J Mark Access Health Policy. 2020;8(1):1810905.
27. Morgan RL, Kelley L, Guyatt GH, Johnson A, Lavis JN. Decision-making frameworks and considerations for informing coverage decisions for healthcare interventions: a critical interpretive synthesis. J Clin Epi. 2018;94:143–150.
About this article
Cite this article
Norris, S.L., Aung, M.T., Chartres, N. et al. Evidence-to-decision frameworks: a review and analysis to inform decision-making for environmental health interventions. Environ Health 20, 124 (2021). https://doi.org/10.1186/s12940-021-00794-z
- Evidence-to-decision frameworks
- Guideline development
- Environmental health interventions
- Risk management