Table 2 Evidence-to-decision frameworks: Characteristics of the organizations that developed the frameworks
Organization* | Type of organizationa | Target audience; Goal | Year established; current version | Methods for development of the EtD framework | Funderb Declarations of interestc | Use and sources of evidence to inform EtD criteria | Assessment of the quality/certainty of the body of evidence | Names for recommendation or evaluation | No recommendation | Research or knowledge gaps | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Advisory Committee on Immunization Practices (ACIP) (US Centers for Diseased Control and Prevention) [20] | “US Federal advisory committee that provides expert advice to the Director of CDC and the Secretary of the US Department of Health and Human Services in the form of recommendations on the use of vaccines and related agents for control of vaccine-preventable diseases” | Public health programs in the US; health care providers and persons in the US civilian population; To develop recommendations on how to use vaccines to control disease in the US | Established 1964; EtD framework updated June 2018 | Result of expert meeting Feb 2018; adapted from GRADE framework | NR Authors report no conflicts of interest | Systematic reviews of the evidence on benefits and harms | GRADE system | “Recommendations will be communicated in the framework in one of three categories: 1) ACIP recommends vaccination for all persons in an age group or a group at increased risk for vaccine-preventable disease; 2) ACIP does not recommend the use of a vaccine; or 3) the ACIP recommendation relies upon guidance of the clinician in the context of individual clinician-patient interactions to determine whether or not vaccination is appropriate for a specific patient.” | “In some instances (e.g., when additional information is needed), ACIP might not make a recommendation, and this option is also reflected in the EtR framework separately.” | “ACIP workgroups should identify research needs and, if appropriate, prioritize them. In formulating research needs, workgroups should be as specific as possible about what is needed and why. One format is EPICOT (ref: Brown P et al. BMJ 2006;333:804–6):” | |
2 | Breast Cancer Prevention Partners (BCPP) [21] | Non-governmental organization in the US | State policy-makers, health systems and healthcare providers; Prevention of breast cancer; focus on the intersection of breast cancer and environmental health | Established 1992; Methods published September 2020 | Expert committee, input from community representatives and other stakeholders | California Breast Cancer Research Program NR | Systematic reviews of the evidence on interventions | NR | NR | NR | Research gaps highlighted for each risk factor | |
3 | California Environmental Protection Agency (CalEPA) [22] | US state governmental agency | Policy-makers and regulators in the State of California, USA; To restore, protect and enhance the environment, to ensure public health, environmental quality and economic vitality | Agency established 1991 A Guide to Pesticide Regulation, updated in 2017 | NR | NR NR | Systematic reviews for risk assessment | NR | NR | NR | Discusses authorization of agents in the context of research | |
California Environmental Protection Agency (CalEPA) [23] | US state governmental agency | Alternative Analysis analysts, preparers, practitioners, and responsible entities; To provide tools, information sources, and best practice approaches to help conduct Alternatives Analysis; challenges “responsible entities to reduce or eliminate toxic chemicals in the products consumers buy and use.... To identify Priority Products containing Chemicals of Concern, and for responsible entities to identify, evaluate, and adopt better alternatives.” | Agency established 1991 Alternative Analysis Guide, version 1.1, July 2020 | NR | NR NR | “Relevant factors” for alternatives analysis “can be quantified by available information or based on qualitative information” | Uncertainty analysis performed for individual factors assessed (sensitivity analysis or scenario analysis); no recommendation for quality of the body of evidence | NR | NR | NR | ||
4 | Evidence and Values Impact on DEcision Making (EVIDEM) [24] | Developed by a group of authors at private and academic institutions | Variable depending on decision maker; To provide a practical framework to facilitate decision making in a variety of contexts and to enhance the communication of decisions | Established 2006; first published 2008; Current framework: 2018 | Review of the literature and or decision-making processes in use; identification of the steps and components of decision-making processes; framework developed by the study authors with input from thought leaders and stakeholders | “The publication costs for this article were funded by Mark O’Freil, the Brinson Foundation, and the Payne Family Foundation”; “No sources of funding were used to conduct this study and internal sources of support for the study were provided by the WSB and BioMedCom Consultants.” Authors declare no completing interests | Relevant evidence is collected and assessed | “Quality of evidence” is assessed using bespoke tools based on existing tools, tailored to each type of evidence; sub-criteria include relevance, validity, completeness of reporting, type of evidence, and consistency. | Varies across end-users of this framework | NR | NR | |
5 | Grading of Recommendations, Assessment, Development and Evaluation (GRADE) (clinical - individual or population perspective )[11, 17] | Consortium of academics and other guideline stakeholders | Varies with the organization/entity using the GRADE system; To standardize assessment of the certainty (quality) of a body of evidence and the formulation of recommendations | Established 2000; Methods continuously updated; EtD framework published 2016 | Started with GRADE Working Group approach (Guyatt 2008); iterative process; included brainstorming, feedback from stakeholders, application to recommendations and decisions, user testing (Alonso-Coello BMJ 2016.Introduction) | European Commission Authors report no conflicts of interest | Systematic reviews, primary research, expert opinion; systematic review preferred depending on the criteria | GRADE system | Strong, weak/conditional/discretionary, for or against the intervention | Possible when “pros and cons of the intervention or option and the comparison are so closely balanced that the panel is not prepared to make a weak recommendation in one direction or the other.” and when “there is so much uncertainty that the panel concludes… that a recommendation would be speculative.” | Rcommended when: i) there is insufficient evidence supporting an intervention for a guideline panel to recommend the intervention’s use; ii) further research has a large potential for reducing uncertainty about the effects of the intervention; and iii) further research is deemed good value for the anticipated costs. | |
GRADE (coverage decisions) [9] | Consortium of academics and other guideline stakeholders | Third-party payers (public or private) for the purpose of deciding whether and how much to pay for drugs, tests, devices or services and under what conditions; To standardize assessment of the certainty (quality) of a body of evidence and the formulation of recommendations | Established 2000; Methods continuously updated; EtD framework published 2017 | Iterative process: brainstorming workshops, consultation with advisory group, user testing, feedback, application to different types of coverage decisions | European Commission One author reports having received funding ffrom the pharmaceutical industry; all other authors report no conflicts of interest | Systematic reviews, primary research, expert opinion; systematic review preferred depending on the criteria | GRADE system | Not covering, coverage only in the context of research, covering with price negotiation, restricted coverage, and full coverage | GRADE clinical EtD guidance likely applies | GRADE clinical EtD guidance likely applies | ||
GRADE (health system and public health decisions) [6] | Consortium of academics and other guideline stakeholders | Population or health system; specific population perspective depends on the nature of the decision; e.g., could be societal or governmental; To standardize assessment of the certainty (quality) of a body of evidence and the formulation of recommendations | Established 2000; Methods continuously updated; EtD framework published 2018 | Iterative process based on the GRADE clinical EtD: brainstorming workshops, consultation with stakeholders, survey of policy-makers, experience with policy briefs, applied the framework to examples, conducted workshops, observed guideline panels using the framework, conducted user testing | European Commission Authors report no conflicts of interest | Research evidence (“information derived from studies that used systematic and explicit methods”); “additional considerations include other evidence such as routinely collected data, and assumptions and logic” | GRADE | Strong, weak/conditional/discretionary, for or against the intervention | GRADE clinical EtD guidance likely applies | GRADE clinical EtD guidance likely applies | ||
6 | Guide Community Preventive Services (US Centers for Disease Control and Prevention) [25, 26] | Independent body of experts, funded by the US government and supported by the US Centers for Disease Control and Prevention | Policy-makers at the state or community level, or in community or healthcare organizations, businesses, or schools; To improve health or prevent disease | Established 1998; Methods updated 2017 (unpublished) | Review of the US Preventive Services Task Force methods, input from experts in systematic reviews, the Task Force, and other external advisors | NR NR | Systematic reviews of benefits and harms | Strong, sufficient, insufficient strength of evidence based on quality of execution (study limitations), suitability of study design, number of studies, consistency, meaningfulness of effect size | Recommend, recommend against, insufficient evidence | Yes, “insufficient evidence”, i.e. unable to determine effectiveness | Each chapter includes research and knowledge gaps focusing on effectiveness, applicability in other populations, economic consequences, implementation barriers, and opportunities to improve technical efficiency | |
7 | Institute for Clinicaland Economic Review (ICER) [27] | Independent, non-profit research organization based in the US | Health system managers, policy makers, payers “evaluates medical evidence and convenes public deliberative bodies to help stakeholders interpret and apply evidence to improve patient outcomes and control costs.” | Established 2006: Methods updated Oct. 2020 | The initial framework was developed with input from a multi-stakeholder workgroup; followed by national public comment, review and feedback by a broad range of stakeholders. | ICER (which receives its funding from government grants and non-profits foundations); a separate policy program is funded in part by health insurers and other industries NR | Systematic reviews of comparative effectiveness | Systematic reviews of comparative effectiveness include an assessment of certainty of the body of evidence. | Provide assessment as to an intervention’s “value for money”. | NR | Consider future research needs | |
8 | International Society for Pharmacoeconomics and Outcomes Research (ISPOR) [28,29,30] | Non-profit, multidisciplinary, multistakeholder professional organizationin pharmacoeconomics and outcomes research | Variable depending on who is making decisions To assess the value of new technologies (value is defined from an economic perspective and includes “gross value” (what individuals or others acting on their behalf would be willing to pay to acquire more health care or other goods or services), and “opportunity cost” (what benefits or other resources they are willing to forgo to obtain them)) | Established 1995; 2018 | Task force appointed by ISPOR, with input from advisory board and stakeholder panel; reviewed existing examples of value assessment frameworks; through disussion and with feedback and review, arrived at final framework . | NR NR | Data are used to measure the performance of alternatives; sources include systematic reviews, modelling, expert opinion, and other approaches as appropriate. | MDCA includes an “uncertainty analysis to understand the level of robustness of the MCDA results” | MDCA results in an assessment of the “total value” of the alternatives under consideration. | NR | NR | |
9 | National Institute for Health and Care Excellence (NICE); focus on guidelines [31, 32] | UK government organization; a non-departmental public body that provides national guidance and advice to improve health and social care in England. | Individual healthcare providers; local authorities, commissioners and managers; other providers of health and social services; To produce evidence-based recommendations on a range of topics, including prevention and management of specific considtions, improving health, managing medicines, providing social care and support, and planning services for communities | Established 1999 Methods guidance published 2014, updated October 2018 | “The processes and methods described in this manual are based on internationally recognized standards, and the experience and expertise of the teams at NICE, the contractors..., NICE committee members and stakeholders. They are based on internationally accepted criteria of quality...and primary methodological research and evaluation undertaken by the NICE teams. They draw on the Guideline Implementability Appraisal tool to ensure that recommendations are clear and unambiguous, making them easier to implement.” | NR NR | Systematic reviews of the evidence; “colloquial evidence” can be included (e.g. expert testimony); for economic analyses: systematic review of existing models; may perform de novo model | Individual study quality assessed according to study design; the certainty or confidence in the findings should be presented at outcome level using GRADE or GRADE-CERQual; body of evidence for each outcome is high, moderate, low, very low “certainty or confidence of evidence”; it integrates a review of the quality of cost-effectiveness studies... it does not use ‘overall summary’ labels for the quality of the evidence across all outcomes: “strength of evidence (reflecting the appropriateness of the study design to answer the question and the quality, quantity and consistency of evidence)”: classified as no, weak, moderate, strong or inconsistent evidence | NICE uses the wording of recommendations to reflect the strength of the evidence (e.g. “offer, advise, refer versus consider”) | “If evidence of efficacy or effectiveness for an intervention is either lacking or too low quality for firm conclusions to be reached, the committee.... may: make a ‘consider’ recommendation based on the limited evidence... decide not to make a recommendation and make a recommendation for research... recommend that the intervention is used only in the context of research... recommend not to offer the intervention.” | Include recommendations for research; “The committee should select up to 5 key recommendations for research that are likely to inform future decision-making (based on a systematic assessment of gaps in the current evidence base).” | |
10 | Non-profit collaboration between governmental and non-governmental (including academic) organizations in the US and Europe | Clinicians,policy-makers, professional societies, health care organizations, goernment agencies making prevention-oriented guidelines To provide a methodology for evaluating the evidence and to support evidence-basesd decision-making in environmental health | Published 2011 Current version 2014 | Collaborative process among clinicians, systematic review and guidelines experts, statistics, epidemiology, and environmental health scientists; based on GRADE | For 2009–2013, support for the development and dissemination of the Navigation Guide methodology was provided by the Clarence Heller Foundation, the Passport Foundation, the Forsythia Foundation, the Johnson Family Foundation, the Heinz Endowments, the Fred Gellert Foundation, the Rose Foundation, Kaiser Permanente, the New York Community Trust, the Philip R. Lee Institute for Health Policy Studies, the Planned Parenthood Federation of America, the National Institute of Environmental Health Sciences ... and U.S. EPA STAR grants. Authors report no conflicts of interest. | Systematic reviews of the evidence on the risks to human health of exposure to chemicals, and the effects of prevention and mitigating interventions | The quality of individual studies and the overall body of evidence is rated, including for human and animal data | NR (Statements about the health risks of substances include: known to be toxic, probably toxic, possibly toxic, not classificable, or probably not toxic.) | NR | NR | ||
11 | Supported by the Scottish government, but with editorial independence | Health and social care professionals, patients; To understand and use medical evidence to make decisions about healthcare, reduce unwarranted variations in practice, make sure patients get the best care available, improve healthcare across Scotland | Established 1993; Handbook first published 2008; Current version: November 2019 | Based on 2013 GRADE/DECIDE work | Core funding for SIGN activities comes from Healthcare Improvement Scotland NR | Systematic reviews of the evidence | GRADE system | Strong recommendation against; conditional recommendation against; recommendation for research and possibility conditional recommendation for use restricted to trials; conditional recommendation for; strong recommendation for | NR | Include recommendations for research | ||
12 | Independent body of experts, funded by the US government | Primary care clinicians, also policy-makers, payers, patients; Tp provide recommendations for preventive care for general, primary care populations in the US who are asymptomatic with respect to the condition addressed by the intervention | Established 1984; Procedure manual Dec 2015 | Developed by the Methods Working Group of the USPSTF using an iterative process based on the methods literature, international standards and practices; approved by the Task Force | NR NR | Systematic reviews of benefits and harms; sometimes on contextual questions also | Assessment of certainty across the analytic framework, where certainty is “the likelihood tha thte USPSTF assessment of the net benefit of a preventive service is correct”; “assessing the certainty of evidence requires a complex synthesis of all evidence across the entire analytic framework” in order to determine if “the results observed in the indivudal studies in the body of evidence would be expected when the intervention is delivered to asymptomatic persons by providers in US primary care settings”. | Grades (or strength) of recommendations: A. The USPSTF recommends the service. There is high certainty that the net benefit is substantial. B The USPSTF recommends the service. There is high certainty that the net benefit is moderate, or there is moderate certainty that the net benefit is moderate to substantial. C. The USPSTF recommends selectively offering or providing this service to individual patients based on professional judgment and patient preferences. There is at least moderate certainty that the net bene☐t is small. D. The USPSTF recommends against the service. There is moderate or high certainty that the service has no net bene☐t or that the harms outweigh the benefits. | I Statement. The USPSTF concludes that the current evidence is insufficient to assess the balance of benefits and harms of the service. Evidence is lacking, of poor quality, or conflicting, and the balance of benefits and harms cannot be determined. | Reports on evidence gaps for each clinical preventive service the Task Force reviews; and there is an annual report to congress that focuses on evidence gaps as well | ||
13 | World Health Organization (WHO) - guidelines [37] | United Nations agency | National and local policy makers and program managers; To prevent disease and promote health | Established 1948; Handbook for guideline development, 2nd edition: 2014 | Adopted directly from the then-current (2014) GRADE approach | The Bill & Melinda Gates Foundation NR | Systematic reviews of benefits and harms and other considerations as indicated | GRADE system | Strong, conditional, for or against | No explicit guidance provided; not prohibited | Added as a requirement in 2019; specific methods and guidance under development | |
14 | WHO-INTEGRATE [38] | Developed for WHO; applies to any entity making public health or health system guidelines | National and local policy makers and program managers; To prevent disease and promote health | Publshed 2019 | i) an analysis of WHO’s norms and values; ii) a systematic review of EtD criteria in clinical care and public health; iii) key informant interviews; iv) application to completed WHO guidelines; v) focus groups; vi) peer review; and vii) the development of guidance and prompts for completing the EtD. | Funding provided by the World Health Organization Department of Maternal, Newborn, Child and Adolescent Health through grants received from the United States Agency for International Development and the Norwegian Agency for Development Cooperation One author is a WHO employee; two authors are members of the GRADE Working Group | Evidence gathered as needed to inform key EtD considerations; systematic reviews for key criteria | No specific guidance provided but an assessment is recommended | No specific guidance provided | NR | NR |