Table 3 Criteria included in evidence-to-decision frameworks
Priority | Benefits and harms | Values | Economc implications | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
Organization | Priority of the problem | Desirable effects | Undesirable effects | Certainty of evidence regarding desirable and undesirable effects | Balance of effects | Values | Certainty of evidence regarding values | Resource considerations | Certainty of evidence regarding resources | Cost-effectiveness | |
1 | Advisory Committee on Immunization Practices (ACIP) (US Centers for Diseased Control and Prevention) [20, 39] | Is the problem of public health importance? | How substantial are the desirable anticipated effects? | How substantial are the undesirable anticipated effects? | What is the overall certainty of this evidence for the critical outcomes? | Do the desirable effects outweigh the undesirable effects? | Does the target population feel that the desirable effects are large relative to undesirable effects? | Is there important uncertainty about or variability in how much people value the main outcomes? | Is the intervention a reasonable and efficient allocation of resources? | NI | Is the intervention a reasonable and efficient allocation of resources? (Cost-effectiveness is included in the explanatory text.) |
2 | Breast Cancer Prevention Partners (BCPP) [21] | Does the intervention address cross-cutting, systemic problems? | Is there evidence that the intervention has been successful in the past or does it show potential for success? | Was there general agreement that the intervention would do no harm, i.e. not create unintended consequences? | NI | NI | NI | NI | NI | NI | NI |
3 | California Environmental Protection Agency (CalEPA) [22] | NI | NI | Deciding whether the proposed or current use of a pesticide results in an unacceptable risk; identifying options to minimize those risks | NI | NI | Evaluating those options according to a value system that includes scientific, social, legal and economic factors, as well as practicality and enforceability. | NI | Evaluating those options according to a value system that includes scientific, social, legal and economic factors, as well as practicality and enforceability. | NI | NI |
California Environmental Protection Agency (CalEPA) [23] | Exposure assessment (identify exposure pathways and estimate the exposure impact, including chemical quantities and household, workplace and market presence across the lifecycle) | Product function (service or utility the product provides) and performance: includes: Principal manufacturer-intended uses or applications; functional and performance attributes, and relative function and performance; applicable legal requirements; useful life of the product; whether an alternative exists that is functionally acceptable, technically feasible, and economically feasible. | Adverse impacts (Adverse environmental impacts; Adverse public health impacts; Adverse waste and end-of-life impacts; Environmental fate Materials and resource consumption impacts; Physical chemical hazards; Physicochemical properties, Associated exposure pathways and life cycle segments). | Uncertainty analysis performed for individual factors assessed (sensitivity analysis or scenario analysis) | NI | NI | NI | Economic impacts (costs) Public health and environmental costs; cost to government agencies and non-profit organizations; internal cost Materials resource and consumption impacts | NI | NI | |
4 | Evidence and Values Impact on DEcision Making (EVIDEM) [24] | Disease severity; size of affected population; unmet needs; population priorities and acess | Comparative effectiveness; comparative patient-perceived health / patient-reported outcomes, type of preventive or therapeutic benefit | Comparative safety/tolerability | Quality of evidence (validity, relevance, completeness of reporting, type of evidence) | NI | NI | NI | Comparative cost consequences - cost of intervention, other medical costs, non-medical costs; Opportunity costs and affordability | NI | NI |
5 | Grading of Recommendations, Assessment, Development and Evaluation (GRADE) (clinical - individual perspective) [17] | Is the problem a priority (from the perspective of an individual patient)? | How substantial are the desirable anticipated effects? | How substantial are the undesirable anticipated effects? | What is the overall certainty of the evidence of effects? | Does the balance between desirable and undesirable effects favour the intervention or the comparison? | Is there important uncertainty about, or variability in, how much people value the main outcomes? | (combined with values criterion) | How large are the resource requirements (costs)? | What is the certainty of the evidence of resource requirements (costs)? | Does the cost-effectiveness of the intervention (the out-of-pocket cost relative to the net desirable effect) favour the intervention or the comparison? |
GRADE (clinical - population perspective) [17] | Is the problem a priority? | Same as clinical recommendations, individual perspective | Same as clinical recommendations, individual perspective | Same as clinical recommendations, individual perspective | Same as clinical recommendations, individual perspective | Same as clinical recommendations, individual perspective | Ni | Same as clinical recommendations, individual perspective | Same as clinical recommendations, individual perspective | Does the cost-effectiveness of the intervention favor the intervention or the comparison? | |
GRADE (coverage decisions) [9] | Is the problem a priority? | Same as clinical recommendations, individual perspective | Same as clinical recommendations, individual perspective | Same as clinical recommendations, individual perspective | Does the balance between desirable and undesirable effects favour the option or the comparison? | Is there important uncertainty about how much people value the main outcomes? | Ni | How large are the resource requirements (costs)? | What is the certainty of the evidence of resource use? | Does cost-effectiveness favor the option or the comparison? | |
GRADE (health system or public health decisions or recommendations) [6] | Is the problem a priority? | Same as clinical recommendations, individual perspective | Same as clinical recommendations, individual perspective | Same as clinical recommendations, individual perspective | Same as clinical recommendations, individual perspective | Same as clinical recommendations, individual perspective | Ni | Same as clinical recommendations, individual perspective | Same as clinical recommendations, individual perspective | Does the cost-effectiveness of the intervention favor the option or the comparison? | |
6 | Guide Community Preventive Services (US Centers for Diseased Control and Prevention) [26] | ___ | Benefits | Harms | “Strength of evidence” based on number of studies, study design, quality of execution, consistency, and meaningful effect | Not explicit | ___ | Ni | NI | NI | (Examined but does not contribute to decisions.) |
7 | Institute for Clinical and Economic Review (ICER) [27] | ___ | Comparative clinical effectiveness (involves weighing the benefits and harms/burdens of one treatment option versus another) | Potential other benefits or disadvantages | Confidence in the body of evidence and the accuracy of estimates of risks and benefits; certainty of net benefit | Comparative clinical effectiveness (involves weighing the benefits and harms/burdens of one treatment option versus another) | ___ | Ni | Potential budget impact (for short-term affordability assessment) | NI | Incremental cost-effectiveness; long-term value for money |
8 | International Society for Pharmacoeconomics and Outcomes Research (ISPOR) [29, 30] | Severity of disease | Assessed as QALYs | Assessed as QALYs | ___ | ___ | Value of hope; Also (paraphrased): value as incorporated into QALYS | Ni | Net costs (resulting directly from the intervention) | NI | QALYs gained |
9 | National Institutes for Health and Care Excellence (NICE) (UK) [32] | __ | Benefits | Harms | Quality/certainty of the evidence | Balance of benefits and harms; magnitude and importance of the benefits and harms of an intervention, and the potential for unintended consequences. | Relative values placed on outcomes | Ni | Costs, resource use and economic considerations | NI | Cost effectiveness and other types of economic analysis |
10 | Exposure prevalence | Benefits | Assessment of risk of adverse health outcomes from (paraphrased) “exposure to a chemical or class of chemicals or other environmental exposure” | Assess quality of evidence (on risk or toxicity) | __ | Values and preferences | Ni | Costs and benefits | NI | NI | |
11 | Scottish Intercollegiate Guideline Network (SIGN)a [34] | Is this question a priority? | What benefit will the proposed intervention/action have? | What harm might the proposed intervention/action do? | Quality of evidence (Subcriteria: How reliable are the studies in the body of evidence? Are the studies consistent in their conclusions? Are the studies relevant to our target population? Are there concerns about publication bias?) | Balancing benefits and harms | How do patients value different outcomes? | Ni | Is the intervention /action implementable in the Scottish context? Consider existing SMC advice, cost effectiveness, financial, human and other resource implications. | NI | Is the intervention/action implementable in the Scottish context? Consider existing SMC advice, cost effectiveness, financial, human and other resource implications. |
12 | US Preventive Services Task Force (USPSTF) [36] | NI | Benefits | Harms | Certainty of net benefit | Magnitude of net benefits | Ni | Ni | NI | NI | NI |
13 | World Health Organization (WHO) [37] | Priority of the problem | NI | NI | Quality of the evidence | Balance of benefits and harms | Values and preferences | Ni | Resource implications | NI | NI |
14 | NI | NI | NI | Quality of evidence | Balance of benefits and harms | NI | Ni | Financial and economic considerations | NI | NI | |
Other | ||||||||
|---|---|---|---|---|---|---|---|---|
Equity | Acceptability | Feasibility | Autonomy | Sustainability | Legal and regulatory considerations | Political considerations | Human rights | Other considerations |
NI | Is the intervention acceptable to key stakeholders? | Is the intervention feasible to implement? | NI | NI | NI | NI | NI | Framework includes additional criteria “balance of consequences” |
NI | NI | NI | NI | NI | NI | NI | NI | Does the intervention support the science-based intervention goals? Is the intervention in alignment with the Guiding Principles of Paths to Prevention? Mentioned in the text only: Can the intervention be implemented in a wide range of realms? |
NI | Evaluating those options according to a value system that includes scientific, social, legal and economic factors, as well as practicality and enforceability. | Evaluating those options according to a value system that includes scientific, social, legal and economic factors, as well as practicality and enforceability. | NI | NI | Evaluating those options according to a value system that includes scientific, social, legal and economic factors, as well as practicality and enforceability. | NI | NI | Identifying options to minimize those risks Selecting an effective course of action to reduce or eliminate unacceptable health or environmental risks |
NI | “Whether an alternative exists that is functionally acceptable..” - Table 3-1D (page 233) | “Whether an alternative exists that is ... technically feasible…” (page 233) | NI | NI | “to identify the product function, performance, and the legal requirements of the Priority Product and alternatives to ensure the selected choice is feasible” | NI | NI | Life cycle impacts (from raw materials extraction through end-of-life disposal) |
(Subcriteria of “population priorites and acess”: special populations, rare diseases, etc.) | NI | Mandate and scope of the health system; System capacity and appropriate use of intervention; Political, historical and cultural context | (Sub-criteria of”comparative patient-perceived health”: impact on autonomy) | NI | NI | NI | NI | Expert consensus/clinical practice guidelines Common goals and specific interests Environmental impact |
What would be the impact on health equity? | Is the intervention acceptable to patients, their caregivers, and healthcare professionals? | Is the intervention feasible for patients, their caregivers and healthcare providers? | NI | NI | NI | NI | NI | NI |
Same as clinical recommendations, individual perspective | Is the intervention acceptable to key stakeholders? | Is the intervention feasible to implement? | NI | NI | NI | NI | NI | NI |
Same as clinical recommendations, individual perspective | Is the option acceptable to stakeholders? | Is the option feasible to implement? | NI | NI | NI | NI | NI | NI |
Same as clinical recommendations, individual perspective | Is the option acceptable to key stakeholders? | Is the option feasible to implement? | NI | NI | NI | NI | NI | NI |
What would be the impact on health equity? | NI | “Barriers” - a secondary consideration | NI | NI | NI | NI | NI | Evidence gaps Applicability to US populations, US settings of implementation, and intervention characteristics |
Degree to which the intervention differentially benefits a historically disadvantaged or underserved community | Delivery mechanism or relative complexity may affect adherence and outcomes | NI | NI | NI | NI | NI | NI | “...Impact of the condition on family and caregivers...” “...Impact on improving return to work/or overall productivity...” “…potential effectiveness of future treatments…” Similarity of mechanism of action to that of other active treatments |
Equity | NI | NI | NI | NI | NI | NI | NI | Labor productivity Adherence-improving factors Value of reducing uncertainty due to a new diagnostic Fear of contagion (benefit in reducing anxiety of future disease spread) Insurance value (physical and financial risk protection from new treatments) Value of hope (uncertainty: benefit may be greater than the mean) Real option value (opportunity to benefit from future advances in medicine) Scientific spillover (impact of a new technology on future patients) |
Impact on health inequities | NI | Feasibility of implementation | NI | NI | NI | NI | NI | Effect on population health, if appropriate to the topic; any ethical issues, social value judgements, equity considerations and inequalities in outcomes, and policy imperatives, as well as equality legislation |
NI | NI | NI | NI | NI | NI | NI | NI | Availability of safer alternatives |
Equity | Is the intervention/action acceptable to patients and carers compared to comparison? Consider benefits vs harms, quality of life, other patient preferences. | Is the intervention/action implementable in the Scottish context? Consider existing SMC advice, cost effectiveness, financial, human and other resource implications | NI | NI | NI | NI | NI | “Are there any common comorbidities that could have an impact on the efficacy of the intervention? “(in the “considered judgement pro-forma 2014″ form). “Equity” and “How do patients value the different outcomes?” are mentioned in the SIGN 50 manual but not in the “considered judgement pro-forma 2014″ form. |
NI | NI | NI | NI | NI | NI | NI | NI | Other factors can be considered for determing the adequacy of evidence for a recommendation: prevalence or natural history of the target condition, and biological plausibility, clinical relevance and applicability of the evidence, among others. “Grade changes may also result from changes in context (clinical context, societal values for specifc outcomes, and context of intervention and treatment.”. |
Equity and human rights | Acceptability | Feasibility | NI | NI | NI | NI | Equity and human rights | NI |
Health equity, equality and non-discrimination | Human rights and sociocultural acceptability | Feasibility and health system considerations | NI | NI | NI | NI | Human rights and sociocultural acceptability | Societal implications |