Table 1 Toolkit of inappropriate applications of the epidemiological method
--- Part A --- Epidemiology-specific methods/techniques used to foment uncertainty and cast doubt about cause-and-effect [through biased study designs and measurements producing invalid science] | |||
Item # | Method/technique | Effects | Reference(s) |
A1 | Relying on statistical hypothesis testing; Using “statistical significance” at the 0.05 level of probability as a strict decision criterion to determine the interpretation of statistical results and drawing conclusions | Increases the probability of Type-II error; highly dependent upon sample size and statistical power; common strategy for dismissing study results that are indeterminate because of low power, or yield elevated risk ratios but do not reach an arbitrary level of statistical significance | |
A2 | Conducting statistically under-powered studies; ignoring Type-II errors | Sample size too small to detect an adverse effect, or adverse effect is too rare to be detected by a statistical study; asserting that a “negative” study (even if RR > 1) is proof of no effect. This can be addressed transparently by providing a power calculation Token studies are undertaken as a delay tactic because decision makers would rather not know the answer to a question, but want to give an appearance of concern. Thus, under-powered studies can arise through underfunding, which results in a shortfall in resources needed to be able to have the statistical power to detect a difference when one, in truth, exists. In addition, if not all needed information can be gathered to, for instance, properly control for confounding, the study will be of limited-to-no use. In such circumstances, what is actually being undertaken is a “token” study, not one that is capable of demonstrating an effect. For any number of reasons, epidemiologists may find themselves undertaking such a study. Token studies can serve special interests in two ways: (a) no effect will be able to be demonstrated, thus ensuring that the status quo is maintained; and (b) the special interest will be armed to say that a team of scientists is exploring a concern. This has the effect of being seen to be doing social good in addressing a health concern when, in fact, it will be a null study | |
A3 | Interpreting the statistical analysis or results inappropriately (see B8 below) | Concluding that a study with a risk ratio > 1 is “null” if it is not statistically significant at the 0.05 level; concluding that a risk ratio < 2 is a “null” result; concluding that lack of an elevated risk ratio is proof of no elevated risk (i.e., proof of the null hypothesis) | [62] |
A4 | Failing to use adequate follow-up methods | Not measuring appropriate endpoints through the pathogenesis of a disease process so that adverse effects can be identified (i.e., incomplete accrual problem) | [68] |
A5 | Failing to allow for adequate follow-up time | Not allowing sufficient time in a study for disease to manifest as with the latency between in utero exposure to diethylstilbestrol (DES) and appearance of cervical cancer of about 20 years, or the latency between exposure to asbestos and appearance of cancers of up to 45 years | |
A6 | Introducing inappropriate representation of total person-years of exposure, seen especially in occupational health studies | Analyses based on a seemingly large number of person-years of exposure, which often represents short-term exposure among a large number of young workers in whom duration of exposure and latency are too short to observe an effect | [72] |
A7 | Contaminating controls | Control groups that include exposed persons (cohort studies) and early disease manifestation (case–control studies). It also includes placing controls into the exposed group, and exposed subjects/participants into control groups. For example, studies may use all employees as exposed, which includes unexposed office workers. And then they use nearby (fence line) residents as unexposed when this includes workers and fence line exposed residents | [73] |
A8 | Failing to statistically analyze or account for a broad range of exposure characteristics among the exposed group (cohort studies) | Potential dilution of effect of exposure-related risks by combining individuals with a broad range of exposure characteristics or histories without proper statistical adjustment or stratification | [74] |
A9 | Selecting inappropriate controls; failing to adhere to the requirement that controls should be representative of the population from which the exposed group (cohort studies) or the cases (case–control studies) emerged | Invalidates comparisons between study groups. A control group should be representative of the population from which the “exposed” group, or the case group, emerged. A good example is one comparing exposed workers in an industrial setting to the general population. Exposed workers in occupational studies are generally young and physically able to perform heavy jobs. They are not representative of the more biologically diverse general population. Such comparisons suffer from bias due to the well-established healthy worker effect. Thus, a general population control is not appropriate in an occupational study. A proper control group would be comprised of unexposed employees from the same industry with similar demographic characteristics | [75] |
A10 | Diluting / washing out / averaging effects in descriptive population comparisons | Combining all risk groups when it is in only a relatively small susceptible group in which the signal of effect will be demonstrated (that is akin to toxicology in which the correct strain of rodent is needed for demonstrating effects) Also, dividing the exposed population into so many exposure groups that each one fails to reach statistical significance (whereas an ‘ever-never’ comparison may be more appropriate) Also, using an ‘ever-never’ categorization to hide effects, when, in fact, it is the “peak exposed” workers that have the cancers (for example, formaldehyde for leukemia in the NCI studies) | [76] |
A11 | Ignoring known synergies among components of the mixture of chemicals | To study only a portion of a mixture to which people are exposed so as to dilute the risk of the whole, in which all components may not only interact to cause effects, but also work synergistically; to assess the risk of pesticide active ingredients individually, whereas commercial pesticide products contain multiple active ingredients plus adjuvants to enhance toxicity to the target species | |
A12 | Failing to account for the effects of exposure to complex mixtures in risk assessments | Virtually all exposures—be they ambient, occupational, or through other vectors—are complex mixtures. Analysis and representation of risk associated with one chemical or agent without taking into account the effects of the mixture may lead to distorted and erroneous risk estimates | [79] |
A13 | Using inadequate or insensitive laboratory methods, measurement practices, or instrumentation | If the criteria for a positive test or detection are stringent, then false positives will be reduced (high specificity), but false negatives will be increased (low sensitivity). Laboratory methods (which may include timing of sampling, storage conditions, etc.) and/or instrumentation that do not have adequate sensitivity or levels of detection will systematically underestimate exposure levels or effects; use of varying levels of detection in analyzing blood samples by different laboratory methods | |
A14 | Inappropriate analytical methods used in the statistical analysis | Failure to utilize appropriate statistical analytical techniques, as well as failure to adjust for confounding and/or effect-modifying variables, may lead to biased or inaccurate results. Examples also include analyzing matched case–control designs using methods that do not retain the matching This can shift the outcome in either direction resulting in false negatives or false positives. However, it is important to note that some of these will always shift to the null (like from small sample sizes, common cancer endpoints, etc.) | |
A15 | Suppressing data | Suppression bias from: Failure to include, in the statistical analysis, key findings in subgroups, or failure to report or publish the findings. Deliberate omission of findings or inappropriate groupings of outcomes to hide or dilute their impact. Omission of rare events from statistical analysis, or removing outliers could include removing peak exposures where all the cancers are to be found | |
A16 | Failing to recognize the validity of evidence from qualitative methods | The exclusive reliance on quantitative methods when qualitative research can provide both a context for the variables included in the quantitative analysis as well as a context for the interpretation of the quantitative findings | [85] |
A17 | Producing erroneous or biased meta-analyses and reporting them as representing a weight-of-evidence summary result | Meta-analysis includes studies with different study designs, or it selectively excludes studies that should have been included | [79] |
A18 | Using mortality instead of morbidity data for a cancer endpoint with a high survival rate | For example, using mortality instead of morbidity for breast cancer risk associated with ethylene oxide reduces the risk estimates | |
--- Part B --- Arguments used to delay action, maintain the status quo, and create divisions among scientists [imposing inappropriate standards and methods of suppression] | |||
Item # | Argument | Effects | Reference(s) |
B1 | Insisting on the erroneous application of “criteria” for causation proposals (e.g., Bradford Hill viewpoints or aspects) in interpreting the weight of evidence in a causation analysis to infer causation | Guidelines in the form of “viewpoints” or “aspects” proffered for interpreting causation, including Bradford Hill, have been erroneously interpreted as required criteria, thereby leading to the dismissal of the weight of evidence that should properly be considered in health-protective policies. Despite outright errors in the Bradford Hill suggested guidelines, and his own expressed caveats about his proposed guidelines, the Bradford Hill guidelines are still cited by regulatory agencies, in legal proceedings, and by epidemiologists and healthcare professionals as a requirement for causation | |
B2 | Failing to disclose a conflict-of-interest in the presence of a financial conflict-of-interest, financial control of agenda-driven funders, political influences, or vested interest goals (see C6 below) | The absence of objectivity / impartiality resulting in the application of a biased design or analysis, or selective interpretation of the findings | |
B3 | Ignoring mechanistic information suggestive of adverse effects | Ignoring or dismissing information pertaining to susceptible populations having increased risk so they can be studied rather than only studying the whole population; insistence on demonstrating a consistently elevated RR associated with a very rare outcome vs. showing elevated risk for broader classifications that are mechanistically related by virtue of similar biological activity | |
B4 | Exaggerating differences, or dismissing them, when toxicological studies suggest a potential human health hazard | Failure to synthesize knowledge from all disciplines that relate to a disease process | |
B5 | Ignoring related or families of molecular structures that predict potential health hazards | Insisting on the need for more research by ignoring prior knowledge or information on structurally related compounds (e.g., the perfluorinated alkylate substances [PFASs]) | |
B6 | Focusing on studying and reporting only general population effects to the detriment of identifying and protecting from adverse health impacts the most vulnerable, chemically sensitive, and genetically susceptible in society, including children and pregnant women | Failure to protect vulnerable sub-populations and failure to recognize heightened susceptibilities; e.g., of the developing brain to neurotoxicants, or from heightened risks to immune-compromised persons; erroneously assuming that a lack of data in the literature about sub-populations or about rare conditions indicates no risk of disease associated with exposure Neurotoxic chemicals tend to be more harmful when exposures take place during fetal and early life development, as recognized by scientists, but not by regulatory agencies | |
B7 | Demanding an unusually high degree of certainty for the public health problems to be addressed; claims that more data are needed for “proof” of elevated risks; rejection of the Precautionary Principle | Demanding proof “beyond a reasonable doubt,” typical of criminal law proof requirements, although risk of a health hazard may vary due to differential susceptibility and may not be discernable beyond a reasonable doubt for an individual. In U.S. tort litigation, the typical standard of proof is “preponderance of the evidence” or “balance of probabilities” that require a determination of “more probable than not.” Environmental health advocates and public health scientists endorse a lower level of probabilistic evidence, whereas industry argues for higher standards of proof. In summary, public interest groups err on the side of caution to protect public health, whereas polluting industries press for an unattainable standard of proof that science cannot often meet. Thus, the appropriate application of science and weight of evidence support are not used in judgments as to public health policies or litigation outcomes | |
B8 | Demanding that any observed odds ratio / relative risk between exposure and disease must be 2 or greater before the study can be admitted to support expert testimony [See A3 above] | The odds ratio for a population may be 1.5 and represent millions of people at risk in a large population—as in pesticide exposure and autism—leading to a larger public health impact. This demand fails to recognize the public health importance of population attributable risk for prevalent exposures; while risk estimates may be low, the absolute number of affected people can be large. In the example provided, it is difficult to get an OR higher than about 3 because of all the practical issues of conducting a study such as multiple exposures, intermittent (seasonal) exposures, undocumented exposures due to undocumented workers not wanting to report, etc | |
--- Part C --- Tactics invoked to misdirect policy priorities through influence [imposing undisclosed values from the positions taken by special interests] | |||
Item # | Tactic | Effects | Reference(s) |
C1 | Assuming that “no data” equates to “no risk” | Lack of research about a public health issue—and a paucity of data—does not equate to “no risk.” However, the absence of data (because of the failure to conduct studies) is often invoked or misinterpreted as evidence of no risk. The absence of scientific research, including the absence of epidemiological research, does not equate to “no risk.” Mechanistic and toxicological data can be sufficient evidence to indicate human risk | |
C2 | Failing to study a critical public health issue because of political influence, financial interests, or influence of special interest groups resulting in a Repression Bias. We should not lose sight of the fact that some studies are never done because approval for them was, for some reason, not granted. Sometimes the reason is because the topic is repressed | Critical public health threats, including climate change, firearm violence, obesity/diet, and others have not been properly addressed due to the improper influence of special interests. Repression Bias arises in situations in which a line of inquiry is not pursued because the researcher is, consciously or subconsciously, aware that pursuing such a research question would upset the dominant culture/paradigm, or the funding agency. The research question may never be investigated because funding is not made available from the funding agency for its study. In practice, students may be directed away from such questions if the funding support needed to complete the research component of their graduate program cannot be secured. In the absence of new information, no action is demanded of those bearing responsibility. Students and researchers persisting in researching that which may offend the establishment could find themselves unemployable or unemployed, respectively | |
C3 | Failing to generalize health risks, and restricting the assignment of risk to local populations of exposed people despite demonstrated effects in humans elsewhere | Refusing to accept that health effects observed in one exposed population are likely to operate in much the same way in a similarly exposed population in a different location | |
C4 | Neglecting to apply or dismissing the Precautionary Principle when there is evidence to justify interventions to reduce/eliminate exposures | Insistence on occurrence of dire public health impacts (e.g., significantly increased morbidity or mortality rates) before action is taken although the weight of the evidence supports excess risk of adverse effects from exposure | |
C5 | Failing to be transparent in making explicit those value judgments that underlie decisions about selecting appropriate standards of evidence to draw policy-relevant conclusions (i.e., in suppressing dominant interests and values) | Failing to discern acceptable risks as a policy determination vs. the actual risk of exposure | |
C6 | Infiltrating editorial boards, scientific review panels, and decision-making bodies of all kinds (see B2 above) | By gaining a presence, the ability of impartial representatives to influence decisions and ensure a voting majority to support a particular stakeholder’s vested interest that is not consistent with that of the public interest | |
C7 | Misdirecting policy priorities through influence | Influencing the research agenda by funding research that supports a stakeholder industry’s position OR detracts from harm of their product. This is important because it is not about influencing individual studies, but rather the entire evidence-base relevant to a policy. The review includes a number of examples: for instance, tobacco industry funding research on contaminants of indoor air OTHER than second-hand smoke; and, sweetened beverage industry funding research on the benefits of exercise rather than on the harms of sugar | [101] |