Response to Toshihide Tsuda, Yumiko Miyano and Eiji Yamamoto 
Environmental Health volume 21, Article number: 100 (2022)
In August 2021, we published in Environmental Health a Toolkit for detecting misused epidemiological methods with the goal of providing an organizational framework for transparently evaluating epidemiological studies, a body of evidence, and resultant conclusions. Tsuda et al., the first group to utilize the Toolkit in a systematic fashion, have offered suggestions for its modification.
Among the suggested modifications made by Tsuda et al., we agree that rearrangement of Part A of the Toolkit to reflect the sequence of the epidemiological study process would facilitate its usefulness. Expansion or adaptation of the Toolkit to other disciplines would be valuable but would require the input of discipline-specific expertise. We caution against using the sections of the Toolkit to produce a tally or cumulative score, because none of the items are weighted as to importance or impact. Rather, we suggest a visual representation of how a study meets the Toolkit items, such as the heat maps used to present risk of bias criteria for studies included in Cochrane reviews. We suggest that the Toolkit be incorporated in the sub-specialty known as “forensic epidemiology,” as well as in graduate training curricula, continuing education programs, and conferences, with the recognition that it is an extension of widely accepted ethics guidelines for epidemiological research.
We welcome feedback from the research community about ways to strengthen the Toolkit as it is applied to a broader assemblage of research studies and disciplines, contributing to its value as a living tool/instrument. The application of the Toolkit by Tsuda et al. exemplifies the usefulness of this framework for transparently evaluating, in a systematic way, epidemiological research, conclusions relating to causation, and policy decisions.
We note that our Toolkit has, most recently, inspired authors with discipline-specific expertise in the field of Conservation Biology to adapt it for use in the Biological Sciences.
Tsuda et al.  recently applied our Toolkit for detecting misused epidemiological methods  that serves to organize and formalize the transparent evaluation of research papers or reports. Our focus in this comment is not on the specialized content area of Tsuda et al., but rather on the recommendations made by them to improve and manage the Toolkit. In addition, the potential value of the Toolkit in the training of professionals in our field, as well in the sub-specialty of forensic epidemiology, is emphasized.
Response to recommendations to improve the Toolkit
We welcome the suggestion by Tsuda et al.  that the items in Part A (i.e., method-related misuses) of the Toolkit  be rearranged in the order of how a study is designed, conducted, analyzed, and reported. This could help other potential users of the Toolkit navigate through all of the items. Most items in the Toolkit apply to epidemiological studies individually, or to a collection of topic-related studies that constitute the body of evidence, including review articles, meta-analyses, and policy documents. The suggestion to expand the Toolkit to other related health disciplines, such as mechanistic studies, requires methodological expertise across a variety of disciplines. The Toolkit could indeed be adapted to transparently evaluate studies beyond epidemiology, paving the way to multi- and inter-disciplinary collaborations leading to even more useful toolkits.
Additional Part A items suggested by Tsuda et al., are, on consideration, adequately covered by existing items in the Toolkit and, as noted, they do sometimes overlap. We did not consider intention, that is whether a problem with a study is “deliberate” or “intentional,” because our aim is to help reviewers identify flaws in a body of evidence, regardless of how these flaws were created. One of us (L.A.B.) has grappled with a similar issue regarding the identification of problematic studies in systematic reviews . There, she and her coauthors found that intention does not really matter when the objective is to identify untrustworthy studies, but that it has legal implications. In our Toolkit, we are interested in identifying misused (i.e., bad) studies.
Caution against using a simple tally to suggest the extent of misuse
We caution against the simple tallying of overall Toolkit items (n = 33) or reporting the proportion of items met for comparing the extent of misuse across studies. In so doing, this suggests a score based on the Toolkit per se. The purpose of the Toolkit remains one of assisting educators, reviewers, researchers, and policymakers to identify how epidemiological studies can be flawed and misinterpreted. It was not designed to produce an overall, cumulative score. Since none of the items are weighted, an overall score would be misleading in making comparisons. At minimum, scores across each of the three dimensions of the Toolkit (i.e., Part A, Methods / Techniques [n = 18]; Part B, Arguments [n = 8]; and Part C, Tactics [n = 7]) could be more meaningful. We suggest, instead, however, a visual representation of how a study meets the Toolkit items, such as the heat maps used to present risk of bias criteria for studies included in Cochrane reviews . Akin to risk of bias assessment, it might be appropriate to have more than one reviewer independently assessing the studies; in fact, at least two evaluators are normally required for risk of bias assessment in Cochrane reviews . Furthermore, it would be interesting to explore the level of consensus and inter-rater agreement when the same studies are evaluated by multiple reviewers using the Toolkit.
Reinforcing the role of ethics in professionalism
We note that the principles delineated in the Toolkit are complementary to long-standing and widely accepted ethics guidelines for epidemiologists and biomedical researchers [5,6,7]. These guidelines were developed to raise awareness of, among others, undeclared conflicting interests, transparency, erroneous assumptions, and unethical behavior carried out by our own colleagues in the field of epidemiology, emanating from public health agencies, scientific journals, international expert panels, and academia. The guidelines are intended to prevent such breaches. We can only re-emphasize the critical importance of incorporating into graduate training curricula, continuing education programs, and research conferences such topics as they relate to professional ethics and research integrity. Every opportunity to broaden discussion, and to increase both awareness and understanding of ethics writ large in the health professions should be taken, especially by mentors. This applies not only to epidemiology, but across all specialty and sub-specialty areas in the health sciences.
Enhancing the evaluative and forensic utility of the Toolkit
We look forward to learning of additional applications of our Toolkit and would welcome further suggestions for its improvement. Its wider application globally would likely help with its refinement and utility.
The Toolkit, in our view, has potential value to the branch of evaluative public health science currently defined as “forensic epidemiology” . We recommend that professionals in our field receive training, incorporating our Toolkit, allowing for a systematic approach in transparently deconstructing the way that the effect of exposures on health has been, and continues to be, misused in relation to health effects.
The Toolkit as a living tool/instrument
We believe that a system by which examples of misuse are accumulated should be created. Examples of misuse could serve as a reference for assessing the utility of the Toolkit in protecting the public’s health and for its further enhancement. This task could be undertaken by an established public health agency or a professional organization whose members are not entirely made up of volunteers. In this way, the Toolkit would become a living document; its periodic updates could be made accessible on a website. We propose that the International Society for Environmental Epidemiology (ISEE) be asked to consider taking on this task.
As awareness of the utility of toolkits increases, their application to various sub-specialty areas within the field of epidemiology will further help to identify their respective strengths and weaknesses, paving the way to more refined and useful toolkits. With ever-expanding research outputs globally, toolkits could well become an effective aid in the evaluation process, facilitating transparency in critiques of published studies.
A year after publication of our Toolkit article , not only do we see its application by Tsuda et al.  in the field of Epidemiology, but now, a few weeks later in Conservation Letters, a journal of the Society for Conservation Biology, our Toolkit is seen to have inspired its authors to fashion it for use in the Biological Sciences . Burgman et al.  have used our Toolkit  as a starting point and adapted the items to reflect issues of special relevance to Conservation Biology, focusing on issues that are relevant in conservation and environmental science. We are gratified to see, as noted above, awareness of the utility of toolkits increasing, not only in the field of Epidemiology, but now too its utility to a broader assemblage of research studies and disciplines, the latest being the field of Conservation Biology, a field where rational evidence for preventing harms on an ecological and global scale is so critical. This wider application of our Toolkit  globally can only help with its refinement and utility in serving the public interest.
Availability of data and materials
This declaration is not applicable.
Tsuda T, Miyano Y, Yamamoto E. Demonstrating the undermining of science and health policy after the Fukushima nuclear accident by applying the Toolkit for detecting misused epidemiological methods. Environ Health21, 77 (2022). https://doi.org/10.1186/s12940-022-00884-6. Accessed 12 September 2022.
Soskolne CL, Kramer S, Ramos-Bonilla JP, Mandrioli D, Sass J, Gochfeld M, et al Toolkit for detecting misused epidemiological methods. Environ Health20, 90 (2021). https://doi.org/10.1186/s12940-021-00771-6. Accessed 12 September 2022.
Parker L, Boughton S, Lawrence R, Bero L. Experts identified warning signs of fraudulent research: a qualitative study to inform a screening tool. J Clinical Epidemiology, 151; 1–17 (2022). https://doi.org/10.1016/j.jclinepi.2022.07.006 Accessed 12 September 2022.
Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. on behalf of the Cochrane Bias Methods Group. (2019). Chapter 7: Considering bias and conflicts of interest among the included studies. In: Cochrane Handbook for Systematic Reviews of Interventions. https://training.cochrane.org/handbook/current/chapter-07#section-7-4. Accessed 12 September 2022.
Kramer S, Soskolne CL. Ethics Guidelines in Environmental Epidemiology: Their Development and Challenges We Face. Curr Envir Health Rpt4, 142–155 (2017). https://doi.org/10.1007/s40572-017-0138-z. Accessed 12 September 2022.
Kramer S, Soskolne CL, Mustapha BA, Al-Delaimy WK. Revised ethics guidelines for environmental epidemiologists. (Editorial.) Environ. Health Perspect120(8), August 2012:A299-A301. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3440101/pdf/ehp.1205562.pdf. Accessed 12 September 2022.
International Society for Environmental Epidemiology. Ethics Guidelines for Environmental Epidemiologists, 25 April 2012. https://www.iseepi.org/docs/ISEE_Ethics_Guidelines_adopted_april_25_2012-English.pdf. Accessed 12 September 2022.
Forensic epidemiology. From Wikipedia, the free encyclopedia. https://en.wikipedia.org/wiki/Forensic_epidemiology. Accessed 12 September 2022.
Burgman M, Chiaravalloti R, Fidler F, Huan Y, McBride M, Marcoci A, et al A toolkit for open and pluralistic conservation science. Conservation Letters, e12919 (2022). https://doi.org/10.1111/conl.12919. Accessed 2 October 2022.
No funding was received for the development and production of this response.
I declare that the authors have no competing interests as defined by BMC, or other interests that might be perceived to influence the results and/or discussion reported in this response.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Soskolne, C.L., Bero, L.A., Kramer, S. et al. Response to Toshihide Tsuda, Yumiko Miyano and Eiji Yamamoto . Environ Health 21, 100 (2022). https://doi.org/10.1186/s12940-022-00913-4