An assessment of survey measures used across key epidemiologic studies of United States Gulf War I Era Veterans

Inclusion and exclusion criteria

We used three primary criteria to select candidate studies for inclusion in this review. First, we included epidemiologic studies of U.S. veterans that were considered “major cohorts” by the Institute of Medicine’s (IOM) 2009 Gulf War and Health review [22], according to their specified criteria for methodologic rigor, health outcomes assessment, medical evaluations, and use of laboratory testing. Second, we included two Gulf War veteran registries established by the Departments of Veterans Affairs and Defense; while these necessarily suffer from the limitations common to registries, such as non-generalizability and absence of a control group, they jointly comprise a vast repository of standardized data that may be useful in future carefully constructed research efforts. Finally, we considered studies reported since the IOM’s 2009 review, and ongoing studies.

We excluded studies of non-U.S. veterans and clinical studies or trials related to specific disease entities. Substudies of primary studies were included under the umbrella of their original study.

Survey instruments

As described in further detail below, the domains of mental and physical health include health status, functional status, symptoms, fatigue complex (fatigue, chronic fatigue syndrome (CFS), fibromyalgia, and multiple chemical sensitivities), post-traumatic stress disorder (PTSD), neurocognitive and/or psychological status, and clinical evaluations and validations. The domains of environmental exposures include vaccinations, medications, airborne exposures, radiation sources, sources of infection or contaminants, psychological trauma, and behavioral risk factors. Each of these exposure domains has multiple subdomains. This review is not a comprehensive representation of all areas of research in these studies, but includes the domains most frequently studied thus far, that are arguably perceived to be the most critical.

Mental and physical health domains

Symptoms

Symptoms were assessed by all of the studies. They were typically queried by asking if the veteran had experienced persistent or recurring symptoms during the 12 months prior to the survey, using a symptom checklist. The number of symptoms surveyed varied by study (8 to 78 symptoms). In addition, some studies collected information about symptom severity and date of onset [1, 3, 4, 10, 12, 14, 16, 17, 24, 29, 30, 34, 35, 48, 60, 62, 136, 151–153]. New Orleans [8] and Devens [25] evaluated symptoms using the Health Symptom Checklist [81, 82]. Devens also used a variant of the Psychosomatic Complaint Scale (Psychological Well-Being Scale) [26, 79, 154, 155]. In later survey versions, Devens used the Expanded Health Symptom Checklist, which included additional questions on symptom presence during the past 30 days, duration, trajectory, and frequency [3, 25, 26]. National Health Survey used the 15-item Patient Health Questionnaire to measure the occurrence of somatic symptoms during the 4 weeks prior to survey administration [29, 156].

In this subdomain, the registries and Iowa differed the most from the rest of the studies. The registries both used open-ended questions to assess symptoms, instead of a binary format [9, 30, 48]. Iowa asked additional questions about symptoms of asthma and bronchitis from the American Thoracic Society Questionnaire [157, 158].

Functional status and health status

Of the eight studies that evaluated functional status, all used the SF-12, SF-36, or Veterans SF-36, either alone or in combination [2–4, 16, 29, 34, 39, 53, 63, 159–163]. The only study that asked questions to further evaluate functional status was the Iowa study, which administered the Health Utilities Index-Mark 3 [164, 165].

Health status was one of the most commonly assessed domains. All of the studies, with the exception of Pennsylvania-Hawaii, administered a health status evaluation. However, few of the instruments were directly comparable because their response scales varied, making it difficult to compare the responses. For instance, the Military Health, National Health Survey, Air National Guard, Kansas II, VA Registry, and Seabees all asked a general health rating question [4, 9, 12, 14, 24, 29, 35, 48, 153]. However, National Health Survey, Air National Guard, and the VA Registry used a five-category response scale, while Kansas II used a four-category response scale, and the Seabees used a three-category response scale. In addition, the response scale anchor points used by Air National Guard and National Health Survey differed from that used by the VA Registry, and the responses cannot be directly compared.

Chronic multisymptom illness and related diagnoses

It is estimated that chronic multisymptom illness and other symptom-based diagnoses affect up to 25 to 32% of the Gulf War veteran population [23]. Although several instruments have been developed to diagnose these conditions, they are still challenging to distinguish due to the number of their symptoms that overlap with other illnesses. Eleven of the studies collected data on at least one of these illnesses: fatigue, chronic fatigue syndrome, fibromyalgia, multiple chemical sensitivity, and multi-symptom illness.

Fatigue was assessed by Iowa, Devens, and a VA Registry substudy using the Chalder Fatigue Scale [2, 60, 166], the Fatigue Severity Scale [40, 167], and the Multidimensional Fatigue Inventory [49, 168], respectively. Chronic fatigue syndrome was most commonly identified in National Health Survey, Iowa, VA Registry, Air National Guard, Kansas II, Military Health Study, a Devens substudy, and Seabees [2, 4, 12, 14, 24, 29, 31, 35, 41, 53, 169] using the Centers for Disease Control definition developed by Fukuda [169]. In addition, a National Health Survey substudy validated self-reported chronic fatigue syndrome, fibromyalgia, and other conditions by physician examination [53]. Fibromyalgia was evaluated by four studies (Iowa, a National Health Survey substudy, Military Health Study, and Oregon-Washington) using the American College of Rheumatology criteria [2, 35, 53, 62, 170, 171]. Multiple chemical sensitivity was queried in five studies, but they each used different measures. These measures included the Chemical Odor Intolerance Index and a scale developed by the Agency for Toxic Substances and Disease Registry [35, 49, 50, 172, 173] and multiple internally-developed measures [12, 25, 40, 41, 174]. Several studies inferred the presence of medically unexplained multisymptom illness based on the symptoms and medical conditions that they surveyed [1, 4, 12, 14, 17, 24, 35, 41, 175, 176]; however, the 2005 National Health Survey asked questions specifically about participants’ experiences with unexplained multisymptom illness. These included years of first and most recent experience, activities and treatments that improve or aggravate the condition, and status relative to initial diagnosis [29, 153]. In addition to questions about chronic fatigue syndrome, fibromyalgia, and multiple chemical sensitivity, the Military Health Study contained a large number of self-constructed questions about illness symptoms in support of their goal of validating a case definition for Gulf War Illness [16, 35].

Post-traumatic stress disorder (PTSD)

Combat-related stress was initially hypothesized to be one of the factors responsible for the illnesses afflicting Gulf War veterans [177]; as a result, it was one of the few factors surveyed by all of the studies. The most frequently used instruments included the Clinician Administered PTSD Scale (4 studies) [30, 42, 48, 54, 178, 179], PTSD Checklist Military and Civilian versions (6 studies) [2, 8, 25, 26, 29, 40, 60, 63, 64, 180, 181], Mississippi Scale for Combat-Related PTSD (3 studies) [61, 63, 182], Mississippi Scale for Desert Storm War Zone Personnel (3 studies) [4, 8, 42, 183], the Structured Clinical Interview for DSM-III-R - Patient Edition (3 studies) [30, 44, 48, 90, 184], and the Impact of Events Scale (2 studies) [51, 58, 185]. Additional PTSD instruments, including the Penn Inventory for PTSD, were used by individual studies [63].

Psychological status

Psychological status is a very broad subdomain and is by far one of the most discordant areas in assessment across studies. It includes several fields, such as family and social support, intelligence, personality, psychiatric status, depression, anxiety, memory, executive function, psychomotor function, health perception, and quality of life. There were a total of 59 tests used in this domain, and all of the studies except Kansas I and Kansas II evaluated at least some of the fields. Many of the survey instruments and tests were only used by one study, but the most frequently used instruments were the Wechsler Adult Intelligence Scale and the Trail-Making Tests, which were each used by 5 studies to measure intelligence and executive function, respectively. A summary of the instruments used to evaluate psychological status can be found in Table 4.

Clinical evaluations and stored biospecimens

Several of the studies have performed detailed clinical evaluations, including complete physicals with standard laboratory tests (VA and DoD Registries, Iowa, Air National Guard, Oregon-Washington, and a National Health Survey substudy) [4, 9, 13, 24, 30, 53, 165]. Both registries have implemented a battery of additional laboratory tests for veterans requiring further evaluation [9, 30, 52]. The Devens and Military Health studies have performed MRIs [16, 35, 43]. Many of the studies have performed analyses using blood specimens, particularly for clinical laboratory measures; however, for most, it is not clear whether sufficient samples remain to support additional biomarker research. To our knowledge, only Seabees, Military Health Study, and Kansas II have collected biospecimens for future studies [14, 16, 31, 35, 186].

Healthcare expenditures and utilization

Veterans of this era varied in their approaches to utilizing healthcare services, and this variability was noted both within and between deployed and non-deployed veterans. In particular, participation in health registries that involved a clinical evaluation, such as the VA and DOD registries, accounted for a significant number of medical encounters. Health expenditures were directly addressed only by Iowa, using the National Health Interview Survey and the National Medical Expenditure Survey [2, 187–189]. Iowa also administered a telephone survey on health utilization to a subsample of their population who suffered from multiple chemical sensitivity [190]. Devens and National Health Survey used rough measures of health utilization, asking the number of clinic visits or hospitalizations [10, 26].

Deployment-related exposure domains

Behavioral risk factors

All but Kansas I evaluated behavioral risk factors. The most common of these were alcohol and tobacco use. Alcohol use was primarily evaluated using self-constructed questions about current drinking status, alcohol abuse, and number of drinks consumed [2, 4, 10, 12, 13, 17, 25, 35, 45, 51, 195, 196]. However, Iowa, National Health Survey, and Millennium Cohort used validated instruments. Iowa used the CAGE (Cut back, Annoyance, Guilt, Eye-openers) questionnaire [2, 197], and both National Health Survey and Millennium Cohor used the alcohol component of the Patient Health Questionnaire [92].Tobacco use was most commonly queried by asking participants about their smoking status (never, past, or current smoker) [2–4, 10, 12–14, 17, 25, 35, 45, 48, 139, 198]. Additional information obtained varied by study, but included increases or decreases in daily smoking habits [198] and whether participants had smoked at least 100 cigarettes [51, 198].

Validation studies

The studies reviewed herein all gathered self-reported data, which has well-documented limitations, including recall bias and lack of correlation with objective measures. In light of these challenges to data quality, several studies performed reliability and validation analyses, using test-retest methods and comparing self-reported data to clinical evaluations and medical records.

Seabees, Iowa, Millennium Cohort, and Oregon-Washington measured the internal reliability of self-reported survey data, including demographics, physical and functional status, symptoms, health histories, vaccinations, and exposures, using test-retest methods. The results of these studies were mixed, particularly with respect to self-reported exposures, diagnosed health conditions, and symptoms. The Seabees study [31] reported high reliability of demographic attributes (kappas 0.89-1.00), and moderate reliability of exposures (0.60-0.70) and “other” survey items (0.51-0.67). However, self-reported diagnoses and symptoms had widely ranging kappa coefficients of −0.01-1.00 and −0.01-0.86, respectively [31]. The Iowa cohort reported test-retest agreement percentages of 89.6-97.0% and kappa coefficients of 0.39-0.70 for self-reported medical and psychiatric concerns [2]. In the Oregon-Washington evaluation of self-reported exposure reliability, only eight of thirteen exposures had a kappa coefficient statistically greater than 0.4, reflecting poor agreement on the majority of measures [32]. The Oregon-Washington investigators also assessed self-reported exposure misclassification by comparing reported exposure to anthrax and botulinum toxoid vaccines, chemical warfare agents, and pyridostigmine bromide to reported deployment periods, as each exposure was only possible during known periods of time. The results of this comparison suggested that these exposures may be overreported among certain subgroups of veterans [32]. Similarly, an analysis of National Health Survey data found evidence of reporting bias in self-reported anthrax vaccination data [199]. We note that it has been difficult to validate reported exposures due to the absence of objective documentation.

Clinical evaluations and medical records were used to externally validate self-reported information by several studies. As was the case for test-retest reliability analyses, the results were mixed. A New Orleans substudy examined the validity of survey-based PTSD assessments by calculating the agreement between the Structured Clinical Interview for DSM-III-R (SCID) and other PTSD diagnostic measures (agreement 82-100%) [44]. The Seabees validation study, which compared personnel and medical records to survey responses, found kappa coefficients of −0.02-0.48 for self-reported diagnosed illness, 0.41-0.76 for demographic traits, and 0.59-0.66 for “other” characteristics [31]. In the National Health Survey, comparison of reported reasons for clinic visits and hospitalizations to medical records found agreement rates greater than 90% [10, 29], and clinical evaluations confirmed an increased risk of certain conditions among deployed veterans [53]. The Iowa study drew on state registries to validate reported birth defects and cancers [60]. In addition, a case validation study in a subset of Iowa cohort members found that only 32% of those who originally reported depressive symptoms met criteria for lifetime depression after a later SCID-IV interview; however, multiple factors may have contributed to this difference [200]. The Millennium Cohort found excellent negative agreement (generally > 95%) and moderate positive agreement when comparing self-reported medical conditions to Department of Defense electronic medical records, suggesting that self-reported data may be useful in excluding the presence of conditions [146]. Finally, in a small-sample comparison of thirteen self-reported medical conditions to medical records, the Devens Cohort Study observed low-moderate kappa coefficients of 0.35-0.64 for most conditions [39]. These heterogeneous findings regarding reliability and validity lend support to concerns about the utility of self-reported data, and emphasize the importance of evaluating reliability and validity early in the study implementation period, and including objective data sources, when resources permit.