PROMIS:
The HEAL measures were developed using the instrument development methodology of the NIH Roadmap Initiative, Patient Reported Outcomes Measurement Information System (PROMIS). For more information about PROMIS measures and methodology, visit http://www.nihpromis.org/
PROMIS tools confer 2 major advantages: They are standardized to be comparable across diseases, treatments, and patients of various ages and literacy levels, and they can be presented as computerized adaptive tests (CATs), in which items are presented to test takers based on their responses to previously presented items. This reduces the number of items needed (down to typically 3-6) to determine a patient’s score, thus minimizing patient burden.
Step 1: Model formulation
The first step in module formulation consisted of semistructured interviews with 22 complementary and alternative medicine (CAM) clinicians and clinician-researchers to gain their insights and opinions regarding important nonspecific factors in healing. We also consulted an expert on optimal healing environments, Dr. Wayne Jonas, director of the Samueli Institute.
Additionally, we conducted 6 focus groups, each consisting of 6–9 patients from racially and educationally diverse backgrounds. We recruited 2 groups from an integrative medicine clinic, 2 groups from a conventional medicine clinic, and 2 groups of community members who had participated in CAM and/or conventional medicine treatments. The focus group scripts invited participants to share their opinions, for example, on what characteristics are most helpful in a health care provider and how can patients themselves influence their own healing.
From these expert interviews and patient focus groups, sets of key words and key concepts were identified. Based upon the concepts expressed by clinicians and patients, and from the literature on placebo and nonspecific factors that influence health outcomes, we developed an initial conceptual model of domains important to healing.
Step 2: Creating the Item Pool
Using the key words and concepts identified in step 1, a research librarian conducted a comprehensive literature search for any existing instruments that measure the same or similar domains. Search strategies integrated the opinion of our CAM coinvestigators as well as instrument development experts’ opinions. The search strategies were applied to 9 scientific databases for each of the domains in the model, yielding a total of over 12,000 abstracts, with the goal of identifying and obtaining any full length article that contained existing questionnaires, or articles whose content could be used to further develop the conceptual model.
At the end of this process, all questions from 535 existing instruments were entered into a database, or initial item “pool,” of over 17,000 items.
Step 3: Binning
Our team of CAM investigators and instrument development experts formed subgroups of 2–5 persons per domain to define conceptually meaningful categories within their specific domain of the model, and sort the existing items into these categories. This was an iterative process, meaning that the categories themselves could be further refined based on information gained during the review of the specific items.
For example, the domain of Healthcare Environment Perceptions included distinct factors such as Process of Care and Physical Space. Within Physical Space, items could be sorted into “bins” such as general comfort of the waiting room, privacy of the treatment room, and sensory elements such as quietness, etc.
The research team then eliminated any items that were redundant with other items, too vague, or too narrow in focus. This phase of the project also included rewriting any items that were difficult to understand, or that contained more than one concept (e.g., “double-barreled” items). (TABLE) provides examples of problematic items and our efforts at rewriting to increase clarity and simplicity.
Step 4: Cognitive Interviews
In a cognitive interview, the patient is asked to think aloud while viewing and answering items, one-by-one. As the patient speaks, a research associate takes precise notes, and queries the patient not only on the item but also on the appropriateness of the response category. Each item was reviewed by at least 6 patients, at least one of whom had a low level of education (high school or less).
This process revealed several items that were difficult for some respondents to understand as they were intended, or remained confusing. These items were then reviewed by the principal investigator and research staff and either rewritten or eliminated. Any rewritten items were reviewed again by patients in further cognitive interviews. Thus, the cognitive interview stage resulted in refinement and reduction of the item banks.
Step 5: Statistical Calibration
Consistent with the methodology of PROMIS [34], Classical test theory (CTT) and modern psychometric methods of item response theory (IRT) and tests of Differential Item Functioning (DIF) are used to determine which items should be retained from a statistical and measurement standpoint. In order to conduct these analyses, we tested the sets of items on an internet sample of 1400 adults who had experienced CAM or conventional treatments currently or within the past year, 127 patients at the Center for Integrative Medicine at University of Pittsburgh Medical Center (UPMC), and on 131 conventional medicine patients at UPMC. The internet sample was representative of the US population in terms of age and race and was collected through YouGov (http:/www.yougov.com/), an internet polling company.
IRT methods are applied to determine whether items provide sufficient information about the underlying domain or construct. At this stage, items may be dropped if they are not informative about the subject’s level on the construct of interest. Confirmatory and Exploratory Factor Analysis was used to assess model fit.
*Note: The CAT administration does not require that all questions be answered. Many times, fewer than 6 item responses are needed to obtain a valid score. Additional short-form fixed questionnaires of 5-7 items are also available for each item bank.
References:
1. Greco CM, Glick RM, Morone NE, Schneider MJ. Addressing the “It Is Just Placebo” Pitfall in CAM: Methodology of a Project to Develop Patient-Reported Measures of Nonspecific Factors in Healing. Evid Based Complement Alternat Med. 2013;2013:613797. doi: 10.1155/2013/613797. Epub 2013 Dec 19. PMID: 24454501; PMCID: PMC3880690. https://pubmed.ncbi.nlm.nih.gov/24454501/
2. Using Surveys to Assess Patient-Centered Factors
that May Affect Responses to Chronic Pain
Treatment
Carol Greco, PhD; Lan Yu, PhD; Paul Pilkonis, PhD
https://www.pcori.org/sites/default/files/Greco095-Final-Research-Report.pdf
3. Greco CM, Yu L, Johnston KL, Dodds NE, Morone NE, Glick RM, Schneider MJ, Klem ML, McFarland CE, Lawrence S, Colditz J, Maihoefer CC, Jonas WB, Ryan ND, Pilkonis PA. Measuring nonspecific factors in treatment: item banks that assess the healthcare experience and attitudes from the patient’s perspective. Qual Life Res. 2016 Jul;25(7):1625-34. doi: 10.1007/s11136-015-1178-1. Epub 2015 Nov 12. PMID: 26563249; PMCID: PMC4865446.