A. George Awad, M.B., B.Ch., Ph.D., FRCP
Nina R. Schooler, Ph.D.

--

Patient Reported Outcomes:  New Approaches Using Item Banks and Computerized Adaptive Assessments

Chairpersons:
A.George Awad, University of Toronto
Nina Schooler, Georgetown University & State University of New York

Speakers in the order of presentation:
William Riley, NIMH
David Victorson, North Western University, Evanston
Joel Weissman, MGH/Harvard Institute of Health Policy
Dennis Revicki, United BioSource, Maryland
Laurie Burke, FDA, Maryland
Robert Morlock, Biogen Idec

In his introduction, A. George Awad emphasized the important role of health outcomes in research and clinical practice. Traditionally, health outcomes have been captured using scales that suffer from a variety of shortcomings, including: questionable psychometric properties, lack of an underlying conceptual or theoretical construct, lack of sensitivity to small changes, lengthiness, cumbersomeness, poorly suited to address the realities of life for psychiatric patients and failure to capture the patients' perspective on their illness. Further, the lack of standardized scoring metrics makes it difficult to compare data across various studies. Looking forward, new methodologies based on contributions from item response theory (IRT), advances in computer technology that allow for computer-based assessment using computerized adaptive testing (CAT) may provide new approaches for more precise and individually tailored assessments of health status. Yet, these technological advances are not without their limitations. This session focused on these issues. Experts introduced new methodologies and discussed how they might be applied to clinical research in psychiatry and neurology. In addition they critically assessed the limitations of these methodologies and outlined a road map for advancing their use in the future.

Individual presentations are summarized below:

William Riley, National Institute of Mental Health: Patient-Reported Outcomes Measurement Information Systems (PROMIS) An NIH Roadmap Initiative  The major themes of this NIH roadmap initiative were outlined. Prominent among these were: Re-engineering the clinical research enterprise by implementing new paradigms in how clinical research information is collected, used and reported. The plan also incorporates advances in information technology, psychometrics and qualitative, cognitive and health survey research. Important in this initiative is the development of new partnerships of research with organized patient communities, community-based health care providers, industry and academic researchers. The broad objectives of PROMIS include the development and testing of a large item-banks measuring patient reported outcomes (PROs), and the creation of a computerized adaptive testing system that will allow for efficient and psychometrically robust assessment of PROs for a wide range of chronic disease-outcome research. It will also provide a publicly available database that can be periodically added to and modified. It will allow clinical researchers access to a common item repository and to computerized adaptive testing. The advantages of item-response theory (IRT) and computerized adaptive testing (CAT) include improved efficiency, and improved measurement precision. Item banks will be developed that will provide instruments with psychometric properties at the item level that will permit flexible administration and ongoing item refinement. Further information about PROMIS can be obtained at www.nihpromis.org

David Victorson: PROMIS AND NEURO-qol: New NIH Initiatives To Standardized Patient-Reported Outcome Assessment In Clinical Research  In his talk Dr. Victorson provided details about "PROMIS-1". Among the highlights of PROMIS-1 are: a consensus patient reported outcomes framework, a qualitative study of more than a thousand people, a quantitative study of more than twenty-thousand people, nine item banks available to collaborators, website and functional assessment centre, and an upcoming clinical validation study in chronic disease populations. This should lead to a clear path/agenda for PROMIS-2. Among the initial target domains identified: Physical function, pain, fatigue, social function and emotional distress. One of the developments is the creation of large "item banks", collections of items measuring a single domain which provide the basis for tailored/adaptive testing. The advantages of computerized adaptive testing (CAT), include: ability to select questions based on the response to previous questions, the desired level of precision can be obtained using a minimum number of questions, as well as, estimation of a person's standing in the domain without asking questions that cover the full range of the domain, e.g. depressive symptoms. In summary, calibrated item banks can be used: To create a standard static instrument, construct short forms, enable CAT, select items based on unique content interests and formulate customize short form or full-length instruments. Importantly, it can produce scores on a common metric.

To make PROMIS and, related tools applicable to neurological conditions, the NEURO-qol project was introduced. Its objectives include development of a core set of questions that cut across chronic neurological disorders, development of supplemental questions that address additional concerns of specific diseases or groups of patients, creation of a publicly available, adaptable and sustainable system allowing clinical researchers access to a common item repository and CAT. Among the target disorders for field-testing are: stroke, multiple sclerosis, Parkinson's disease, epilepsy, neuromuscular disorders. Pediatric conditions that will be targeted subsequently include epilepsy and muscular dystrophies. The overlap between NEURO-qol and PROMIS has been examined. NEURO-qol adds a number of domains not included in PROMIS: Stigma, perceived cognitive deficits, applied cognition, personality and behavioural change.

Joel Weissman: Patient Reported Outcomes and the 'Experiences of Participants In Clinical trials' (EPIC) Study  The observation that most studies of participants in clinical trials are from single trials or single diseases has provided the background for the EPIC study. Few, if any, studies have explored the experiences of participants before, during and post trial. In addition, there is particular interest in the experiences of the uninsured compared with insured participants. The EPIC study is probably one of the first applied studies derived from the PROMIS initiative. The PROMIS short form for the EPIC study included the domains of general health (one item), functional (six items), emotional (twelve items), and pain (seven items). The study population included 8% currently uninsured, 19% uninsured in the last twelve months, 26% did not get a prescription filled in the last twelve months due to cost, 21% skipped a test or treatment in the last twelve months due to cost, 68% had enrolled in more than one medical research study. When assessing self-reported changes in health status during their most recently completed clinical trial (n=67), the majority of subjects felt better (n=47). On the other hand, their self-reported change in health status after their most recently completed trial (n=67) indicated that seventeen subjects felt worse, twenty felt the same, and only seven felt better. When evaluating trials as a source of medical care, most participants sought symptomatic relief from clinical trials. Many participants reported that their health improved during the trial and most participants thought it was appropriate to compare the quality of trial care to regular care.

Dennis Revicki: Application of New Patient-Reported Outcome Measures And Methods In Psychiatry And Neurology Clinical Trials  Dr. Revicki provided a critical appraisal of the new methodologies and their application to trials in psychiatry and neurology, examining challenges to application of IRT and CAT-based measures. For clinical trial researchers, a number of barriers has been identified and include: Lack of understanding and familiarity with IRT-based methods, skepticism about patient-reported outcomes for some study populations (e.g., schizophrenia, mania) and practical issues and feasibility. CAT requires computer availability and administration that may not be practical in many research settings and may add costs to these trials. However, a sufficient trade-off can be achieved when increased responsiveness and ability to detect clinically meaningful changes is considered. Challenges for regulatory agencies include: the limited experience with IRT and CAT methods; challenges associated with demonstrating content/face validity of subsets of questions drawn from item banks; and cognitive discontinuities associated with IRT-based and tailored health outcome measures. A question was raised about whether acceptable short, disease specific measures can be developed from large, generic item domain banks? A challenge for industry is the fact that few sponsors will select IRT-based measures if they are not acceptable to regulatory agencies.

Among the challenges for clinicians when using these PRO measures is that they often do not care about precise assessment of single domains but, instead, need good tools for diagnosis and evaluation. Therefore, the measures need to be practical, easily applied and understandable by clinicians. An important question here is whether IRT-based measures should be developed with logical branching networks to assist clinicians in efficiently evaluating patients' status for treatment decision-making. It is clear that acceptability to study participants will be critical for successful application of CAT measures.

Participants sometimes experience challenges when using computers. Additional research among patients with severe psychiatric disorders is particularly needful. On the other hand, potential advantages for study participants include privacy and reduced respondent burden. Interim solutions for sponsors, clinical researchers and regulatory agencies were offered. These included: continued dialogue on the critical issues, starting with static health measurements that are based on IRT methods and tailored to specific disease indications (interim stage), accumulation of experience and evaluation of feasibility and responsiveness of CAT in clinical trials as secondary or exploratory endpoints.

Laurie Burke, Food and Drug Administration: FDA Perspective: Use of PROs in Products Labels - Objectives and Hurdles To New Methodologies   Dr. Burke outlined the FDA mission with respect to PROs and provided ideas about streamlining the "critical path" initiative. The challenges for the critical path initiative include: New therapeutics with novel mechanisms often target unique pathways or even diseases that may demand novel assessments methods. As new endpoints emerge "validation" requires that we understand both how these endpoints perform, and what they mean clinically. Among the important considerations in interpreting the clinical meaning of an endpoint (i.e., assessing risk-benefit) is that of assessing the level of change that is important to the patient. The arguments for the use of PROs in clinical trials and their key role in drug development include: the fact that some treatment effects are known only to the patient and the desire to understand the patient's perspective about treatment effectiveness. Only the patient can provide this unique perspective beyond clinically-based observational measures. Recent FDA PRO guidance was introduced to explain how FDA evaluates the value of PRO instruments in measuring and characterizing benefit of medical product treatment as perceived by the patient.

The adequacy of a PRO instrument as a measure to support medical product labeling claims depends on its developmental history and its documented measurement properties. The most critical consideration during the FDA's review of endpoints used to measure treatment effects is the adequacy and availability of a clearly established concept of measurement. Other clinical trial design issues include requirements for study objectives that correspond to the endpoint assessments, control of bias, quantification of meaningful effect sizes, calculation of study sample size and determination of relationships among all trial endpoints.

Considering IRT, PRO item banks and CAT, the FDA's experience with IRT is limited to its use as an alternative to factor analysis during instrument development. Additionally, there is no experience reviewing IRT/CAT to measure and compare medical product treatment effects in the clinical trial setting. It is not clear what specific regulatory review issues will emerge when these methods are used. Amongst specific concerns when using ePROs is that the sponsor should insure that FDA regulatory requirements are met for both the sponsor and the investigator in terms of record keeping, maintenance and access to these records. The FDA encourages demonstrations of the application and added value of item banks, IRT-based instruments and CAT in the clinical trial setting.

Robert Morlock: Item Banks And Computerized Adaptive Assessment: An Industry Perspective  A number of questions were raised in this presentation: Will the pharmaceutical industry benefit from item banks and CAT? What hurdles need to be overcome for item bank use in clinical trials? And, how should industry support the use of item banks and CAT assessment?

In evaluating the IRT/CAT from the industry perspective it is clear that there are benefits, but a number of barriers remain. Among the benefits of this methodology are reduced 'noise," increase in sensitivity, decreased patient and staff burden and provision of broader comparisons across different trials, as well as the potential to reduce trial size and lower costs. On the other hand, these methodologies raise numerous operational and administrative hurdles, including: communication of complex topics, clarification of ownership, translations and fees, and, importantly, the acceptance by regulators. An additional question is: how should industry support the use of item banks and CAT? Answers to these questins will require industry sponsorship of development of new instruments and translation of existing tools, development of global registries and provision of educational grants.

Concluding remarks and a summary of the session were provided by Nina Schooler, who emphasized that PRO measures have become an increasingly important aspect of CNS clinical trials as healthcare providers and payers for healthcare are demanding them. PROs are particularly relevant, because they prove the most direct measure of the consumer's overall evaluation of the treatment being investigated. Further, for certain treatments such as pain, mood, etc., key information on outcome is uniquely available from the patients' report. To meet the needs of stakeholders in healthcare delivery, investigators need to take advantage of methodological advances in outcome as quickly as possible. Regulators are developing guidance on how PROs should be used. PROMIS, the recently developed NIH roadmap initiative has added impetus in advancing new methodologies as presented and critically evaluated by all the speakers. All the speakers were excellent and to be thanked for bringing forward complex methodologies based on modern information and measurement theories. It is clear that the information presented in this session has the potential not only to open new methodological grounds, but also can prove important in eliciting more precise, valid and less cumbersome patient reported outcomes.

It is hoped that by continuously refining these new methodologies, and testing their application in psychiatry and neurology clinical trials, the FDA and other regulatory agencies are moving towards their acceptance and use during conduct of registration-quality clinical trials.