Methodological Issues in Examining Cognition Across The CNS Spectrum

This session was chaired by  Richard Keefe from the Departments of Psychiatry & Behavioral Sciences and Psychology at Duke University and Steve Rao from the Shey Center for Cognitive Neuroimaging at the Cleveland Clinic.  The speakers included the following:

Michael Green, from the Department of Psychiatry and Biobehavioral Science, Geffen School of Medicine, UCLA Semel Institute, spoke on Cognitive Assessment in Schizophrenia Clinical Trials:  MATRICS and Beyond. 

Ralph Benedict, from the Department of Neurology, Jacobs Neurological Institute, University at Buffalo, State University of New York spoke on Psychometric Considerations and Consensus Standards for Cognitive Testing in MS.

David Loring, from the Department of Neurology at Emory University, spoke on Cognitive and Behavioral Endpoints as Primary Outcome Variables in Epilepsy Clinical Trials.

Kate Burdick, from The Zucker Hillside Hospital, North Shore-Long Island Jewish Medical Center, Feinstein Institute for Medical Research, Albert Einstein College of Medicine spoke on Application of Cognitive Assessment Methods for Bipolar Disorder Clinical Trials.

Additional Panelists included: Thomas Laughren, FDA; Russell Katz, FDA; and Gretchen Tucker

The purpose of this session was to discuss, in the context of clinical trial design, the methodology of assessing cognition in psychiatry and neurology.  The focus of the session was on disorders where cognitive dysfunction has not traditionally been viewed as a primary manifestation or target outcome in the context of clinical trials, specifically schizophrenia, multiple sclerosis, epilepsy and bipolar disorder.  The following primary issues were outlined in an introductory presentation by the Session Chairs, and addressed sequentially by each speaker for each disorder.  The session concluded with an extended interactive discussion period involving the chairs, panelists, and audience.

1.    The current perspective regarding the role of cognition in the disorder, including the profile of cognitive impairments, the common clinical perspective regarding the role of cognition in the disorder, and the extent to which cognitive deficits contribute to the overall functional disability and quality of life decrement associated with the illness

2.    The types of assessments that should be completed to assess the relevant cognitive impairments in the disorder, including the extent to which a typical assessment battery overlaps with that from other disorders, and whether a MATRICS-like initiative is needed or whether the work initiated by MATRICS for identifying cognitive domains is applicable

3.    The challenges and proposed solutions for assessing cognition accurately in clinical trials, including existing trial designs, any general or disease-specific regulatory hurdles to getting indications for treatment of cognition and identification of any particular areas that need development, log jams, logical inconsistencies

The session began with a presentation from Dr. Green on schizophrenia. Dr. Green reviewed the extensive progress associated with the MATRICS Initiative.  He described the process by which several hundred experts from academia, NIMH, FDA, and industry convened to develop a battery of cognitive tests that met established criteria for inclusion. The battery, named the MATRICS Consensus Cognitive Battery (MCCB), was required to include a set of tests that are valid measures of cognition in seven domains of cognitive function determined to be most relevant to schizophrenia clinical trials.  The domains included attention, working memory, verbal learning and memory, visual learning and memory, processing speed, reasoning and problem solving, and social cognition.  The tests were required to demonstrate high test-retest reliability, high utility as a repeated measure, an empirical relationship to functional outcome, and tolerability and practicality.  Dr. Green noted that FDA endorses the use of the MCCB for clinical trials in schizophrenia, and that the MCCB is available in 7 different languages, including: English, German, Russian, Chinese, Hindi, and Spanish (for Spain and a version for Central and South American).  Other translations are underway. The MCCB is currently in use in over 30 ongoing or completed trials.  Dr. Green also described that the  MATRICS process has led to other advanced initiatives, including efforts to identify satisfactory functional capacity measures to be used as co-primary measures in schizophrenia trials (MATRICS-CT), efforts to identify cognitive neuroscience tasks that can be used in early phase schizophrenia cognition research (CNTRICS), the establishment of a network of sites available for schizophrenia cognition and biomarker trials (TENETS), and an effort to develop a negative symptom scale for use in schizophrenia clinical trials.

In the subsequent presentations, the disorders discussed - MS, epilepsy, and bipolar disorder – traditionally use other primary outcomes, and the assessment of cognition is emerging or novel.  Each presenter identified common and unique (i.e., disorder specific) methodological issues, validity, relationship to other outcome measures, and laid out concrete examples.  The audience was thus able to see how fundamental principles of cognitive assessment must be altered to meet the needs of that specific disorder. 

Dr. Loring described the current state of research on cognition in epilepsy.  He noted that traditional treatments for epilepsy can have deleterious effects on cognitive performance. Thus, the inclusion of cognitive measures in epilepsy clinical trials needs to be sensitive to potential worsening as well as potential improvement.  Patients with epilepsy are impaired on a broad range of cognitive domains, with z-scores ranging between 1 – 2 standard deviations from healthy controls.  Since current adjunctive treatments in epilepsy focus on reduction of seizures in partially responding patients, it is of great concern that additional medications and their interactions may reduce seizures at the cost of cognitive worsening. There is no standard battery for assessing cognition in epilepsy, which limits comparisons across studies.  Dr. Loring noted that future trials in epilepsy clinical research should assess cognition independent of standard efficacy measures, include new onset patients, include a control group of monotherapy patients, use reliable change indices as cognitive endpoints, and assesses comorbid psychiatric symptoms such as mood effects. 

Dr. Benedict gave a clear and relevant overview of the neurobiology of multiple sclerosis and the profile of cognitive deficits reported in MS patients in general.  He described various batteries that are currently being used in MS research.  One of the batteries emerged from an expert panel composed of neuropsychologists and psychologists from the United States, Canada, United Kingdom, and Australia in April, 2001. The objectives of that panel were to: (a) propose a minimal neuropsychological (NP) examination for clinical monitoring of MS patients and research, and (b) identify strategies for improving NP assessment of MS patients. A 90-minute NP battery, the Minimal Assessment of Cognitive Function in MS (MACFIMS), resulted from this work. The MACFIMS is composed of seven neuropsychological tests, covering five cognitive domains commonly impaired in MS (processing speed/working memory, learning and memory, executive function, visual-spatial processing, and word retrieval). It is supplemented by a measure of estimated premorbid cognitive ability.  Dr. Benedict noted that subsequent work with the MACFIMS has suggested that over 50% of patients with MS demonstrate clinically significant impairment on 2 or more of the tests in the battery.  Dr. Benedict described that very few trials emphasize cognitive outcomes in MS because medications are intended to treat immunological activity at the site of brain lesions, and there is a general assumption that treatments that improve key MS measures such as reduced relapse, reduced brain lesions, and improved physical functioning will also be good for cognition.  However, Dr. Benedict noted that very recent work has suggested that some key cognitive measures demonstrate modest improvement compared to placebo.  Dr. Benedict also acknowledged that cognition as a treatment target in MS is at the very beginning stages. Several issues remain, such as the difficult balance between proper medication exclusion, and successful enrollment of patients; generalizability of results across different type of MS, such as relapsing-remitting versus stable progressive disease, the identification of which patients have cognitive impairments that are sufficient to warrant treatment; the co-occurrence of psychiatry factors such as depression which has a 90% lifetime prevalence in MS; the choice of the best cognitive outcome measure; and the challenges associated with assessing cognition in a typical clinical setting. 

Dr. Burdick described how cognitive deficits in bipolar disorder are not as severe as in schizophrenia, and are not present prior to the onset of illness, yet are still a significant component of the clinical picture, and that while they are associated with clinical symptoms such as depression, they remain even in patients who's manic or depressive symptoms have remitted.  She noted that cognitive impairment may be particularly severe in psychotic bipolar patients.  Like in schizophrenia, Dr. Burdick described that the cognitive impairments in bipolar disorder are associated with key aspects of daily life such as social and occupational disability.  She argued that cognition is a legitimate treatment target in bipolar disorder, and encouraged clinical trials to address this unmet need.  She raised the question as to whether a MATRICS-like process should be initiated for bipolar disorder, and suggested that since many of the MATRICS Consensus Cognitive Battery domains are also impaired in bipolar patients, that a “MATRICS-plus” battery could be developed with the addition of tests that measure cognitive deficits repeatedly reported in affective disorders such as emotion recognition and affect regulation.  Dr. Burdick also suggested since not all bipolar patients demonstrate cognitive impairment that is severe enough to warrant treatment, the identification of homogeneous samples or bipolar ‘subtypes’ that may optimally benefit from cognitive enhancement trials will be critical

The session concluded with a 90 minute structured question and answer period for the panelists and audience, in which a series of questions were addressed:

• How can we determine if changes on a cognitive test battery translate into changes in activities of daily living?
• Can a common battery of cognitive tests be designed to measure outcomes across a range of disorders?
• How comprehensive should the battery be (i.e., length of administration)?
• What are the relative advantages/disadvantages of computerized tests relative to paper-and-pencil tests?
• What are the criteria for validating a cognitive test battery for use in clinical trials?
• What is an acceptable level of test-retest reliability?
• Does sensitivity in discriminating groups translate into sensitivity in detecting change?
• How do we address practice effects beyond creating alternate equivalent forms?
• Should trials use yoked healthy control groups? Are practice effects in healthy individuals' equivalent to those in patient populations?
• Should we develop a normative database based on the natural history of cognitive dysfunction in a particular disorder, which can be compared to treated and placebo groups in a randomized trial?
• Can statistical methods (e.g., reliable change scores) adequately adjust for practice effects?
• How should we address basal and ceiling effects, especially in progressive disorders?
• How do we assess patient compliance?
• How can we assess whether the patient adequately understands test administration instructions?
• Should cognitive testing be performed by individuals without formal neuropsychological training (nurses, social workers)?
• What are the standards for rater training and data quality review?
• Can we design a cognitive test battery that is valid in multiple languages?  Should we develop a nonverbal cognitive test battery?
• Can we develop a test battery that is not contaminated by non-cognitive sensorimotor deficits?
• Should clinical trials only include patients with evidence of cognitive dysfunction at baseline?

Key issues during the question and answer period regarded the following:

Co-primary measures. While FDA has made clear that face-valid measures of functioning or functional capacity must also demonstrate improvement for a cognition indication in dementia and schizophrenia, the role of these measures in clinical trials for disorders such as epilepsy, MS, and bipolar disorder is not clear.  In addition, since far less research has been conducted on these measures, their capacity to demonstrate improvement during the course of a clinical trial is not known. This is an area that requires substantial additional research.

General versus specific outcome measures. While genetic advances offer the promise that future treatments will be individualized, very few treatments have developed a sufficient data base to implement this strategy.  With regard to cognition, it appears as though general measures are more reliable and most strongly associated with functional outcomes, yet more specific measures may reflect more basic neurobiological processes.  FDA, as in MATRICS, has encouraged composite measures yet is open to data suggesting that a treatment targets a specific outcome, however it will need to be demonstrated that improvement is limited to these domains and that other cognitive domains do not worsen with treatment.

Cross-diagnosis core battery. Since many tests are used in the assessment of cognition in various disorders, it may be possible to develop a core set of tests that would be used in all cognition clinical trials. This effort would require collaboration among researchers who do not usually collaborate and who may not know about each others’ work. 

Computerized versus paper-and-pencil tests. While computerized tests have appeal for their potential standardized administration and ease of implementation, there was concern about the ability of some patient populations to interact with computerized tests and whether valid data would be collected in the context of clinical trials. It was the view of the panelists that this question needs to be addressed empirically, and data are sorely needed.

Cognitive side effects. While the emphasis for dementia and schizophrenia clinical trials has been on improving cognition, it was noted that in disorders such as epilepsy, cognitive endpoints are useful additions to assessment schedules in order to determine whether a treatment worsens cognition.