Original Research

Newly self-administered two-step tool for screening cognitive function in an ageing Chinese population: an exploratory cross-sectional study

Abstract

Background Early screening of cognitive function is critical to dementia treatment and care. However, traditional tests require face-to-face administration and are often limited by implementation costs and biases.

Aims This study aimed to assess whether the Thoven Cognitive Self-Assessment (TCSA), a novel, innovative two-step touchscreen-based cognition assessment tool, could identify early cognitive impairment due to dementia in older adults.

Methods The TCSA was administered to 61 healthy controls (HCs), 46 participants with mild cognitive impairment (MCI) and 44 participants diagnosed with dementia recruited from Shanghai. Two outcome measures were generated from the TCSA test: the TCSAprimary task score and the TCSAsecondary task score.

Results The total average scores in the control group for the TCSAprimary task and TCSAsecondary task were significantly higher than those in the MCI and dementia groups (TCSAprimary task: HCs vs MCI group vs dementia group, 8.58±1.76 vs 5.40±2.67 vs 2.74±2.11, F=75.40, p<0.001; TCSAsecondary task: HCs vs MCI group vs dementia group, 23.02±3.31 vs 17.95±4.93 vs 11.93±5.50, F=76.46, p<0.001). Moreover, receiver operating characteristic analysis showed that a score below 7.5 for the TCSAprimary task and a score below 22.5 for the TCSAsecondary task were indicators of MCI.

Conclusions The TCSA appears to be efficacious for the detection of cognitive impairment in older adults. It demonstrates the potential for large-scale cognition screening in community service settings.

What is already known on this topic

  • The widespread incidence and prevalence of dementia have prompted an urgent call for accessible and affordable early detection assessments.

What this study adds

  • The Thoven Cognitive Self-Assessment (TCSA) tool is the first two-step progressive touchscreen-based cognitive assessment tool and shows good accuracy in detecting both clinically diagnosed mild cognitive impairment (MCI) and dementia.

How this study might affect research, practice or policy

  • Digital cognitive screen instrument has the potential for clinical use and large-scale cognition screening in community service settings, especially in China, one of the world's fastest-aging countries.

Introduction

By 2050, it is expected that the number of people with Alzheimer’s disease (AD) and related dementia (ADRD) may exceed 131 million worldwide.1 AD’s incidence rate and morbidity have steadily increased to become the fifth leading cause of death among urban and rural residents in China.2 Delivering clinical treatment and care to this population has become a public health concern. Unfortunately, no effective medical treatments are available to cure ADRD at this time.3 4 However, early screening and detection can help promote lifestyle changes, which can reduce the risks of cognitive decline.5 6 Therefore, a reliable screening tool to detect mild cognitive impairment (MCI), a stage where interventions can be delivered to postpone disease onset, is needed.

Currently, several tests are used for the clinical assessment of MCI, such as positron emission tomography imaging, structural magnetic resonance imaging (MRI) and genetic biomarkers in the peripheral blood and cerebrospinal fluid.5 7 8 However, these clinical tests are limited by their high cost, inaccessibility and invasiveness. For now, paper-based neurocognitive screening scales remain the most commonly used tools.

Established cognition screening scales such as the Mini-Mental State Examination (MMSE),9 the Montreal Cognitive Assessment (MoCA),10 the Mini-Cog Test,11 the General Practitioner Assessment of Cognition (GPCOG)12 and the Brief Alzheimer Screen (BAS)13 are commonly used to screen for dementia in the community. Challenges with the application of these screening tools include the demand for qualified assessors and other implementation resources. Furthermore, the accuracy of results in many of these screening tools is affected by the cultural background of the participants, especially in non-English-speaking countries.14 One plausible way to overcome these limitations is the utilisation of a self-administered online cognitive test.

Several computerised cognitive tests have been applied to older adults over the past few years. Two examples, the Computer-Administered Neuropsychological Screen for Mild Cognitive Impairment (CANS-MCI) and the Automatic Neuropsychological Assessment Metrics (ANAM) battery, were designed as self-administered screening tests for MCI.15 16 However, the longer administration time makes them less suitable for screening compared with the MMSE, although their self-evaluated formats are more suitable for large-scale assessment and research.17 The MemTrax, another technology-based assessment tool for MCI, has improved its administration time. However, it only captures episodic memory, which makes it insufficient to identify subtle impairments at the very early stage of cognitive dysfunction.18 As a result, a quick and comprehensive cognitive online test to identify individuals with cognitive deficits in the earliest stages of the disease has not yet been developed.

This gap in the literature led us to design a time-saving and comprehensive cognition assessment tool called the Thoven Cognitive Self-Assessment (TCSA). The objective of this study was to show the details of this tool as well as the validity and reliability of the TCSA.

Methods

Study population

Subjects were recruited between November 2020 and February 2021 from outpatients at the Shanghai Mental Health Center and in the community in Shanghai, China. A total of 164 participants were assessed: 13 were excluded, and 151 individuals were included in the final analysis. Participants were then placed in different groups: an MCI group (n=46), a dementia group (n=44) and healthy controls (HCs) (n=61). Participants were excluded if they withdrew consent (n=4), had depression (n=3) or could not understand the instructions or were unable to complete the practice run for the assessment (n=6). A small sample of participants (n=5 for each group) were chosen by simple randomisation to complete the TCSA tasks 3 weeks later for test–retest reliability analysis. The remaining participants did not take any other tests. The flowchart for this study is shown in figure 1. HCs were selected by convenience sampling from the community. Inclusion criteria for all participants included the following: (1) men and women at least 65 years old, (2) the ability to speak Chinese, and (3) the ability to understand verbal and written directions because the touchscreen-based cognitive assessment task involved visual-auditory functions. Exclusion criteria included the following: (1) psychiatric disorders other than dementia or cognitive decline due to traumatic, substance use or medical causes; (2) severe visual deficits (such as droopy eyelid, cataracts, detached retinas, glaucoma, extremely small pupils, etc); and (3) auditory impairments that prevented participants from hearing instructions clearly.

Figure 1
Figure 1

Flowchart of study enrolment. ADL, Activities of Daily Living; CDR, Clinical Dementia Rating scale; C-MoCA, Chinese Version of Montreal Cognitive Assessment; HCs, healthy controls; MCI, mild cognitive impairment; MMSE, Mini-Mental State Examination; TCSA, Thoven Cognitive Self-Assessment.

Diagnoses of dementia (including Alzheimer’s disease, vascular dementia, and mixed dementia) made by geriatric psychiatry directors were based on the Diagnostic and Statistical Manual of Mental Disorders–Fifth edition (DSM-5)19 criteria; MCI was diagnosed by geriatric psychiatrists based on the criteria given by Petersen and colleagues,20 including (1) memory concerns, usually stated by the patient, preferably corroborated by an informant; (2) objective memory impairment for age; (3) essentially normal general cognitive function as judged by the physician; (4) normal activities of daily living as judged by the physician; and (5) no dementia. Specific neuropsychological cut-off scores were not used to diagnose MCI. The assessment included comprehensive history, physical examination, functional assessment, behavioural scores and depression screening (Geriatric Depression Scale (GDS) score greater than 7). Participants were determined to be HCs if they demonstrated no evidence of cognitive decline as compared with their baseline cognitive functions on a clinical interview and assessment. A detailed medical, social and family history was obtained from each participant.

Measures

All participants completed a paper-based cognitive assessment before the touchscreen-based cognitive assessments were administered.

Paper-based cognitive assessments

All participants completed a series of tests including the following subtests: the Chinese Version of Montreal Cognitive Assessment (C-MoCA), the Mini-Mental State Examination (MMSE), the Activities of Daily Living (ADL, maximum score of 56)21 and the Clinical Dementia Rating scale (CDR).22

The MMSE and C-MoCA were used to determine global cognition. Both of them are scored within a range of 0–30 points (higher scores indicating better function). The ADL consists of 14 items assessing the level of physical functioning. Scores ranged from 14 to 56, with higher scores representing a higher level of independence. The CDR assesses the severity of dementia and was used for our study. CDR classifies scores as the following: 0 (no cognitive decline), 0.5 (questionable), 1 (mild), 2 (moderate) and 3 (severe cognitive decline).

Thoven Cognitive Self-Assessment (TCSA)

The TCSA was developed using HTML5 technology and adapted to the WeChat mobile application (Shenzhen Tencent Computer Systems, Shenzhen, Guangdong, China) for self-administration. Data were stored on a cloud server (Ali Cloud) located in China and licensed from Thoven (Thoven Technology, Shanghai, China).

In the development of TCSA, an item pool of tests was constructed from the literature, previous cognitive assessment scales and by consulting experts in the field of geriatric psychiatry and neurology. The content validity was conducted through expert judgement. Two independent reviews were carried out by a panel of three experts from the fields of psychology, geriatric psychiatry and neurology to select tests that were appropriate, accurate and interpretable. The acceptance, rejection or modification of an item was based on majority opinion. A flow chart of the selection process and included modules is presented in the online supplemental Figure 1.

The TCSA test consisted of two parts: a primary screening section and a secondary screening section. The primary screening section was completed in 1–3 min with a total score of 11 points. The total score of the secondary screening section is 30, and most participants finished it in about 9 min (ranging from 8 to 10 min). Five major assessment modules covering nine tasks were developed to assess cognition that included the following: (1) working and episodic memory tests with the visual, audio and audiovisual examination, which included episodic image recognition, turnover cards and memory gas station, as well as an episodic memory recall test called ‘episodic image recall’; (2) attention and reaction-time tests including Musical Metronome and Shult Grid; (3) examinations of calculation; (4) a language test called ‘scratch-off ticket’; and (5) orientation to time and place tests. A description of all tasks and actions with the outcomes, measures and main cognitive functions assessed through these actions is shown in the online supplemental Table 1.

To begin, participants touched the Start button on the screen to log into the system. Then, they were required to provide information, including phone number, name, birth date, gender and years of education. Before the actual TCSA test was administered, the test was explained in detail and a test introduction video was given to participants. This included not only the test itself but also the instruction and count-down pages to let the participant get accustomed to the layout of the site and the initial actions needed before the start of the test.

Statistical analysis

Statistical analysis was performed using SPSS V.22. Normal distributed variables were presented as mean and SD. Non-normal distributed variables were presented as median and IQR (25% percentile–75% percentile). Sex was presented as number (n) and percentage (%). We used the analysis of variance (ANOVA), Kruskal-Wallis test and χ2 test to make demographic comparisons between the three groups. To examine the performance differences in cognitive tests, neuropsychological tests and TCSA scores among the three groups, analysis of covariance was used while adjusting for demographic variables that differed statistically among the three groups.

Partial correlation analysis was used to assess correlation between the C-MoCA scores and TCSA scores, while controlling for age and years of education. Pairwise comparison of receiver operating characteristic (ROC) curves tested the equality of two or more ROC areas obtained from applying two or more test modalities to the same subject. It was calculated to compare the area under curve (AUC) of the two screening tests (TCSA vs MMSE).

ROC curves were used to obtain the sensitivity and specificity of the TCSA to determine whether it can be a useful instrument for differentiating (1) patients with MCI from HCs, (2) patients with MCI from those with dementia and (3) cognitively impaired subjects (patients with MCI and dementia) from HCs. The cut-off point was identified as the value corresponding to the maximum value of Youden’s index, calculated as (sensitivity+specificity–1). The AUC with 95% confidence interval (CI) for the TCSA score was used to measure its accuracy to differentiate patients with and without cognitive impairment.

Cronbach’s α coefficient reliabilities were calculated to analyse the internal consistency of the scale items. Cronbach’s alpha values of 0.6 or higher were considered acceptable.23 Test–retest reliability was demonstrated by measuring the TCSA on two separate occasions (3 weeks apart) for a small sample of subjects (n=15) chosen by simple randomisation.5 It was evaluated using partial correlation analysis. Statistical significance was set at p<0.05.

Results

Demographic and clinical characteristics

There was no significant difference in age or gender between the HC group, the MCI group and the dementia group, but an educational level difference was found between the three groups (F=24.98, p<0.001). The control group had a significantly higher mean (SD) MMSE scores (25.35 (5.95) vs 21.51 (5.61) vs 14.90 (6.96), F=23.51, p<0.001) and C-MoCA scores (26.09 (2.81) vs 18.84 (4.97) vs 10.74 (6.04), F=136.90, p<0.001) than the MCI group and the dementia group. The control group also showed significantly lower ADL and CDR scores (table 1).

Table 1
|
Demographics and clinical characteristics of the participants

Discriminant validity: between-group differences on test parameters

The HC group showed significantly higher mean primary screening task scores (8.58 (1.76) vs 5.40 (2.67) vs 2.74 (2.11), F=75.40, p<0.001) and secondary screening task scores (23.02 (3.31) vs 17.95 (4.93) vs 11.93 (5.50), F=76.46, p<0.001) than the MCI or dementia groups, as shown in figure 2.

Figure 2
Figure 2

Thoven Cognitive Self-Assessment test total scores for MCI, dementia and HC groups. Data are expressed as mean (SD). Significance levels are indicated on the upper portion of the figure (*p<0.05, **p<0.01). MCI, mild cognitive impairment; HCs, healthy controls.

As shown in table 2, the one-way ANOVA tests yielded a significant difference in the score for different cognitive domains among the three groups. There was no significant difference found for working memory (Z=3.11, p=0.211) among the groups. TCSA was found to be more sensitive than the MMSE in discriminating MCI from HC, as reflected by the AUC scores. The AUC for the TCSAprimary task was found to be superior to the MMSE, in differentiating MCI from HC, irrespective of the educational level (over or under 12 years) of subjects.24 The TCSAsecondary task performance over the MMSE was more evident for older adults who had over 12 years of formal education, with an AUC of 0.759 (95% CI: 0.56 to 0.96) vs 0.685 (95% CI: 0.46 to 0.91).

Table 2
|
TCSA subtests: mean scores by diagnosis, and AUC scores for MMSE and TCSA for differentiating HC from MCI by education

Internal consistency was assessed using the Cronbach’s α coefficient. The Cronbach’s α of 0.88 suggests relatively high internal consistency.

Concurrent validity and test–retest reliability

Concurrent validity was evaluated by presenting the partial correlation coefficient with age and years of education as the control factors between TCSA subtests and the C-MoCA scores. Memory and language factors were moderately correlated with scores from paper-based tests which ranged from 0.41 to 0.49 (p<0.001). The factors that measured attention and orientation were strongly correlated with the C-MoCA test results (r=0.74 and 0.87 respectively, p<0.001). The correlation coefficient between TCSA and C-MoCA total scores was 0.87 (p<0.001) (online supplemental Table 2).

Test–retest reliability was demonstrated by measuring the total scores of the TCSA primary and secondary tests on two separate occasions among a small sample of subjects (five dementia patients, five MCI patients and five HCs) 3 weeks apart. Pearson’s correlation coefficient showed good test–retest correlation for primary (r=0.85) and secondary tasks (r=0.88).

Discriminant validity: ROC

The results of the ROC analyses, which assessed the ability of TCSA to discriminate between HC and MCI groups, and between MCI and dementia groups, are presented in figure 3.

Figure 3
Figure 3

ROC curves for the TCSA scores. (A) Receiver operating characteristic curve for the cognitive screening tool to discriminate between the (1) MCI and HC groups, (2) MCI and dementia groups, and (3) cognitive impairment (MCI+dementia) and HC groups through the primary screening total score. (B) Receiver operating characteristic curve for the cognitive screening tool to discriminate between the (1) MCI and HC groups, (2) MCI and dementia groups, (3) cognitive impairment (MCI+dementia) and HC groups through the secondary screening total score. HC, healthy control; MCI, mild cognitive impairment; ROC, receiver operating characteristic; TCSA, Thoven Cognitive Self-Assessment.

For the primary task, to distinguish MCI participants from healthy older adults, the area under the curve (AUC) was found to be 0.813 (95% CI: 0.71 to 0.92). A cut-off value of TCSAprimary task of 7.5 (a phonemic advantage) indicated 71.4% specificity, 82.5% sensitivity, 71.1% positive predictive values (PPV) and 82.4% negative predictive values (NPV). To distinguish MCI subjects from patients with dementia, the AUC was found to be 0.873 (95% CI: 0.79 to 0.95). An optimal statistical cut-off of TCSAprimary task was achieved at 3.5 (85.7% sensitivity, 73.2% specificity, 73.3% PPV, 83.8% NPV). To distinguish cognitive impairment participants (MCI+dementia) from HCs, the AUC was found to be 0.937 (95% CI: 0.90 to 0.98). An optimal statistical TCSAprimary task cut-off score was achieved at 4.5 (85.3% sensitivity, 85.4% specificity, 83.1% PPV, 88.4% NPV) as shown in figure 3A.

For the secondary task scores, to distinguish MCI participants from healthy older adults, the AUC was found to be 0.809 (95% CI: 0.70 to 0.91). An optimal statistical cut-off value of TCSAsecondary task was achieved at 22.5 (75.0% sensitivity, 71.4% specificity, 72.0% PPV, 75.9% NPV). To distinguish MCI subjects from patients with dementia, the AUC was found to be 0.878 (95% CI: 0.80 to 0.96). An ideal statistical TCSAsecondary task cut-off was achieved at 14.5 (89.3% sensitivity, 63.4% specificity, 77.6% PPV, 81.3% NPV). To distinguish cognitive impairment participants (MCI+dementia) from HCs, the AUC was found to be 0.942 (95% CI: 0.90 to 0.98). An optimal cut-off TCSAsecondary task score was 18.5 (82.4% sensitivity, 95.1% specificity, 97.6% PPV, 73.8% NPV) as shown in figure 3B.

Discussion

Main findings

The current study demonstrated a newly developed two-step progressive touchscreen-based cognitive assessment tool called the TCSA and investigated its validity and accuracy for the early detection of cognitive impairment in older adults. Overall, the TCSA showed good discriminant validity in distinguishing participants with cognitive impairment from healthy controls. We found evidence that the touchscreen-based cognitive test probably had better discriminant validity than the MMSE in participants with higher education levels. The results of the ROC analyses indicated that the two tasks of the TCSA to measure cognition had discriminant validity, respectively. This brief and comprehensive cognitive test is feasible and suitable for screening patients with cognitive impairment in the Chinese community.

Numerous attempts have been made to develop cognitive function tests based on a touchscreen battery.25–28 Such tests can be carried out at home without an examiner and data can be stored digitally and simultaneously in the computer during testing. However, there are several disadvantages associated with using a computer. First, previous assessment tools did not offer tailored questions to participants depending on their cognitive status. Second, several tablet-based tests only measure some specific cognition features, such as attention tests or memory tests.29 30 Third, longer administration time makes it less suitable for older people.16 In the present study, the TCSA consisted of two parts: TCSAprimary task and TCSAsecondary task. The primary (taking about 1–3 min to complete) and secondary screening tasks (taking about 8–10 min to complete) were conducted on a mobile phone that automatically executes the test.

The TCSAprimary task is similar to a widely used screening tool called the Mini-Cog that needs 3–5 min to finish. However, the questions on the Mini-Cog may be too simple for MCI patients, which can easily lead to a ‘ceiling effect’. One meta-analysis study found that the Mini-Cog had a sensitivity of 63.4% and a specificity of 65.4% in distinguishing MCI.31 In this study, the TCSAprimary task had a sensitivity of 82.5% and a specificity of 71.4% for MCI, with a score cut-off of 7.5 indicating significant impairment. Although it is not definitive to conclude the TCSAprimary task is better than the Mini-Cog given different study samples used, the TCSA does offer an alternative tool, that is, short and easy to access, for MCI screening. The TCSAsecondary task was shorter than the time needed to complete the MMSE or C-MoCA. Previous studies suggest the MMSE had a sensitivity of 0.62 (95% CI: 0.52 to 0.71) and a specificity of 0.87 (95% CI: 0.80 to 0.92) for MCI.32 In this study, AUC values for the MMSE in differentiating MCI from HC were 0.660 and 0.685, respectively, for those with a 12-year education or longer and those with less than 12 years of education. The total scores of secondary tasks and touch screen-based cognitive assessment tool (CAT) showed better accuracy than the MMSE in differentiating MCI from HC. However, comparing the TCSA to the MMSE, which are not gold standards for diagnosing MCI or dementia, subjects with MCI showed lower education levels which may also have created bias.

Our method successfully distinguished subjects with MCI from HCs and patients with dementia, with AUCs of 0.813 and 0.873. Both the TCSAprimary task and TCSAsecondary task could differentiate MCI from dementia and HCs. In order to shorten the tests used to identify MCI, we recommend using the TCSAprimary task to distinguish them from HCs. Then, participants with who met the threshold for cognitive impairment in TCSAprimary task were prompted to take the TCSAsecondary task to receive a more in-depth assessment. Therefore, TCSA has the advantage that it can provide comprehensive cognitive assessment and is able to shorten the tests used to identify probable cognitive impairment.

In the present study, all older participants recruited completed the whole task and there were no dropouts or immediate adverse effects being reported. Participants in this study reported that taking the MMSE was boring like taking an examination, while the TCSA was more fun, like playing a game. Although our application did provide examples and an introduction video for first time users, elderly participants still had difficulty in finding important buttons (eg, ‘next step’ and ‘cancel’). For that, we suggest that large font, clear buttons and high-contrast texts are needed, and that caregivers can also help.

There was a significant correlation between the touchscreen-based subtests and paper-based tests, namely, the C-MoCA. The C-MoCA is considered a valid means of screening for MCI and dementia,33 34 despite the different testing platforms. The TCSA is also comprised of several subdomains, such as attention, and orientation to time and space, which are also found to be related to C-MoCA score. The total score of the TCSAsecondary task had the highest correlation coefficient (0.87), supporting high concurrent validity of the total score with the C-MoCA. Because of this, the TCSA should also be considered valid. Furthermore, based on the results of our test–retest results, the TCSA is a stable test that is not affected by learning effect. These findings offer promise that the TCSA can be used to augment clinical diagnostic assessments. It still needs to be noted that the TCSA is only a screening test to indicate the possible presence of cognitive impairment, not a diagnostic tool or measurement of cognitive dysfunction.

Limitations

This study also had several limitations. First, the sample size was small in this feasibility study. Second, the MCI group was heterogeneous. The diagnosis of MCI can be quite challenging and needs more comprehensive neuropsychological assessments. It may have created bias in MCI participants with low levels of education. Third, the cross-sectional design of the study also limits the generalisability of our findings. Further studies are needed to increase the statistical power by using large and multisite samples, testing the effectiveness of the TCSA in differentiating between HCs, patients with different MCI subtypes and patients with various neurodegenerative diseases. Future studies should also assess the TCSA at different time points in a normal ageing population and those with dementia of differing severity.

Implications

In conclusion, this study demonstrated that the TCSA is an easily self-administered screening tool for early cognitive impairment. In addition to its validity and reliability, it is easy to use and time-saving. The novel two-step screening model adapted in the TCSA allows rapid screening among healthy people and a more comprehensive assessment of those at risk for dementia. A more comprehensive range of tests would be incorporated in the next phase to improve the precision of this cognitive function measurement.

Jing Nie is a PhD student in the Department of Geriatric Psychiatry at the Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine in China. She obtained a bachelor’s degree in clinical medicine from the Xuzhou Medical College and a master’s degree from Shanghai Jiao Tong University, China in 2020. Her main research interests include the longitudinal study on cognitively impaired/demented elderly adults.

author bio image