Newly self-administered two-step tool for screening cognitive function in an ageing Chinese population: an exploratory cross-sectional study

Study population

Subjects were recruited between November 2020 and February 2021 from outpatients at the Shanghai Mental Health Center and in the community in Shanghai, China. A total of 164 participants were assessed: 13 were excluded, and 151 individuals were included in the final analysis. Participants were then placed in different groups: an MCI group (n=46), a dementia group (n=44) and healthy controls (HCs) (n=61). Participants were excluded if they withdrew consent (n=4), had depression (n=3) or could not understand the instructions or were unable to complete the practice run for the assessment (n=6). A small sample of participants (n=5 for each group) were chosen by simple randomisation to complete the TCSA tasks 3 weeks later for test–retest reliability analysis. The remaining participants did not take any other tests. The flowchart for this study is shown in figure 1. HCs were selected by convenience sampling from the community. Inclusion criteria for all participants included the following: (1) men and women at least 65 years old, (2) the ability to speak Chinese, and (3) the ability to understand verbal and written directions because the touchscreen-based cognitive assessment task involved visual-auditory functions. Exclusion criteria included the following: (1) psychiatric disorders other than dementia or cognitive decline due to traumatic, substance use or medical causes; (2) severe visual deficits (such as droopy eyelid, cataracts, detached retinas, glaucoma, extremely small pupils, etc); and (3) auditory impairments that prevented participants from hearing instructions clearly.

Request permission

Diagnoses of dementia (including Alzheimer’s disease, vascular dementia, and mixed dementia) made by geriatric psychiatry directors were based on the Diagnostic and Statistical Manual of Mental Disorders–Fifth edition (DSM-5)19 criteria; MCI was diagnosed by geriatric psychiatrists based on the criteria given by Petersen and colleagues,20 including (1) memory concerns, usually stated by the patient, preferably corroborated by an informant; (2) objective memory impairment for age; (3) essentially normal general cognitive function as judged by the physician; (4) normal activities of daily living as judged by the physician; and (5) no dementia. Specific neuropsychological cut-off scores were not used to diagnose MCI. The assessment included comprehensive history, physical examination, functional assessment, behavioural scores and depression screening (Geriatric Depression Scale (GDS) score greater than 7). Participants were determined to be HCs if they demonstrated no evidence of cognitive decline as compared with their baseline cognitive functions on a clinical interview and assessment. A detailed medical, social and family history was obtained from each participant.

Measures

All participants completed a paper-based cognitive assessment before the touchscreen-based cognitive assessments were administered.

Paper-based cognitive assessments

All participants completed a series of tests including the following subtests: the Chinese Version of Montreal Cognitive Assessment (C-MoCA), the Mini-Mental State Examination (MMSE), the Activities of Daily Living (ADL, maximum score of 56)21 and the Clinical Dementia Rating scale (CDR).22

The MMSE and C-MoCA were used to determine global cognition. Both of them are scored within a range of 0–30 points (higher scores indicating better function). The ADL consists of 14 items assessing the level of physical functioning. Scores ranged from 14 to 56, with higher scores representing a higher level of independence. The CDR assesses the severity of dementia and was used for our study. CDR classifies scores as the following: 0 (no cognitive decline), 0.5 (questionable), 1 (mild), 2 (moderate) and 3 (severe cognitive decline).

Thoven Cognitive Self-Assessment (TCSA)

The TCSA was developed using HTML5 technology and adapted to the WeChat mobile application (Shenzhen Tencent Computer Systems, Shenzhen, Guangdong, China) for self-administration. Data were stored on a cloud server (Ali Cloud) located in China and licensed from Thoven (Thoven Technology, Shanghai, China).

In the development of TCSA, an item pool of tests was constructed from the literature, previous cognitive assessment scales and by consulting experts in the field of geriatric psychiatry and neurology. The content validity was conducted through expert judgement. Two independent reviews were carried out by a panel of three experts from the fields of psychology, geriatric psychiatry and neurology to select tests that were appropriate, accurate and interpretable. The acceptance, rejection or modification of an item was based on majority opinion. A flow chart of the selection process and included modules is presented in the online supplemental Figure 1.

The TCSA test consisted of two parts: a primary screening section and a secondary screening section. The primary screening section was completed in 1–3 min with a total score of 11 points. The total score of the secondary screening section is 30, and most participants finished it in about 9 min (ranging from 8 to 10 min). Five major assessment modules covering nine tasks were developed to assess cognition that included the following: (1) working and episodic memory tests with the visual, audio and audiovisual examination, which included episodic image recognition, turnover cards and memory gas station, as well as an episodic memory recall test called ‘episodic image recall’; (2) attention and reaction-time tests including Musical Metronome and Shult Grid; (3) examinations of calculation; (4) a language test called ‘scratch-off ticket’; and (5) orientation to time and place tests. A description of all tasks and actions with the outcomes, measures and main cognitive functions assessed through these actions is shown in the online supplemental Table 1.

To begin, participants touched the Start button on the screen to log into the system. Then, they were required to provide information, including phone number, name, birth date, gender and years of education. Before the actual TCSA test was administered, the test was explained in detail and a test introduction video was given to participants. This included not only the test itself but also the instruction and count-down pages to let the participant get accustomed to the layout of the site and the initial actions needed before the start of the test.

Statistical analysis

Statistical analysis was performed using SPSS V.22. Normal distributed variables were presented as mean and SD. Non-normal distributed variables were presented as median and IQR (25% percentile–75% percentile). Sex was presented as number (n) and percentage (%). We used the analysis of variance (ANOVA), Kruskal-Wallis test and χ² test to make demographic comparisons between the three groups. To examine the performance differences in cognitive tests, neuropsychological tests and TCSA scores among the three groups, analysis of covariance was used while adjusting for demographic variables that differed statistically among the three groups.

Partial correlation analysis was used to assess correlation between the C-MoCA scores and TCSA scores, while controlling for age and years of education. Pairwise comparison of receiver operating characteristic (ROC) curves tested the equality of two or more ROC areas obtained from applying two or more test modalities to the same subject. It was calculated to compare the area under curve (AUC) of the two screening tests (TCSA vs MMSE).

ROC curves were used to obtain the sensitivity and specificity of the TCSA to determine whether it can be a useful instrument for differentiating (1) patients with MCI from HCs, (2) patients with MCI from those with dementia and (3) cognitively impaired subjects (patients with MCI and dementia) from HCs. The cut-off point was identified as the value corresponding to the maximum value of Youden’s index, calculated as (sensitivity+specificity–1). The AUC with 95% confidence interval (CI) for the TCSA score was used to measure its accuracy to differentiate patients with and without cognitive impairment.

Cronbach’s α coefficient reliabilities were calculated to analyse the internal consistency of the scale items. Cronbach’s alpha values of 0.6 or higher were considered acceptable.23 Test–retest reliability was demonstrated by measuring the TCSA on two separate occasions (3 weeks apart) for a small sample of subjects (n=15) chosen by simple randomisation.5 It was evaluated using partial correlation analysis. Statistical significance was set at p<0.05.

Demographic and clinical characteristics

There was no significant difference in age or gender between the HC group, the MCI group and the dementia group, but an educational level difference was found between the three groups (F=24.98, p<0.001). The control group had a significantly higher mean (SD) MMSE scores (25.35 (5.95) vs 21.51 (5.61) vs 14.90 (6.96), F=23.51, p<0.001) and C-MoCA scores (26.09 (2.81) vs 18.84 (4.97) vs 10.74 (6.04), F=136.90, p<0.001) than the MCI group and the dementia group. The control group also showed significantly lower ADL and CDR scores (table 1).

Discriminant validity: between-group differences on test parameters

The HC group showed significantly higher mean primary screening task scores (8.58 (1.76) vs 5.40 (2.67) vs 2.74 (2.11), F=75.40, p<0.001) and secondary screening task scores (23.02 (3.31) vs 17.95 (4.93) vs 11.93 (5.50), F=76.46, p<0.001) than the MCI or dementia groups, as shown in figure 2.

Request permission

As shown in table 2, the one-way ANOVA tests yielded a significant difference in the score for different cognitive domains among the three groups. There was no significant difference found for working memory (Z=3.11, p=0.211) among the groups. TCSA was found to be more sensitive than the MMSE in discriminating MCI from HC, as reflected by the AUC scores. The AUC for the TCSA_{primary task} was found to be superior to the MMSE, in differentiating MCI from HC, irrespective of the educational level (over or under 12 years) of subjects.24 The TCSA_{secondary task} performance over the MMSE was more evident for older adults who had over 12 years of formal education, with an AUC of 0.759 (95% CI: 0.56 to 0.96) vs 0.685 (95% CI: 0.46 to 0.91).

Internal consistency was assessed using the Cronbach’s α coefficient. The Cronbach’s α of 0.88 suggests relatively high internal consistency.

Concurrent validity and test–retest reliability

Concurrent validity was evaluated by presenting the partial correlation coefficient with age and years of education as the control factors between TCSA subtests and the C-MoCA scores. Memory and language factors were moderately correlated with scores from paper-based tests which ranged from 0.41 to 0.49 (p<0.001). The factors that measured attention and orientation were strongly correlated with the C-MoCA test results (r=0.74 and 0.87 respectively, p<0.001). The correlation coefficient between TCSA and C-MoCA total scores was 0.87 (p<0.001) (online supplemental Table 2).

Test–retest reliability was demonstrated by measuring the total scores of the TCSA primary and secondary tests on two separate occasions among a small sample of subjects (five dementia patients, five MCI patients and five HCs) 3 weeks apart. Pearson’s correlation coefficient showed good test–retest correlation for primary (r=0.85) and secondary tasks (r=0.88).

Discriminant validity: ROC

The results of the ROC analyses, which assessed the ability of TCSA to discriminate between HC and MCI groups, and between MCI and dementia groups, are presented in figure 3.

Request permission

For the primary task, to distinguish MCI participants from healthy older adults, the area under the curve (AUC) was found to be 0.813 (95% CI: 0.71 to 0.92). A cut-off value of TCSA_{primary task} of 7.5 (a phonemic advantage) indicated 71.4% specificity, 82.5% sensitivity, 71.1% positive predictive values (PPV) and 82.4% negative predictive values (NPV). To distinguish MCI subjects from patients with dementia, the AUC was found to be 0.873 (95% CI: 0.79 to 0.95). An optimal statistical cut-off of TCSA_{primary task} was achieved at 3.5 (85.7% sensitivity, 73.2% specificity, 73.3% PPV, 83.8% NPV). To distinguish cognitive impairment participants (MCI+dementia) from HCs, the AUC was found to be 0.937 (95% CI: 0.90 to 0.98). An optimal statistical TCSA_{primary task} cut-off score was achieved at 4.5 (85.3% sensitivity, 85.4% specificity, 83.1% PPV, 88.4% NPV) as shown in figure 3A.

For the secondary task scores, to distinguish MCI participants from healthy older adults, the AUC was found to be 0.809 (95% CI: 0.70 to 0.91). An optimal statistical cut-off value of TCSA_{secondary task} was achieved at 22.5 (75.0% sensitivity, 71.4% specificity, 72.0% PPV, 75.9% NPV). To distinguish MCI subjects from patients with dementia, the AUC was found to be 0.878 (95% CI: 0.80 to 0.96). An ideal statistical TCSA_{secondary task} cut-off was achieved at 14.5 (89.3% sensitivity, 63.4% specificity, 77.6% PPV, 81.3% NPV). To distinguish cognitive impairment participants (MCI+dementia) from HCs, the AUC was found to be 0.942 (95% CI: 0.90 to 0.98). An optimal cut-off TCSA_{secondary task} score was 18.5 (82.4% sensitivity, 95.1% specificity, 97.6% PPV, 73.8% NPV) as shown in figure 3B.

Main findings

The current study demonstrated a newly developed two-step progressive touchscreen-based cognitive assessment tool called the TCSA and investigated its validity and accuracy for the early detection of cognitive impairment in older adults. Overall, the TCSA showed good discriminant validity in distinguishing participants with cognitive impairment from healthy controls. We found evidence that the touchscreen-based cognitive test probably had better discriminant validity than the MMSE in participants with higher education levels. The results of the ROC analyses indicated that the two tasks of the TCSA to measure cognition had discriminant validity, respectively. This brief and comprehensive cognitive test is feasible and suitable for screening patients with cognitive impairment in the Chinese community.

Numerous attempts have been made to develop cognitive function tests based on a touchscreen battery.25–28 Such tests can be carried out at home without an examiner and data can be stored digitally and simultaneously in the computer during testing. However, there are several disadvantages associated with using a computer. First, previous assessment tools did not offer tailored questions to participants depending on their cognitive status. Second, several tablet-based tests only measure some specific cognition features, such as attention tests or memory tests.29 30 Third, longer administration time makes it less suitable for older people.16 In the present study, the TCSA consisted of two parts: TCSA_{primary task} and TCSA_{secondary task}. The primary (taking about 1–3 min to complete) and secondary screening tasks (taking about 8–10 min to complete) were conducted on a mobile phone that automatically executes the test.

The TCSA_{primary task} is similar to a widely used screening tool called the Mini-Cog that needs 3–5 min to finish. However, the questions on the Mini-Cog may be too simple for MCI patients, which can easily lead to a ‘ceiling effect’. One meta-analysis study found that the Mini-Cog had a sensitivity of 63.4% and a specificity of 65.4% in distinguishing MCI.31 In this study, the TCSA_{primary task} had a sensitivity of 82.5% and a specificity of 71.4% for MCI, with a score cut-off of 7.5 indicating significant impairment. Although it is not definitive to conclude the TCSA_{primary task} is better than the Mini-Cog given different study samples used, the TCSA does offer an alternative tool, that is, short and easy to access, for MCI screening. The TCSA_{secondary task} was shorter than the time needed to complete the MMSE or C-MoCA. Previous studies suggest the MMSE had a sensitivity of 0.62 (95% CI: 0.52 to 0.71) and a specificity of 0.87 (95% CI: 0.80 to 0.92) for MCI.32 In this study, AUC values for the MMSE in differentiating MCI from HC were 0.660 and 0.685, respectively, for those with a 12-year education or longer and those with less than 12 years of education. The total scores of secondary tasks and touch screen-based cognitive assessment tool (CAT) showed better accuracy than the MMSE in differentiating MCI from HC. However, comparing the TCSA to the MMSE, which are not gold standards for diagnosing MCI or dementia, subjects with MCI showed lower education levels which may also have created bias.

Our method successfully distinguished subjects with MCI from HCs and patients with dementia, with AUCs of 0.813 and 0.873. Both the TCSA_{primary task} and TCSA_{secondary task} could differentiate MCI from dementia and HCs. In order to shorten the tests used to identify MCI, we recommend using the TCSA_{primary task} to distinguish them from HCs. Then, participants with who met the threshold for cognitive impairment in TCSA_{primary task} were prompted to take the TCSA_{secondary task} to receive a more in-depth assessment. Therefore, TCSA has the advantage that it can provide comprehensive cognitive assessment and is able to shorten the tests used to identify probable cognitive impairment.

In the present study, all older participants recruited completed the whole task and there were no dropouts or immediate adverse effects being reported. Participants in this study reported that taking the MMSE was boring like taking an examination, while the TCSA was more fun, like playing a game. Although our application did provide examples and an introduction video for first time users, elderly participants still had difficulty in finding important buttons (eg, ‘next step’ and ‘cancel’). For that, we suggest that large font, clear buttons and high-contrast texts are needed, and that caregivers can also help.

There was a significant correlation between the touchscreen-based subtests and paper-based tests, namely, the C-MoCA. The C-MoCA is considered a valid means of screening for MCI and dementia,33 34 despite the different testing platforms. The TCSA is also comprised of several subdomains, such as attention, and orientation to time and space, which are also found to be related to C-MoCA score. The total score of the TCSA_{secondary task} had the highest correlation coefficient (0.87), supporting high concurrent validity of the total score with the C-MoCA. Because of this, the TCSA should also be considered valid. Furthermore, based on the results of our test–retest results, the TCSA is a stable test that is not affected by learning effect. These findings offer promise that the TCSA can be used to augment clinical diagnostic assessments. It still needs to be noted that the TCSA is only a screening test to indicate the possible presence of cognitive impairment, not a diagnostic tool or measurement of cognitive dysfunction.

Limitations

This study also had several limitations. First, the sample size was small in this feasibility study. Second, the MCI group was heterogeneous. The diagnosis of MCI can be quite challenging and needs more comprehensive neuropsychological assessments. It may have created bias in MCI participants with low levels of education. Third, the cross-sectional design of the study also limits the generalisability of our findings. Further studies are needed to increase the statistical power by using large and multisite samples, testing the effectiveness of the TCSA in differentiating between HCs, patients with different MCI subtypes and patients with various neurodegenerative diseases. Future studies should also assess the TCSA at different time points in a normal ageing population and those with dementia of differing severity.

Implications

In conclusion, this study demonstrated that the TCSA is an easily self-administered screening tool for early cognitive impairment. In addition to its validity and reliability, it is easy to use and time-saving. The novel two-step screening model adapted in the TCSA allows rapid screening among healthy people and a more comprehensive assessment of those at risk for dementia. A more comprehensive range of tests would be incorporated in the next phase to improve the precision of this cognitive function measurement.