Cost-benefit and discriminant validity of a stepwise dementia case-finding approach in an Asian older adult community

Study design

The Singapore Epidemiology of Eye Diseases (SEED) Study used an age-stratified random sampling strategy to select residents between the ages of 40 and 80+ years from 15 residential districts in the southwestern part of Singapore, an area fairly representative of the country’s population in age, housing and socioeconomic status. The cohort profile of this study has been published previously.12 The SEED Study comprised the Singapore Chinese Eye Study from 2009 to 2011, the Singapore Malays Eye Study from 2010 to 2013, and the Singapore Indian Eye Study from 2013 to 2015. Details of the above studies have been described previously.13–15 Among all Singaporean adult participants, a convenient sample of senior residents aged 60 years and older were included in the present Epidemiology of Dementia in Singapore (EDIS) Study. The study excluded older adults who met any of the following criteria: (1) suffering from a malignant disease, such as cancer, tumour, etc; (2) diagnosed with major depressive disorder or other psychiatric illnesses; (3) with severe visual, hearing or communication impairments. All eligible older adults and their caregivers were sent an invitation via telephone, email and/or home visit to go to the Singapore Eye Research Institute for the assessment. A person was termed ‘uncontactable’ after six unsuccessful telephone calls and/or home visits.

Phase I: epidemiological survey and cognitive screening

The EDIS Study was a two-phase study as the SPEED Study.10 In phase I, a questionnaire on demographic information and relevant risk factors, along with a cognitive screening test comprised of the AMT16 and the PFQ, were administered by trained investigators. Previously validated in Singapore, AMT’s adjusted optimal cut-off for participants with 0–6 years of education was 6/7 (sensitivity 89.6% and specificity 92.6%), and for those with more education was 8/9 (sensitivity 82.1% and specificity 92.9%).17 The PFQ refers to a single question of subjective cognitive complaints that asks the primary caregiver who had at least 10 hours of interaction with the participant weekly about the older adult’s experience of progressive forgetfulness18; an affirmative response was considered positive. Older adults who screened positive for either of the above two tests were invited to enter phase II of the study, at which participants underwent extensive clinical and neuropsychological evaluations, as described in more details previously.15

Phase II: cognitive assessments and dementia diagnosis

The MoCA, MMSE and a formal neuropsychological battery were performed in phase II. All the above tests have been locally validated for Singaporean older adults.19 20 The formal neuropsychological battery was administered by trained research psychologists. The domains of this battery included the following15:

1. Executive function: the Frontal Assessment Battery and Maze Task.

2. Attention: Digit Span, Visual Memory Span and Auditory Detection.

3. Language: the Boston Naming Test and Verbal Fluency.

4. Visuomotor speed: the Symbol Digit Modality Test and Digit Cancellation.

5. Visuoconstruction: the Weschler Memory Scale-Revised Visual Reproduction Copy task, Clock Drawing and the Weschler Adult Intelligence Scale-Revised subtest of Block Design.

6. Verbal memory: Word List Recall and Story Recall.

7. Visual memory: Picture Recall and the Weschler Memory Scale-Revised Visual Reproduction.

Cognitive impairment and dementia were diagnosed by consensus at formal research team meetings, using the results from the above-listed tests. Individual test scores below the education-adjusted cut-offs of 1.5 SDs were categorised as test failures.15 Impairment in a cognitive domain was defined as failure in at least half of the tests in that domain. Cognitive impairment with no dementia (CIND) was classified into CIND-Mild (when ≤2 cognitive domains were impaired) and CIND-Moderate (when >2 domains were impaired). Dementia was diagnosed according to the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition criteria.21

Statistical analyses

One-way analysis of variance and Χ² tests were used to compare differences in sample characteristics. Receiver operating characteristic curve analyses were conducted to establish the area under the curve (AUC) for different cognitive instruments (MoCA, MMSE, AMT) between PFQ=yes and PFQ=no groups. The discriminant indices, including sensitivity, specificity, positive and negative predictive values (NPV), and overall accuracy, were calculated using the optimal cut-off points for each screening tool.22 Meanwhile, the verification bias of the PFQ for diagnostic groups was adjusted using the Bayesian correction method for differential verification.23

We investigated the discriminant indices of the PFQ as a stepwise method, followed by other commonly used cognitive instruments, including the AMT, MoCA and MMSE, for dementia detection. Meanwhile, we further examined whether the AUCs of MoCA, MMSE and AMT for detecting dementia differed between the two PFQ groups (PFQ=yes and PFQ=no).

In addition, we calculated the overall screening time and the number of older adults requiring further comprehensive evaluation for different screening approaches in the following two scenarios. In the first scenario, only the MoCA, MMSE or AMT was used to identify dementia high-risk individuals. In the second scenario, those who screened positive on the PFQ were subsequently administered the MoCA, MMSE or AMT to identify participants who were at high risk of developing dementia. The administration time of the MoCA, MMSE and AMT was 10 min, 10 min and 8 min, respectively.17 24 25 We assumed the time required for the PFQ was 10 s.

Number with dementia: true positives need to be identified.

PR%: the prevalence of dementia in this study.

Time_a: the time cost of step 1.

Time_b: the time cost of step 2.

All analyses were done on IBM SPSS V.26.0 (IBM Corp Released 2019. IBM SPSS Statistics for Windows), and a p value <0.05 was considered statistically significant.

Demographics

Figure 1 shows the study recruitment flow chart. A total of 3800 community residents (age ≥60 years) from the SEED Study were eligible for inclusion, 20 of whom refused to participate. Finally, a total of 3780 older adults were included in the EDIS Study and completed the brief cognitive screening of phase I; among these, 918 completed phase II (887 PFQ=yes, 31 PFQ=no). The sample characteristics of phase II older adults selected by the PFQ are shown in table 1. In addition, we compared the characteristics of the two screened-positive groups that entered phase II versus those who did not enter phase II (see online supplemental table 1). Older adults who screened positive and entered phase II were younger (mean age 70.3 vs 72.1 years), less likely to be female (52.5% vs 59.0%) and less educated (60.5% vs 71.6%) than those who screened positive but did not enter phase II.

Request permission

Discriminant indices of the stepwise case-finding approach for detecting dementia

Among the 918 older adults in phase II, 45 (4.9%) were finally diagnosed with dementia. We explored the discriminant indices of the PFQ with other cognitive tools for detecting dementia in the PFQ=yes and PFQ=no groups. Results showed good overall accuracy for the MMSE (93.3%), MoCA (86.2%) and AMT (88.4%), respectively; all three tools achieved an optimal NPV exceeding 99% in those older adults with a positive PFQ (table 2 and online supplemental table 2). Also, we further compared the AUCs of the above-mentioned cognitive tools for dementia detection between the PFQ=yes and PFQ=no groups (figure 2). In the PFQ=yes group, MoCA, MMSE and AMT had AUCs of 0.95 (95% CI: 0.62 to 0.98), 0.94 (95% CI: 0.90 to 0.98) and 0.93 (95% CI:0.87 to 0.97), respectively, for discriminating between participants with and without dementia. However, lower AUCs were found in the PFQ=no group as compared with the PFQ=yes group on the MoCA (0.87, 95% CI: 0.71 to 0.99), MMSE (0.83, 95% CI: 0.62 to 0.99) and AMT (0.78, 95% CI: 0.46 to 0.98). A more detailed table is presented in online supplemental table 2.

Request permission

Cost-effective analysis of the stepwise dementia case-finding approach

We assumed two case-finding scenarios in the cost-effectiveness analysis for the case-finding approach, ie, without and with preselection by the PFQ. According to the present study, the prevalence of dementia was 4.9% (45 of 918) in this study sample.

The adjusted sensitivity and specificity with PFQ for dementia detection using the Bayesian correction method were 48.2% and 52.4%, respectively. It is estimated that a total of 1905 (refer to the Methods section for calculation formula) older adults would have been recruited for the PFQ preselected test to identify 45 individuals with dementia. In the first scenario (MMSE or MoCA or AMT), the 1905 individuals would have had to perform one of the following three tests to identify the 45 true positives: the MoCA (cut-off: 13/14, sensitivity: 91.1%, specificity: 85.8%), MMSE (cut-off: 17/18, sensitivity: 80.0%, specificity: 93.7%) or AMT (cut-off: 6/7, sensitivity: 75.6%, specificity: 92.8%). Without the preselection of the PFQ, a total of 159, 302 and 175 older adults would have failed the MMSE, MoCA or AMT, and hence would have been required to undergo further assessment for confirmation of diagnosis. The screening time for the three tools would have been 317.5 hours, 317.5 hours and 254 hours, respectively (figure 3A).

Request permission

In the second scenario (preselection by the PFQ, then MMSE or MoCA or AMT), screening was done to the same 1905 individuals by the PFQ first, among whom 45 true positives and 862 false positives would have failed in the PFQ, leading to a total of 907 older adults entering the next step to perform MMSE, MoCA or AMT. Subsequently, the sensitivity and specificity of the MoCA in the PFQ=yes group were 92.7% and 85.9%, respectively. Thus, 167 older adults (45 true positives, 122 false positives) would have entered the final diagnostic evaluation. In this case, the total screening time for this stepwise approach with the PFQ (1905×10 s) followed by the MoCA (907×10 min) would be 156.5 hours. Following the same calculation logic, preselecting by the PFQ followed by the MMSE or AMT would have required 98 and 145 older adults for further testing, and the overall screening time would have been 156.5 hours and 126.2 hours, respectively (figure 3B).

Significantly, in the stepwise approach, the first step of using the PFQ and then adding the MoCA, MMSE or AMT resulted in a 50.7%, 50.7% and 50.3% reduction in the total screening time cost, respectively. The number of people requiring further evaluation would have decreased by 135 (44.7%), 61 (38.4%) and 30 (17.1%), respectively.

In summary, we provide a stepwise dementia case-finding strategies for community-dwelling older adults: apply a single-question assessment—the PFQ—as the first step to stratify case-finding of dementia in a large population and then perform other objective cognitive tests. Only individuals who test positive in both steps could be included in a final comprehensive cognitive assessment (online supplemental figure 1).

Main findings

In this study, we found that the stepwise combination of using the PFQ with other cognitive tools can remarkably rule out older adults in the community who are at a lower risk of dementia. Hence, implementing dementia screening using this stepwise case-finding approach in large population settings can effectively exclude individuals at low risk of dementia and reduce the time and resources required for further assessment.

Previous studies have shown that single-question SCD assessment tools were not suitable for use alone in community settings to identify individuals with early dementia, including the 10th item on the Geriatric Depression Scale and the 8th item on the Ascertain Dementia 8, as they yielded limited discriminant validity for dementia detection.26 Evidence has shown that participants with cognitive decline might not be able to describe their own mental status accurately.27 Thus, one-question SCD tools might lead to numerous false results, illustrating that using the PFQ alone could not reliably detect cognitive impairment or dementia in community settings. Nevertheless, the forgetfulness emphasised by the PFQ is progressive, which is one of the key components in assessing memory loss in the Clinical Dementia Rating (CDR) instrument, especially in distinguishing normal cognition (CDR=0) from mild cognitive impairment (CDR=0.5). Thus, although the discriminant validity of the PFQ is limited when used alone, using it as a first step in a large-scale dementia screening setting can help identify older adults with possible advancing cognitive impairment so that they can receive an additional assessment as quickly as possible. Meanwhile, adding such single-question assessments before other objective cognitive tests can ease participants’ nervousness and foster good relationships between the investigator and the participants.11

Noteworthy, we found that using the PFQ stepwise with other cognitive tools (MoCA, MMSE or AMT) offers a promising approach for dementia case-finding, especially for ruling out those at low risk for dementia. Our results are consistent with the previous SPEED study, which demonstrated that without the PFQ, there were epidemiological reasons not to proceed with further AMT administration.9 10 Furthermore, we extended the findings of a previous study to the MoCA and MMSE. Among older adults who reported PFQ=yes in our study, the NPV of all the tools slightly improved and achieved excellent AUCs (MoCA=0.95, MMSE=0.94 and AMT=0.93) for detecting dementia. The overall accuracy also yielded favourable results (MoCA: 86.2%, MMSE: 93.3% and AMT: 88.4%). The optimal cut-offs of the MoCA and the MMSE in the present study were lower than in general population studies,28 29 mainly because the sample we included for the final analysis was a population with possibly high cognitive risk who performed poorly at phase I. However, these cut-off values were broadly consistent with previous studies in the Asian population at high risk of dementia.19 30 31 Inevitably, in large-scale dementia screening, older adults with cognitive impairment may report negative PFQ scores, leading to false negative results. However, in the present study, most older individuals who reported ‘no’ on the PFQ were dementia-free, with only 9.7% (3 in 31) of false negatives.

We have also shown that the stepwise case-finding approach can be time cost-effective when implementing early dementia screening within communities. Comprehensive neuropsychological cognitive testing, which is extensive and domain specific with a lengthy administration time of approximately 1 hour, requires a specialist to administer and cannot readily be given to every older community adult for formal cognitive impairment diagnosis.18 A stepwise case-finding approach can significantly reduce the number of participants requiring further assessment for diagnosis confirmation and the associated time costs. We calculated the overall screening time and the number of individuals requiring further evaluation by assuming two scenarios. The total screening time for the preselection by the PFQ and then performing the MMSE or MoCA or AMT would have been 156.5 hours, 156.5 hours and 126.2 hours, respectively, and would have required 98, 167 and 145 individuals for further testing. Without the preselection by the PFQ, the screening time for the three cognitive tools would have been 317.5 hours, 317.5 hours and 254 hours, respectively, with a total number of 159, 302 and 175 individuals who would have been entered for further evaluation. Thus, our study showed the overall screening time would have been decreased by 50.7%, 50.7% and 50.3%, respectively, when preselecting the positive PFQ participants to undergo the MMSE, MoCA or AMT. The number of individuals requiring further evaluation would have decreased by 61, 135 and 30 individuals, demonstrating the effectiveness of the stepwise case-finding approach in the community to minimise human resources and time costs. Therefore, considering the accuracy and time cost-savings, we recommend using a stepwise method that first asks the PFQ and then conducts other cognitive tests to rule out as many individuals at low risk of dementia as possible in the community setting.

Limitations

We acknowledge several limitations of our study. First, as this study was conducted in a community setting in Singapore, the sample is specific, and the external validity of the stepwise case-finding approach in other settings or populations remains to be confirmed. Second, the 43% dropout rate in phase II may have impacted our results; participants who continued to phase II were younger, more educated and differed significantly in gender, ethnicity and hypertension history from those who refused further participation. Third, the optimal cut-off values for the MoCA and MMSE used in this study were lower than those for the general population, mainly because the vast majority of older adults included in our analysis were those with positive initial screening results, which may affect the extrapolation of the study results. Fourth, this study has the inherent property of verification bias, a measurement bias often associated with screening tests, which can conceal the diagnostic ability of the designated screening tool.23 Though we invited participants who screened negative in phase I to continue to phase II, only a few participants consented to further evaluation. Thus, the diagnostic performance of the stepwise case-finding approach may be biased by such attrition, the sensitivity and NPV may be overestimated, and the specificity may be underestimated, even after correction for verification bias. Moreover, it should be noted that the PFQ=no group in the current study may not be representative of the general Singapore population due to its small sample size. In addition, a previous Singapore-based study9 reported a much lower dementia rate of 1.6% (1 out of 61) in the PFQ=no group. As these epidemiological studies were conducted at different periods of time, with differential sample demographics, further comparisons should be made with caution. Future studies could consider including more participants who screened negative on the PFQ for a gold-standard evaluation to validate further the effectiveness of excluding low-risk populations in community settings. In addition, further studies could verify the feasibility and discriminant utility of other brief screening tools equivalent to the PFQ with the same stepwise strategies.

Implications

This study demonstrated a stepwise case-finding approach for dementia detection in large-scale screening. Using a single-question assessment, such as the PFQ, the first step excludes individuals at low risk of dementia and identifies those who may potentially be at high risk; the second step then involves performing another objective cognitive test on the high-risk individuals while minimising time costs.