Tests for paired count outcomes

Paired continuous and binary outcomes

Consider a sample of n subjects indexed by i and let    and    denote the paired outcomes from the ith subject. The subject may be an individual or a pair of twins, depending on applications. For example, in a pre-post study, the paired outcomes correspond to the pretreatment and post-treatment assessment and the subject is an individual. In studies involving twins, the paired outcomes come from each pair of twins. Because the two outcomes are correlated, statistical methods for comparing independent samples such as the t-test cannot be applied.

For a continuous outcome, the paired t-test is generally applied to evaluating differences in the means of the paired outcomes. If we assume that    and    follow a bivariate normal distribution, then the difference between the two outcomes,   , is also normally distributed. Under the null of no difference between the means of the two outcomes,    has a normal distribution with mean 0 and variance   . Thus, we can apply the t-test to the differences    to test the null:

where    denotes the t distribution with k df, and    and    denote the sample mean and SD. We can also use this sampling distribution to construct CIs.

In practice, the bivariate normal distribution assumption for the paired outcomes    and    is quite strong and may not be met in real study data. If the assumption fails, the differences    generally do not follow the normal distribution and thus the    statistic in Equation (1) may not follow the t distribution. For large samples such as   ,    follows approximately a standard normal distribution. Thus, one may replace    with the ‘asymptotic’ standard normal,   , to test the null as well as construct CIs, even if the outcomes    and    are not bivariate normal.

In what follows, we assume large samples, since all the tests to be discussed next are asymptotic tests, that is, they approximately follow a mathematical distribution such as the normal distribution only for large samples. For small to moderate samples, all these tests have unknown distributions and asymptotic mathematical distributions such as the standard normal for large samples for the paired t-test may not work well. We discuss alternatives for small to moderate samples in the discussion section.

If the paired outcomes are binary, the above hypothesis becomes the comparison of the proportions of    and   . McNemar’s test is the standard for comparing the paired outcomes. Let    and    denote the proportions associated with    and   , that is,

Then the hypothesis to be tested is given by:

McNemar’s test is premised on the idea of comparing concordant and discordant pairs in the sample.

Shown in table 1 is a    cross-tabulation for the two levels of the binary outcomes. Let   ,   ,    and    denote the cell probabilities (or proportions) for the four cells in the table, that is,

Then,    can be expressed in terms of the cell probabilities as follows:

Similarly,    can be expressed as:

Thus,    implies    and vice versa. The hypothesis of interest in Equation (2) involving    and    can be expressed in terms of    and   :

McNemar’s test evaluates the difference between the concordant and discordant pairs, b and c, that is,

A large difference leads to rejection of the null. By normalising this difference, the statistic    above follows approximately the standard normal for large sample size.

Paired count outcomes

For count outcomes, McNemar’s test clearly does not apply. The paired t-test is also inappropriate for such outcomes. First, the difference    does not follow a normal distribution. Second, even if both    and    follow a Poisson distribution, the difference    is not a Poisson variable;    in general is not even guaranteed to have non-negative values.

One approach that has been used to compare paired count outcomes is the Wilcoxon signed-rank test. Within our context, let    denote the rank of    based on its absolute value   . The ranks are integers that indicate the position of    after rearranging them in ascending order.1 The signed-rank test has the following statistic:

where    denotes an indicator with the value 1 (0) if the logic   >0 is true (otherwise). Thus,    only adds up the ranks for the positive   ’s.

The statistic    ranges from 0 to   . Under   , about one-half of the   's are positive. Thus, any pair (  ,   ) has 50% chance that   . In terms of ranks, this means that the sum of    for positive    is about half of the range   . Thus, we can specify the null as:

where   . Under the null   ,    has mean   , half of the range   . The normalised   :

has approximately a normal distribution with mean    and SE    for large samples, which is readily applied to calculate p values and/or confidence bands.

Since paired outcomes are a special case of general longitudinal outcomes, longitudinal methods can be applied to test the null. For example, both the generalised linear mixed-effects model (GLMM) and generalised estimating equations (GEE), two most popular longitudinal models, can be specialised to the current setting. When applying GLMM, we specify the following model:

where z denotes a random effect to account for correlation between the paired outcomes,    denotes a Poisson distribution with mean μ ,    denotes the exponential and    denotes the log function. The null of same mean between    and    can be expressed as:

Note that since the random effect z may be positive or negative and the random of the normal distribution is unbounded, the log transformation of the Poisson mean in Equation (4) is necessary to ensure that    and    stay positive.

For applying GEE, we only need to specify the mean of each paired outcome. This is because unlike GLMM, GEE is a ‘semi-parametric’ model and imposes no mathematical distribution on the outcomes. Thus, under GEE, both the Poisson distribution for each outcome and the random effect z for linking the paired outcomes are removed. The corresponding GEE is given by:

Since there is no random effect in Equation (5), the log transformation is also not necessary and thus the GEE can be specified simply as:

Compared with the GLMM in Equation (4), the GEE above imposes no mathematical distribution either jointly or marginally, allowing for valid inference for a broad class of data distributions. The GLMM in Equation (4) may yield biased inference if: (1) at least one of the outcomes does not follow the Poisson; (2) the random effect z follows a non-normal distribution; and (3)    and    are not correlated according to the specified random-effect structure. In contrast, the GEE in Equation (5) forgoes all such constraints and yields valid inference regardless of the marginal distribution and correlation structure of the outcomes    and   .

Bias

If a test performs correctly, it should yield type I error rates at the specified nominal level   . Several factors can affect the performance of the test. First, if data do not follow the assumed mathematical distributions, the test in general is biased. For example, if the paired t-test is applied to paired outcomes that are not bivariate normal, it will generally be biased. Second, with the exception of the paired t-test, all tests discussed above rely on large samples to provide valid results. When applied to small or moderate samples, such tests may have bias. For example, the normal distribution may not provide a good approximation to the sampling distribution of the statistic    of the Wilcoxon signed-rank test    when applied to a sample size of, say,   . Thus, to compare the performance of each different method, we consider sample sizes ranging from n= 10 to 200.

To evaluate the effects of model assumptions on test performance, we simulate correlated count responses    and    using a copula approach,2 where each outcome marginally follows a negative binomial (NB) distribution:

The above model deviates from the GLMM in Equation (4) in two ways. First,    (  ) follows an NB, rather a Poisson. Second, correlation between    and    does not follow the normal distribution based on random effect. Unlike Poisson, NB has an extra parameter τ controlling for dispersion (variability). Thus, although Poisson and NB have the same mean, NB has a different (larger) variance than Poisson.1 Since    converges to    as τ increases, selecting a relatively small τ allows us to examine the impact of the Poisson assumption on inference when the GLMM in Equation (4) is applied to count outcomes that are not compliant with the Poisson model.

For the simulation study, we set    and 5, and correlation between    and    to 0.5. To evaluate bias using MC simulation, we simulate paired outcomes    and    from the model in Equation (7), apply each of the tests discussed in the third section and compute p values for testing the null hypothesis. This process is repeated    times. A test has little or no bias if the proportion of nulls rejected over the    times is close to the nominal value   .

Shown in table 2 are averaged p values for the different tests from their applications to the    simulated paired outcomes, where GLMM (Poisson) denotes the GLMM for Poisson in Equation (4), GLMM (NB) denotes the GLMM for NB distribution (by replacing the Poisson in the GLMM in Equation (4) with NB), GEE denotes the GEE without log transformation in Equation (6) and GEE (log-link) denotes the GEE with log transformation in Equation (5). For moderate to large sample sizes, n=100, 200, all tests yielded p values close to the nominal   , except for GLMM (Poisson), which had highly inflated type I errors for   . As indicated earlier, with a small dispersion parameter such as   , NB has much more variability than its Poisson counterpart, leading to poor fit when fitting simulated data with the GLMM assuming the Poisson. Thus, the high bias in the type I error reflects model mis specification.

Although the paired t-test is not a valid test, it performed well for all sample sizes considered, although showing small downward bias, especially for small sample sizes. For extremely small sample sizes such as n=10, all three asymptotically valid methods, signed-rank test, GLMM (NB) and GEE, showed small upward bias, especially when   . As the sample size increased, the bias diminished, as expected.

Power

If a group of tests all provide good type I error rates, we can further compare them for power. It is common that two unbiased tests may provide different power, because they may use a different amount of information from study data or use the same information differently. For example, within the current study, the signed-rank test may provide less power than the GEE, because the former only uses the ranks of the original count outcomes, completely ignoring magnitudes of   ’s. Thus, it is of interest to compare power across the different tests.

We again use the MC approach to compare power across the different methods. However, unlike the evaluation of bias, we must also be specific about the difference in the means of paired outcomes so that we can simulate the outcomes under the alternative hypothesis. For this study, we specify the null and alternative as follows:

We simulate correlated outcomes    again using the copula from the GLMM in Equation (7), but with    and    as specfied under    in Equation (8).

For each simulated outcome   , we apply the different methods and test the null hypothesis under   . This process is repeated    times and the power for each method is estimated by the per cent of times the null is rejected.

Shown in table 3 are power estimates from testing the null hypothesis in Equation (8) by the different methods from their applications to the    paired count outcomes simulated under the alternative hypothesis in Equation (8). As type I error rates for GLMM (Poisson) were highly biased, power estimates from this method are not meaningful. Among the remaining four tests, the signed-rank test has the lowest power. The paired t-tests, GLMM (NB) and both GEE methods yield comparable power estimates, though both GEE methods and GLMM (NB) appear to perform best with a sample size of at least 25. When    and the sample size is high (more than, say, 50 subjects) all tests have comparable power and correct nominal significance level. Figures 1 and 2 show the power estimates under additional alternative hypotheses. The GLMM (NB) method appears to be less efficient for larger differences in means with sample sizes around 50 when   .

Request permission

Request permission