A semiparametric hypothesis testing procedure for the ROC curve area under a density ratio model

A semiparametric hypothesis testing procedure for the ROC curve area under a density ratio model

Computational Statistics & Data Analysis 50 (2006) 1855 – 1876 www.elsevier.com/locate/csda A semiparametric hypothesis testing procedure for the ROC...

242KB Sizes 0 Downloads 24 Views

Computational Statistics & Data Analysis 50 (2006) 1855 – 1876 www.elsevier.com/locate/csda

A semiparametric hypothesis testing procedure for the ROC curve area under a density ratio model Biao Zhang∗ Department of Mathematics, The University of Toledo, Toledo, OH 43606, USA Received 27 September 2004; received in revised form 2 February 2005; accepted 4 February 2005 Available online 14 March 2005

Abstract We propose a semiparametric Wald statistic to test whether a diagnostic test is capable of discriminating between diseased and nondiseased subjects based on the receiver operating characteristic (ROC) curve area under a two-sample semiparametric density ratio model. The proposed Wald test is constructed on the basis of the maximum semiparametric likelihood estimator of the ROC curve area. The proposed test statistic has an asymptotic chi-squared distribution under the null hypothesis and an asymptotic noncentral chi-squared distribution under local alternatives to the null hypothesis. We present some results on a simulation study and on the analysis of two data sets. © 2005 Elsevier B.V. All rights reserved. Keywords: Binormal model; Chi-squared; Density ratio model; Gaussian process; Local alternative; Logistic regression model; Maximum likelihood; Power; ROC curve; Wald test

1. Introduction The accuracy of a diagnostic test with results which are binary is commonly summarized by the sensitivity and the specificity, defined, respectively, as the probability that the test result is positive given that the subject is truly diseased and the probability that the test result is negative given that the subject is truly nondiseased. The receiver operating characteristic (ROC) curve generalizes notions of sensitivity and specificity from binary tests to nonbinary tests. When the test results are not simply positive or negative, but are ∗ Tel.: +1 419 530 4506; fax: +1 419 530 4720.

E-mail address: [email protected] 0167-9473/$ - see front matter © 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.csda.2005.02.001

1856

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

measured on continuous or ordinal scales, the ROC curve is the most popular approach for assessing the accuracy of a continuous or ordinal-valued diagnostic test and is defined as a plot of the sensitivity versus one minus the specificity across all possible choices of threshold values. It is often useful to summarize the accuracy of a continuous or ordinal test by a single number. One summary measure associated with a ROC curve is the area under the ROC curve (AUC) which conveys important information about the ROC curve. For comprehensive reviews of recent developments in ROC curves see for example Begg (1991), Hsieh and Turnbull (1996), Pepe (2000a), Zhou et al. (2002), and Pepe (2003). Let X1 , . . . , Xn0 denote independent and identically distributed test results from a nondiseased population and, independent of the Xi , let Y1 , . . . , Yn1 be independent and identically distributed test results from a diseased population. If G and F represent, respectively, the distribution functions of X1 and   Y1 , a closed-form expression for the ROC curve is a plot of R(s) = 1 − F G−1 (1 − s) against s ∈ [0, 1]. The area under the ROC curve is given by  1 R(s) ds = P (Y1 > X1 ) . AUC = 0

In the literature, statistical inferences for an ROC curve and its AUC are based either on a fully parametric model, fully nonparametric model, or semiparametric model. In most parametric analyses, F and G are assumed to have normal distributions, but the normality assumption is typically not so robust. Goddard and Hinberg (1990) found that the normal assumption is not a good choice for describing the distributions of blood samples assayed by some diagnostic kits for cancer patients and that one should not assume that the observed data themselves are normal; the sample means and sample variances should not be substituted directly into the estimates for the intercept and the slope in the binormal model. The alternative nonparametric analysis is to leave F and G completely unspecified and to estimate R(s) and AUC by replacing F and G by their corresponding empirical distribution functions based on diseased and nondiseased data. Hsieh and Turnbull (1996) established asymptotic theory for the nonparametric ROC-curve estimator. One disadvantage of the nonparametric analysis is that the estimated ROC curve is not smooth, especially for small sample sizes, so that the clinical interpretation of sensitivity and specificity would be quite different even for close yet different threshold values. Based on kernel estimators of F and G, Lloyd (1998) introduced a smoothed ROC curve which is a kernel-based estimator of R(s). As a compromise between the parametric and nonparametric approaches, the binormal model is a popular semiparametric approach to modeling a ROC curve. In this model, there exists some unknown monotonic transformation of the test results which simultaneously converts the G and F distributions to normal ones, although the raw test results themselves may be decidedly non-normal. Metz and Kronman (1980) studied statistical significance tests for binormal ROC curves, whereas Metz et al. (1984) proposed a new approach for testing the significance of differences between ROC curves measured from correlated data. Metz (1986) considered statistical analysis of ROC data in evaluating diagnostic performance. McClish (1990) studied how to determine a range of false-positive rates for which ROC curves may differ. Dorfman et al. (1992) discussed receiver operating characteristic rating analysis with generalization to the population of readers and cases using the jackknife method. Jiang et al. (1996) proposed a receiver operating characteristic partial area index for

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

1857

highly sensitive diagnostic tests. Toledano and Gatsonis (1996) studied ordinal regression methodology for ROC curves derived from correlated data. Hsieh and Turnbull (1996) proposed a generalized least-squares procedure for estimating parameters under the binormal model. Metz et al. (1998) discussed statistical comparison of two ROC curve estimates obtained from partially paired data sets. Metz et al. (1998) proposed a rank-based method for estimating R(s) by grouping the observed continuous data on the basis of the “truth-state runs” in rank-ordered test results. In exploring robust and efficient estimates of R(s), Li et al. (1999) considered a semiparametric approach in which they modeled F parametrically and modeled G nonparametrically. Zou and Hall (2000) employed a rank-based likelihood to estimate parameters in the ROC curve. By interpreting each point on the ROC curve as being a conditional expectation of a binary random variable, Pepe (2000b) proposed to use generalized linear model methods for binary data to model the ROC curve and to estimate the underlying parameters. Lloyd (2002a, 2002b) considered fitting ROC curves using a specific nonlinear logistic regression model. Qin and Zhang (2003) considered a semiparametric procedure for estimating the ROC curve and established the large sample results for the semiparametric ROC estimator under the following two-sample semiparametric density ratio model (1.2). Let D = 1 or 0 denote the disease or nondisease status. For a given test result X = x, the logistic regression model is   exp ∗ +  r(x)  , P (D = 1|X = x) = (1.1) 1 + exp ∗ +  r(x) where ∗ is a scalar parameter,  is a p × 1 vector parameter, and r(x) is a p × 1 smooth vector function of x. Then F (x) = P (X x|D = 1) and G(x) = P (X x|D = 0). Let f (x) and g(x) be the density functions corresponding to F (x) and G(x). Qin and Zhang (1997) showed that model (1.1) is equivalent to the following two-sample semiparametric density ratio model in which the density functions f (x) and g(x) are linked by an  two unknown  “exponential tilt” exp  +  r(x) : X1 , . . . , Xn0 are independent with density function g(x),   Y1 , . . . , Yn1 are independent with density function f (x) = exp  +  r(x) g(x), (1.2) where  = ∗ + log[{1 − P (D = 1)}/P (D = 1)]. Kay and Little (1987) discussed various versions of the density ratio  model  for some conventional distributions. In most applications, r(x) = x or r(x) = x, x 2 . For r(x) = x, model (1.2) encompasses many common distributions including two exponential distributions with different means and   two normal distributions with a common variance but different means; for r(x) = x, x 2 , model (1.2) includes two normal distributions with different means and variances. In some applications, we found  that model (1.2)with r(x)  = log x, which contains two log normal distributions log N 1 , 2 and log N 2 , 2 with 1  = 2 , provides a good fit to the observed data. For a given choice of r(x), we can use the Kolmogorov–Smirnov-type statistic of Qin and Zhang (1997) to test the validity of model (1.2). If more than one form of r(x) is possible for any particular situation, we would prefer simple models to complex ones that fit our data almost equally well according to the principle of parsimony.

1858

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

Under model (1.2), the ROC curve R(s) and its AUC can be expressed as  s    R(s) = exp  +  r G−1 (1 − t) dt, 0

  AUC =

  I (y > x) exp  +  r(y) dG(x) dG(y),

(1.3)

where I (A) for any set (event) A denotes the indicator function of A; that is, I (A) assumes the value 1 on A and 0 on A’s complement Ac . Note that the representation of R(s) was deduced in Qin and Zhang (2003). The AUC describes the inherent ability of a diagnostic test to discriminate between healthy and diseased individuals. Our focus of attention in this paper is to test whether a diagnostic test is capable of discriminating between diseased and nondiseased subjects based on the AUC under model (1.2). We propose a semiparametric approach to testing the null hypothesis H0 : AUC = AUC0 versus H1 : AUC  = AUC0 under model (1.2) with AUC0 being any specific hypothesized value. We anticipate that our semiparametric hypothesis testing procedure would be more robust than a fully parametric approach and would be more powerful than a fully nonparametric approach. This paper is organized as follows. In Section 2, we propose our test statistic and establish its asymptotic null distribution under model (1.2). In Section 3, we investigate the power of the test statistic theoretically and via simulation by considering local alternatives to H0 : AUC = AUC0 under model (1.2). In Section 4, we apply the proposed test statistic to two real examples. Proofs of the main theoretical results are provided in the appendix.

2. Construction of test statistics

Let {T1 , . . . , Tn } denote the combined test results X1 , . . . , Xn0 ; Y1 , . . . , Yn1 with n = n0 +n1 . Moreover, let {t1 , . . . , tn } represent the observed values of {T1 , . . . , Tn } . Following Qin and Zhang (2003), let    n 1 exp ˜ + ˜ r (Ti ) I (Ti t) F˜ (t) =   , ˜  r (Ti ) n0 1 +  exp  ˜ +  i=1 n I (Ti t) 1 ˜ G(t) =  ,  n0 ˜ + ˜ r (Ti ) i=1 1 +  exp 

  where  = n1 /n0 and ˜ , ˜ is the solution to the following system of score equations:   n  exp  +  r (ti ) j(, )   = 0, = n1 − j 1 + exp  +  r (ti ) i=1

  n1 n    exp  +  r (ti ) j(, )  r (ti ) = 0.  = r yj − j 1 +  exp  +  r (ti ) j =1

i=1

(2.1)

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

1859

Here (, ) stands for the semiparametric profile log likelihood function of (, ).As shown  ˜ G ˜ is the maximumsemiparametric likelihood estimator of in Qin and Zhang (1997), ˜ , ,   ˜ G ˜ maximizes the following semiparametric full likelihood (, , G) in the sense that ˜ , , function of (, , G): L(, , G) =

n0

dG (xi )

n1

     exp  +  r yj dG yj .

j =1

i=1

According to the invariance property of maximum likelihood estimation, the maximum semiparametric likelihood estimator of AUC in (1.3) under model (1.2) is given by      ˜ ˜  AUC = I (y > x) exp ˜ + ˜ r(y) dG(x) dG(y) =

n n

      p˜ i p˜ j exp ˜ + ˜ r Tj I Tj > Ti ,

(2.2)

i=1 j =1

where p˜ i = Let

1 1 ,  n0 1 +  exp ˜ + ˜  r (T ) i

n1   1 ˆ F (t) = I Yj t n1

and

j =1

i = 1, . . . , n.

n0 1 ˆ G(t) = I (Xi t) n0 i=1

be, respectively, the nonparametric maximum likelihood estimators of F and G. Then the maximum nonparametric likelihood estimator of AUC in (1.3) is given by   n0 n1   1 ˆ  A UC = I (y > x)G(x) dFˆ (y) = I Yj > Xi . n0 n1 



i=1 j =1

Throughout this paper, let 0 , 0 be the true value of (, ) under model (1.2). Moreover, write    t exp 0 + 0 r(x)   dG(x), A0 = A0 (∞), A0 (t) =  −∞ 1 +  exp 0 + 0 r(x)    t exp 0 + 0 r(x)   r(x) dG(x), A1 = A1 (∞), A1 (t) =  −∞ 1 +  exp 0 + 0 r(x)      exp 0 + 0 r(x) A0 A1    . A2 = r(x)[r(x)] dG(x), A = A1 A2 1 +  exp 0 + 0 r(x) The following theorem whose proof is given in the appendix demonstrates that the max imum semiparametric likelihood estimator A UC is asymptotically more efficient than the  maximum nonparametric likelihood estimator A UC.

1860

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

Theorem 1. Let 0 < a < b < 1 be given. Suppose that F and  G have continuous positive densities f and g, respectively, on G−1 (a) − , G−1 (b) +  for some  > 0. If A is positive   d   √   definite, then under model (1.2) we have n A UC − AUC −→ N 0, 2 0 , 0 , G and  d     √   n A UC−AUC −→ N 0, 2 (F, G) , where the asymptotic variances 2 0 , 0 , G and 2 (F, G) are defined in (A.1) and (A.3) of the appendix and satisfy  ∞ ∞   2 (F, G) − 2 0 , 0 , G = J (s, t) dG(s) dG(t) 0, (2.3) −∞ −∞

where J (s, t) =

 



1+ 1 +  exp 0 + 0 r(s) 1 +  exp 0 + 0 r(t)       −1 A0 (t)  × A0 (s ∧ t) − A0 (s), A1 (s) A A1 (t)

with s ∧ t = min(s, t). Remark 1. The condition under which A is positive definite is satisfied when r(x) =   r1 (x), . . . , rp (x) with r1 (x), . . . , rp (x) being p functionally independent functions. In   particular, if r(x) = x, x 2 , . . . , x p , then A is positive definite. Since the choice r(x) = x   or r(x)= x, x 2 is most frequently encountered in practice, the condition that A is positive definite does not restrict the results of Theorem 1 to a large extent. We now consider the problem of testing H0 : AUC = AUC0 versus H1 : AUC  = AUC0 under the two-sample semiparametric density ratio model (1.2). When AUC0 = 0.5, the null hypothesis H0 : AUC = 0.5 represents a noninformative test without discrimination ability so as to have a sensitivity = 1 − specificity. The ROC curve corresponding to such a noninformative test lies along the 45◦ diagonal line with R(s) = s and has an area of  AUC = ˜ G ˜ 0.5. To formulate our semiparametric test statistics under model (1.2), let ˜ 2 = 2 ˜ , ,     2 be the maximum semiparametric likelihood estimator of  0 , 0 , G with 0 , 0 , G   ˜ G ˜ . Then it is seen from (A.3) and the definitions of G, ˜ F˜ , and p˜ i that replaced by ˜ , , n n        ˜ G ˜ = 1+ ˜ 2 = 2 ˜ , , p˜ i p˜ j F˜ Ti ∧ Tj − F˜ (Ti ) F˜ Tj  i=1 j =1

+ (1 + )

n n i=1 j =1

      p˜ i p˜ j exp ˜ + ˜ r (Ti ) exp ˜ + ˜ r Tj

     ˜ Ti ∧ Tj − G ˜ (Ti ) G ˜ Tj × G  n n       1 +      − p˜ i p˜ j 1+ exp ˜ +˜ r (Ti ) 1+ exp ˜ +˜ r Tj  i=1 j =1

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876





× A˜ 0 Ti ∧ Tj =



  − A˜ 0 (Ti ) , A˜ 1 (Ti ) A˜ −1



1861

   A˜ 0 Tj   A˜ 1 Tj

n n      1 +  p˜ i p˜ j F˜ Ti ∧ Tj − F˜ (Ti ) F˜ Tj  i=1 j =1

+ (1 + )

n n

      p˜ i p˜ j exp ˜ + ˜ r (Ti ) exp ˜ + ˜ r Tj

i=1 j =1

     ˜ Ti ∧ Tj − G ˜ (Ti ) G ˜ Tj × G      n n    A˜ 0 Tj 1 +  1 ˜   −1 − A0 Ti ∧ Tj − A˜ 0 (Ti ) , A˜ 1 (Ti ) A˜ ,   2  n0 A˜ 1 Tj i=1 j =1 (2.4) where

   exp ˜ + ˜ r (Ti ) A˜ 0 (t) = p˜ i   I (Ti t) , A˜ 0 = A˜ 0 (∞),  1 +  exp ˜ + ˜ r (Ti ) i=1    n exp ˜ + ˜ r (Ti ) ˜ A1 (t) = p˜ i   r (Ti ) I (Ti t) , A˜ 1 = A˜ 1 (∞), ˜  r (Ti ) 1 +  exp  ˜ +  i=1      n exp ˜ + ˜ r (Ti ) A˜ 0 A˜ 1  ˜ ˜ A2 = . p˜ i   r (Ti ) [r (Ti )] , A = ˜  A1 A˜ 2 1 +  exp ˜ + ˜ r (Ti ) n

i=1

  ˜ G ˜ is a consistent It can be shown that the estimated asymptotic variance ˜ 2 = 2 ˜ , ,   estimator of 2 0 , 0 , G under model (1.2). According to Theorem 1, a semiparametric Wald-type statistic for testing H0 : AUC = AUC0 versus H1 : AUC  = AUC0 under model (1.2) is given by W˜ n =

 √   n A UC − AUC0 ˜

or

W˜ n2 =

 2  n A UC − AUC0 ˜ 2

.

(2.5)

Clearly, large observed values of W˜ n2 indicate evidence against H0 : AUC = AUC0 under   d model (1.2). If 2 0 , 0 , G > 0, then it follows from Theorem 1 that W˜ n2 −→ 21 under model (1.2). We reject the null hypothesis H0 : AUC = AUC0 in favor of the alternative 2 2 ˜2 hypothesis H1 : AUC  2   = AUC0 at level if Wn > 1 (1 − ), where 1 (1 − ) is such that 2 P 1 1 (1 − ) = 1 − . On the other hand, according to Theorem 1, a nonparametric Wald-type statistic for testing H0 : AUC = AUC0 versus H1 : AUC  = AUC0 is given by  √   n A UC − AUC0 Wˆ n = ˆ

or

Wˆ n2

=

 2  n A UC − AUC0 ˆ 2

,

1862

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

  ˆ is the maximum nonparametric likelihood estimator of 2 (F, G) with where ˆ 2 = ˆ 2 Fˆ , G   ˆ and is given by (F, G) replaced by Fˆ , G 2

ˆ = ˆ

2



 ˆ = Fˆ , G

n0 n0      n ˆ Xi ∧ Xj − Fˆ (Xi ) Fˆ Xj F n20 n1 i=1 j =1

+

n1 n1      n ˆ (Yi ) G ˆ Yi ∧ Yj − G ˆ Yj . G 2 n0 n1 i=1 j =1

d When 2 (F, G) > 0, it follows from Theorem 1 that Wˆ n2 −→ 21 regardless of whether or not   model (1.2) is valid. Since A UC is more efficient than A UC by Theorem 1, we anticipate that the semiparametric Wald statistic W˜ n2 is more powerful than the nonparametric Wald statistic Wˆ n2 for testing H0 : AUC = AUC0 versus H1 : AUC  = AUC0 under model (1.2), as illustrated in a simulation study of the next section.

Remark 2. More generally, we can determine, under model (1.2), whether a diagnostic test has the ability of discriminating between diseased and nondiseased subjects based on the ROC curve area under a restricted range of values of one minus the specificity. For any particular values s1 and s2 of 1−specificity, the null versus alternative hypotheses are expressed in terms of the partial area as H0 : AUC H1 : AUC (s1 , s2 )  = 0 (s1 , s2 ) versus   s (s1s, s2 )=AUC  AUC0 (s1 , s2 ),where AUC (s1 , s2 ) = s12 0 exp  +  r G−1 (1 − t) dt ds is the partial area under model (1.2) and AUC0 (s1 , s2 ) is any specific hypothesized value of AUC (s1 , s2 ) . When s1 = 0 and s2 = 1, AUC (s1 , s2 ) reduces to the total area AUC in (1.3). Similar to (2.5), a semiparametric Wald statistic for testing H0 : AUC (s1 , s2 ) = AUC0 (s1 , s2 ) versus H1 : AUC (s1 , s2 )  = AUC0 (s1 , s2 ) is given by  2  n A UC (s1 , s2 ) − AUC0 (s1 , s2 )   W˜ n2 (s1 , s2 ) = ,  A  Var UC (s1 , s2 )   −1  ˜ r G ˜ (1−t) dt ds is the maximum semiparametexp  ˜ +  s1 0    A  ric likelihood estimator of AUC (s1 , s2 ) under model (1.2) and Var UC (s1 , s2 ) is the esti   A   mated asymptotic variance of A UC (s1 , s2 ). The explicit expression of Var UC (s1 , s2 ) can be derived in a manner similar to (2.4).  where A UC (s1 , s2 ) =

 s2  s

Remark 3. The proposed semiparametric Wald test procedure can be extended to the case of comparing two or more independent ROC areas under model (1.2). In the case of comparing two such ROC areas, a natural test statistic is based on the difference in maximum semiparametric likelihood estimators of AUC. Suppose that there are two types of tests, labeled A and B, that are applied to subjects. We denote the two ROC areas by AUCA and AUCB and their maximum semiparametric likelihood estimators under model (1.2) by     A UCA and A UCB . If A UCA and A UCB are derived from two independent samples, then a semiparametric Wald-type statistic for testing H0 : AUCA = AUCB versus H1 : AUCA  =

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

1863

2      AUCB is given by Dn = A UCA − A UCB / n−1 ˜ 2A + n−1 ˜ 2B , where nA and nB are A  B      sample sizes and ˜ 2A and ˜ 2B are versions of ˜ 2 in (2.4); both nA , ˜ 2A and nB , ˜ 2B correspond to tests A and B. The null hypothesis H0 : AUCA = AUCB is tested by comparing the value of Dn with a 21 distribution. Remark 4. When testing the differences between two or more ROC areas, the proposed semiparametric test approach is restricted to those estimates obtained from independently sampled data sets. On the other hand, some nonparametric and semiparametric approaches proposed previously apply also to correlated data sets; see for example Metz et al. (1984), Wieand et al. (1989), Dorfman et al. (1992), Jiang et al. (1996), Toledano and Gatsonis (1996), Zhou and Gatsonis (1996), and Metz et al. (1998).

3. Power considerations In this section we investigate the power of the proposed test statistic W˜ n2 theoretically and via simulation by considering the following local alternatives to model (1.2): X1 , . . . , Xn0 are independent with density g(x),   Y1 , . . . , Yn1 are independent with density f (x, ) = exp  +  r(x) h(x, )g(x), (3.1)     where = 1 , . . . , q ranges over a neighborhood of a point 0 = 01 , . . . , 0q in R q   and h(x, ) is, for fixed x, a known function from R q to R + for which h x, 0 = 1 for each x and with partial derivative s(x, ) = jh(x, )/j . Note that model (3.1) reduces to model (1.2) when = 0 . Throughout this section, let F (x, ) be the distribution function corresponding to f (x, ). Write     exp 0 + 0 r(x)   s x, 0 dG(x), V0 = V0 (∞), V0 (t) =  −∞ 1 +  exp 0 + 0 r(x)      exp 0 + 0 r(x)   s x, 0 [r(x)] dG(x), V1 =  1 +  exp 0 + 0 r(x)      B0 = G(x)s x, 0 exp 0 + 0 r(x) dG(x), 

 B(t) =

t

t −∞

    s x, 0 exp 0 + 0 r(x) dG(x),

    V0 , m(t) =  V0 (t) − (A0 (t), A1 (t)) A−1 V1 (t) =

 1  1 +  exp 0 + 0 r(t) m(t) − B  (t), 

 =

∞ −∞

(t) dG(t).

1864

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

Additionally, we assume that the function h(x, ) or its partial derivative s(x, ) is chosen such that all the components of V1 and B0 are finite. Under model (3.1), the AUC is equal to     AUC( ) = G(t) dF (t, ) = G(t) exp  +  r(t) h(t, ) dG(t)  = G(t)h(t, ) dF (t). (3.2)   When = 0 , it can be shown by (1.3) and (3.2) that AUC 0 = AUC under model (1.2).     For fixed = , . . . , q ∈ R q , let n = n1 , . . . , nq = 0 + n−1/2 [1 + o(1)] as   n → ∞. Under the sequence of parameter values 0 , 0 , n in model (3.1), we have   AUC n − AUC =



  G(t)h t, n dF (t) −

 G(t) dF (t)



     G(t) h t, n − h t, 0 dF (t)      1  =√ G(t)s t, 0 dF (t) + o n−1/2 n 1  = √ B0 [1 + o(1)]. n

=

3.1. Local asymptotic power We now consider the local asymptotic power of the proposed semiparametric Wald test of H0 : AUC=AUC0 with local alternatives H1 : AUCn =AUC0 +n−1/2  B0 [1+o(1)] under model (3.1),where ∈ R q . The following theorem establishes the large-sample distribution of the proposed semiparametric Wald statistic W˜ n2 for testing H0 : AUC = AUC0 under the  sequence of parameter values 0 , 0 , n ,where n = 0 + n−1/2 [1 + o(1)] as n → ∞ for fixed ∈ R q .    d Theorem 2. Under the conditions of Theorem 1,we have W˜ n −→ N , 2 0 , 0 , G and     d W˜ n2 −→ 21 2 under model(3.1) with (, , ) = 0 , 0 , 0 as n → ∞, where 2 =     2 2 /2 0 , 0 , G and 21 2 is a noncentral chi-squared random variable with one degree of freedom and noncentrality parameter 2 . The proof of Theorem 2 is given in the appendix. The asymptotic null and alternative distributions of W˜ n2 , presented in Theorems 1 and 2, can be employed to obtain critical values of the proposed semiparametric Wald test and power against various local alternatives AUCn  = AUC0 by numerical integration, although explicit computation is unfortunately somewhat complicated.

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

1865

3.2. Comparative study In the following we present a small simulation study of the finite sample performance of the proposed semiparametric Wald statistic W˜ n2 for testing H0 : AUC = AUC0 versus H1 : AUC  = AUC0 under model (1.2). In our simulation study, we first assume that g(x) is the density function of a N 0 , 20 distribution and f (x) is the density function       of a N 1 , 21 distribution. Then model (1.2) holds with r(x) = x, x 2 ,  = 1 , 2 , and     21 0 1 20 1 0 1 1 1  = log + − 2 , 1 = 2 − 2 , 2 = − 2 . 1 2 20 2 20 1 1 0 1  √  The ROC curve area under this binormal model is given by AUC = a/ 1 + b2 , where   a = 1 − 0 /1 and b = 0 /1 . Here (·) is the standard normal distribution function. According to Zhou et al. (2002, pp. 122 and 140), the parametric Wald statistic Wn2 for testing H0 : AUC = AUC0 versus H1 : AUC  = AUC0 under the binormal model is given by     aˆ 2 Wn =  − AUC0 ˆ 2 , 2 ˆ 1+b where       y¯ − x¯ sx  aˆ + dˆ 2 Var  bˆ + 2cˆdˆ Cov  a, ˆ bˆ , , bˆ = , ˆ 2 = cˆ2 Var sy sy       exp − aˆ 2 /2 1 + bˆ 2 aˆ bˆ exp − aˆ 2 /2 1 + bˆ 2 ˆ=− cˆ = , , d     3  2 ˆ 2 1 + bˆ 2 2 1 + b

aˆ =

    n0 aˆ 2 + 2 + 2n1 bˆ 2  aˆ = Var , 2n0 n1

ˆ2    bˆ = (n1 + n0 ) b , Var 2n0 n1

  aˆ bˆ  a, Cov ˆ bˆ = . 2n1

Here x¯ and sx2 are, respectively, the sample mean and the sample variance of X1 , . . . , Xn0 , and y¯ and sy2 are, respectively, the sample mean and the sample variance of Y1 , . . . , Yn1 . The parametric Wald statistic Wn2 is seldom used in practice because it is not robust to nonnormality. Instead, Metz et al. (1998) LABROC method and its corresponding ROCKIT software is mostly used because of its robustness to nonnormality. The LABROCbased semiparametric Wald statistic for testing H0 : AUC = AUC 0 versus H1 : AUC  =      2 =  A   UCM or WM UCM − AUC0 / Var AUC0 for continuous data is given by WM = A  2    A    A UCM − AUC0 /Var UC is the LABROC ROC curve area estiUCM , where A  M   A  UCM is its estimated standard error. mator of AUC of Metz et al. (1998) and Var A FORTRAN program (available at http://www-radiology.uchicago.edu) implementing the LABROC4 algorithm can be employed to calculate the LABROC ROC curve area estimator  A UCM and its estimated standard error.

1866

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

In our simulation study, we assume that 0 = 0 and 0 = 1 = 1 so that g(x) is the standard normal density function and f (x)=g(x −1 ) is the density function of a N 1 , 1     distribution. Moreover, we assume that f x, n is the density function of a N 1 , 2n   distribution with 2n = 1/ 1 − 2n−1/2  for some fixed  ∈ R such that 2n > 0. Then   model (1.2) holds with  = −21 /2 and  = 1 , and model (3.1) holds with h x, n =     exp n1 + n2 x + n3 x 2 where n = n1 , n2 , n3 and       21 1 1 1 1 1 2 1 − 2 − log n , n2 = 1 1− 2 . − 1 , n3 =

n1 = 2 n 2 2n 2 n It is easy to verify that n= 0 + n−1/2 {1 + o(1)} as n → ∞ with 0 = (0, 0, 0)  and =  21 − 1, −21 , 1 . Our aim is to compare the performances of the proposed 2, semiparametric Wald statistic W˜ n2 , the LABROC-based semiparametric Wald statistic WM 2 2 the nonparametric Wald statistic Wˆ n , and the parametric Wald statistic Wn by examining their powers against some local alternatives   = 0 under model (3.1). We considered  = 0, 2.0, 4.0 and sample sizes of (n0 , n1 ) = (40, 60) and (n0 , n1 ) = (60, 40). Furthermore, we let 1 = 1 be fixed so that  = −0.5,  = 1, a = 1, b = 1, and AUC0 = 0.760. Note that, for  = 0, 2.0, 4.0, we have n = 1.0, 1.291, 2.236 and AUCn = 0.760, 0.785, 0.819 when n = 100. For each pair (n0 , n1 ) and each value  of , we generated  1000 indepen  dent sets of combined random samples from the N 0 , 20 , N 1 , 21 , and N 1 , 2n distributions. The simulation results are summarized in Table 1. It is seen that the achieved significance 2,W ˆ n2 , and Wn2 are quite close to the corresponding nominal significance levels of W˜ n2 , WM 2 ,W ˆ n2 , and Wn2 are getting larger as  moves away from 0. levels, and the powers of W˜ n2 , WM Our simulation results also reveal that the powers of W˜ n2 are greater than or equal to those 2 and W ˆ n2 in all cases, and the powers of Wn2 are slightly larger than those of W˜ n2 , W 2 , of WM M and Wˆ n2 when  = 4.0 and are comparable to those of W˜ n2 when  = 2.0. In summary, our simulation study indicates that the proposed semiparametric Wald statistic W˜ n2 is superior to the nonparametric Wald statistic Wˆ n2 and is quite comparable to the LABROC-based 2 and the parametric Wald statistic W 2 , in terms of their semiparametric Wald statistic WM n power performances. In Table 2, we report simulation results of the finite sample performance of the proposed  semiparametric ROC curve area estimator A UC under the same setup as that for the above four test statistics when  = 0. Our aim is to compare the biases and standard deviations  of the proposed semiparametric ROC curve area estimator A UC, the LABROC ROC curve  area estimator A UCM of Metz et al. (1998), the nonparametric ROC  estimator  curve area 2 ˆ  AUC, and the parametric ROC curve area estimator AUC = a/ ˆ 1 + b . In Table 2, Bias and SD stand for, respectively, the average of 1000 biases and the sample standard deviation of 1000 estimates. Moreover, MS represents the mean square deviation of an estimate. Additionally, we use SE to represent the square root of the average of 1000  variance estimates of an estimate. Table 2 reveals that A UC produces a smaller standard  deviation than A UC and a larger standard deviation than AUC. Furthermore, the standard   deviation of A UC is comparable to that of A UCM . While the absolute values of the biases

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

1867

Table 1 2 ,W ˆ n2 , and Wn2 under the two-sample normal model Achieved significance levels and powers of W˜ n2 , WM



(n0 , n1 )

n

AUCn

Nominal level

Power W˜ n2

2 WM

Wˆ n2

Wn2

0.0 0.0 0.0

(40, 60) (40, 60) (40, 60)

1.000 1.000 1.000

0.760 0.760 0.760

0.10 0.05 0.01

0.115 0.063 0.019

0.115 0.066 0.019

0.117 0.066 0.020

0.107 0.057 0.015

2.0 2.0 2.0

(40, 60) (40, 60) (40, 60)

1.291 1.291 1.291

0.785 0.785 0.785

0.10 0.05 0.01

0.148 0.089 0.022

0.146 0.073 0.014

0.140 0.082 0.022

0.143 0.083 0.021

4.0 4.0 4.0

(40, 60) (40, 60) (40, 60)

2.236 2.236 2.236

0.819 0.819 0.819

0.10 0.05 0.01

0.640 0.494 0.254

0.621 0.485 0.242

0.596 0.459 0.222

0.663 0.515 0.252

0.0 0.0 0.0

(60, 40) (60, 40) (60, 40)

1.000 1.000 1.000

0.760 0.760 0.760

0.10 0.05 0.01

0.111 0.057 0.015

0.100 0.058 0.019

0.112 0.056 0.014

0.105 0.047 0.015

2.0 2.0 2.0

(60, 40) (60, 40) (60,40)

1.291 1.291 1.291

0.785 0.785 0.785

0.10 0.05 0.01

0.144 0.091 0.025

0.141 0.077 0.023

0.144 0.079 0.024

0.135 0.081 0.022

4.0 4.0 4.0

(60, 40) (60, 40) (60, 40)

2.236 2.236 2.236

0.819 0.819 0.819

0.10 0.05 0.01

0.509 0.379 0.169

0.503 0.372 0.159

0.473 0.353 0.157

0.534 0.399 0.177

Table 2    UC, and AUC under Biases, standard deviations, standard errors, and mean square deviations of A UC, A UCM , A the two-sample normal model Estimator

(n0 , n1 )

Bias

SD

SE

MS

 A UC  A UCM  A UC AUC

(40, 60) (40, 60) (40, 60) (40, 60)

−0.00179 0.00492 −0.00003 −0.00096

0.04726 0.04724 0.04801 0.04650

0.04722 0.04754 0.04830 0.04735

0.00224 0.00226 0.00231 0.00216

 A UC  A UCM  A UC AUC

(60, 40) (60, 40) (60, 40) (60, 40)

−0.00194 0.00201 −0.00031 −0.00107

0.04714 0.04773 0.04817 0.04633

0.04723 0.04747 0.04829 0.04737

0.00223 0.00228 0.00232 0.00215

   of A UC are slightly larger than those of A UC and AUC, but smaller than those of A UCM ,    the mean square deviations of A UC are smaller than those of A UCM and A UC but larger than that of AUC. As for the variance estimates, Table 2 indicates that the standard errors    UC work well and work better than those of AUC for each value of A UC, A UCM , and A

1868

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

of (n0 , n1 ) . In addition, the ratios of estimated biases to standard errors, Bias/SE, are all within 0.11 in absolute value for each estimate and each value of (n0 , n1 ) . According to  the rule of thumb described in Efron and Tibshirani (1993, p. 128), the biases of A UC,   A UCM , A UC, and AUC are due to sampling variation and can be ignored. In summary, our simulation study indicates that the parametric estimator AUC is, as anticipated, better than  the proposed semiparametric estimator A UC and the LABROC ROC curve area estimator   AUCM , which are in turn superior to the nonparametric estimator A UC, in terms of mean  squared deviation, and that the proposed semiparametric estimator A UC is more efficient  than the nonparametric estimator A UC. 3.3. Robustness Next, we study how well the proposed semiparametric Wald statistic W˜ n2 and the proposed  semiparametric ROC curve area estimator A UC work, under the same scenario as that in Tables 1 and 2, when the underlying data violate the normal assumption. To this end, −x we assume   that g(x)  −1= −e x I (x > 0) is the standard exponential E(1) density function, f (x)  =  /() x e I (x > 0) is the density function of a (, )√distribution, and f x, n is the density function of a  (, n ) distribution with n = +/ n for some fixed  log  ∈ R such that n > 0. Then model (1.2) holds with r(x)=(x,  x) , =  log −log   (),  and  = (1 − ,  − 1) , and model (3.1) holds with h x, n = exp n1 + n2 x where  

n = n1 , n2 and   n , n2 = + 1 − n .

n1 = a log It is easy to verify that n = 0 + n−1/2 {1 + o(1)} as n → ∞ with 0 = (0, 1) and   =  a −1 , −1 . For  = 2, = 1, each value of  = 0, −1.5, −3.0, each pair of (n0 , n1 ) = (40, 60), (60, 40), we generated 1000 independent sets of combined random samples from the E(1), (, ), and (, n ) distributions. It is seen that, for  = 0, −1.5, −3.0, we have n = 1.0, 0.85, 0.7 and AUCn = 0.750, 0.789, 0.831 when n = 100. The simulation results 2 ,W ˆ n2 , Wn2 and in Table 4 for A    are reported in Table 3 for W˜ n2 , WM UC, A UCM , A UC, AUC. 2 2 2 It is seen from Table 3 that the achieved significance levels of W˜ n , WM , Wˆ n are closer to their corresponding nominal significance levels than those of Wn2 in most cases, and the powers 2,W ˆ n2 are all significantly larger than those of Wn2 . Furthermore, the powers of of W˜ n2 , WM 2 ˜ Wn are larger than those of Wˆ n2 except for a single case with (n0 , n1 ) = (60, 40),  = −1.5, 2 . Table 4 shows that and nominal level equal to 0.10, and are comparable to those of WM the biases and mean square deviations of the parametric ROC curve area estimator AUC are both significantly larger than those of the proposed semiparametric ROC curve area   estimator A UC, the LABROC ROC curve area estimator A UCM , and the nonparametric   ROC curve area estimator AUC. Furthermore, AUC has smaller standard deviations and   mean square deviations than A UCM , A UC, and AUC. Moreover, the ratios of estimated   biases to standard errors, Bias/SE, are all within 0.07 in absolute value for A UC, A UCM ,  A UC, whereas |Bias|/SE = 0.60 for AUC when (n0 , n1 ) = (40, 60) and |Bias|/SE = 0.62

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

1869

Table 3 2 ,W ˆ n2 , and Wn2 under the two-sample exponential-gamma Achieved significance levels and powers of W˜ n2 , WM model



(n0 , n1 )

n

AUCn

Nominal level

Power W˜ n2

2 WM

Wˆ n2

Wn2

000 000 000

(40, 60) (40, 60) (40, 60)

1.00 1.00 1.00

0.750 0.750 0.750

0.10 0.05 0.01

0.113 0.062 0.024

0.113 0.061 0.029

0.107 0.057 0.020

0.150 0.085 0.021

−150 −150 −150 −300 −300 −300

(40, 60) (40, 60) (40, 60) (40, 60) (40, 60) (40, 60)

0.85 0.85 0.85 0.70 0.70 0.70

0.789 0.789 0.789 0.831 0.831 0.831

0.10 0.05 0.01 0.10 0.05 0.01

0.286 0.214 0.097 0.647 0.545 0.344

0.284 0.199 0.094 0.645 0.529 0.353

0.285 0.207 0.089 0.644 0.532 0.336

0.125 0.076 0.025 0.298 0.210 0.091

000 000 000

(60, 40) (60, 40) (60, 40)

1.00 1.00 1.00

0.750 0.750 0.750

0.10 0.05 0.01

0.119 0.056 0.017

0.120 0.071 0.024

0.113 0.059 0.017

0.122 0.066 0.014

−150 −150 −150 −300 −300 −300

(60, 40) (60, 40) (60, 40) (60, 40) (60, 40) (60, 40)

0.85 0.85 0.85 0.70 0.70 0.70

0.789 0.789 0.789 0.831 0.831 0.831

0.10 0.05 0.01 0.10 0.05 0.01

0.271 0.194 0.084 0.629 0.528 0.328

0.267 0.186 0.105 0.648 0.549 0.349

0.275 0.190 0.082 0.623 0.514 0.320

0.086 0.043 0.016 0.224 0.140 0.051

Table 4    Biases, standard deviations, standard errors, and mean square deviations of A UC, A UCM , A UC, and AUC under the two-sample exponential-gamma model Estimator

(n0 , n1 )

Bias

SD

SE

MS

 A UC  A UCM  A UC AUC

(40, 60) (40, 60) (40, 60) (40, 60)

−0.00039 0.00307 0.00068 −0.02927

0.04957 0.04989 0.05010 0.05026

0.04828 0.04870 0.04977 0.04875

0.00246 0.00250 0.00251 0.00338

 A UC  A UCM  A UC AUC

(60, 40) (60, 40) (60, 40) (60, 40)

−0.00254 0.00299 −0.00125 −0.03253

0.04833 0.04896 0.04893 0.04810

0.04712 0.04752 0.04847 0.05238

0.00234 0.00241 0.00240 0.00337

for AUC when (n0 , n1 ) = (60, 40). Thus, the bias of AUC is not due to sampling variation and is a systematic bias. In conclusion, our simulation study indicates that the proposed semiparametric procedure is more efficient than the nonparametric procedure, is more robust

1870

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

than the normal-based parametric procedure, and is comparable to the LABROC-based semiparametric procedure.

4. Examples In this section, we consider the problem of testing H0 : AUC=AUC0 versus H1 : AUC  = AUC0 under the density ratio model (1.2) by applying the proposed semiparametric Wald test to two real data sets. Example 1. Wieand et al. (1989) reported a case-control study in which sera from n0 = 51 control patients with pancreatitis and n1 = 90 case patients with pancreatic cancer were assayed at the Mayo Clinic with a cancer antigen (CA-125) and with a carbohydrate antigen (CA19-9). Both the CA-125 and CA19-9 antigens are measured on a continuous positive scale. The raw data is available at http://www.fhcrc.org/labs/pepe/book/data/wiedat2b.dta. On the basis of the test results associated only with CA-125, we consider the problem of whether the antigen CA-125 has any discrimination ability  by testing H 0 : AUC = 0.5 versus H1 : AUC  = 0.5 under model (1.2). For r(x) = log x, (log x)2 , the system of score equations in (2.1) yields ˜ = −4.33700 and ˜ = (2.06145, −0.18325) . Furthermore, the Kolmogorov–Smirnov-type   statistic of Qin and Zhang (1997) for testing model (1.2) with r(x) = log x, (log x)2 is found to be  = 0.77997 with the observed P -value equal to 0.316 based on 1000 bootstrap replications. Thus, the semiparametric density ra tio model (1.2) with r(x) = log x, (log x)2 provides a good fit to the observed data. If   we denote w(x) = exp −4.33700 + 2.06145 × log x − 0.18325 × (log x)2 , then under   model (1.2) with r(x) = log x, (log x)2 and according to (2.2) and (2.4), the maximum  semiparametric likelihood estimate of AUC is calculated as A UC = 0.69216 and its estimated asymptotic variance and standard error are respectively equal to ˜ 2 /n = 0.00204 and √ √ / ˜ n = 0.00204 = 0.04516. It follows from (2.5) that the proposed semiparametric Wald statistic for testing the null hypothesis H0 : AUC=0.5 versus the alternative hypothesis H1 :  √   AUC  = 0.5 is identical to W˜ n = n A UC − 0.5 /˜ = (0.69216 − 0.5)/0.04516 = 4.25500 or W˜ n2 = 18.10504 with P -value equal to 0.00002, which is statistically significant. Since  the maximum semiparametric likelihood estimator of AUC is A UC = 0.69216, we conclude that the antigen CA-125 has some ability of discriminating between patients with and without pancreatic cancer. Example 2. Venkatraman and Begg (1996) discussed a melanoma data set in which the definitive diagnosis of a pigmented lesion suspected of being a melanoma involves a biopsy. There are two systems that can be used to evaluate suspicious lesions. One is the clinical score system given by doctors, and the other system is the dermoscope. The raw data is listed in Table 4 of Venkatraman and Begg (1996, pp. 842 and 843) with n0 = 51, n1 = 21, and n = n0 + n1 = 72. Based on the data only from the clinical score system, we consider the problem of testing whether the clinical score system has any discrimination ability at all ˜ under model (1.2). For model (1.2) with r(x)=x, we have (˜, )=(0.88725, 1.00015), and the Kolmogorov–Smirnov type statistic of Qin and Zhang (1997) for testing model (1.2)

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

1871

equals  = 0.28030 with the observed P -value equal to 0.430 based on 1000 bootstrap replications. Thus, the semiparametric density ratio model (1.2) with r(x) = x fits the melanoma data well. Under model (1.2) with r(x) = x, the maximum semiparametric  likelihood estimate of AUC is, according to (2.2) and (2.4), equal to A UC = 0.88923 2 with its estimated asymptotic variance and standard error identical to  ˜ /n = 0.00159 and √ √ / ˜ n = 0.00159 = 0.03982. The proposed semiparametric Wald statistic for testing the null hypothesis H0 : AUC = 0.5 versus the alternative hypothesis H1 : AUC  = 0.5 is equal √  to W˜ n = n(A UC − 0.5)/˜ = (0.88923 − 0.5)/0.03982 = 9.77580 or W˜ n2 = 95.56631 with P -value < 0.00001. Consequently, there is strong evidence that the clinical score system is capable of discriminating between diseased and nondiseased subjects.

Acknowledgements The author wishes to thank two referees for providing several important references and a number of constructive and helpful comments and suggestions that have greatly improved my original submission. This research was supported in part by a grant from the National Security Agency.

Appendix A. A.1. Proofs Proof of Theorem 1. According to Example 6.1.2 of Lehmann (1999, p. 374), we have   √ d  n(A UC − AUC) −→ N 0, 2 (F, G) , where 20 (F, G) =

1+ 2 01 + (1 + )210 

(A.1)

with 201 = Cov [I (X1 < Y1 ) , I (X2 < Y1 )] = P (X1 < Y1 , X2 < Y1 ) − [P (X1 < Y1 )]2 , 210 = Cov [I (X1 < Y1 ) , I (X1 < Y2 )] = P (X1 < Y1 , X1 < Y2 ) − [P (X1 < Y1 )]2 . (A.2)  √   Furthermore, it follows from Theorem 2 of Qin and Zhang (2003) that n A UC − AUC    d −→ N 0, 2 0 , 0 , G , where    1+ ∞ ∞  2  0 , 0 , G = [F (s ∧ t) − F (s)F (t)] dG(s) dG(t)  −∞ −∞  ∞ ∞ + (1 + ) [G(s ∧ t) − G(s)G(t)] dF (s) dF (t) −∞ −∞  ∞ ∞ − J (s, t) dG(s) dG(t). (A.3) −∞ −∞

1872

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

The reason the  distribution function F does not occur as an argument in the notation of 2 0 , 0 , G is because under model (1.2), F itself depends on , , and G. Since  P (X1
∞ −∞

 P (X1
P (X1 < Y1 , X1 < Y2 ) = =

=

−∞

 G(y) dF (y) =

∞ −∞

[1−F (x)] dG(x),



−∞ ∞ −∞

 P (X1 < Y1 , X2 < Y1 ) =



P (X1 < x, X1 < y) dF (x) dF (y) G(x ∧ y) dF (x) dF (y),



−∞ ∞ −∞

P (x < Y1 , y < Y1 ) dG(x) dG(y) [1−F (y)−F (x)+F (x ∧ y)] dG(x) dG(y),

(A.4)

it follows from (A.2) and (A.4) that 201 = P (X1 < Y1 , X2 < Y1 ) − [P (X1 < Y1 )]2  ∞ = [1 − F (y) − F (x) + F (x ∧ y)] dG(x) dG(y) −∞   ∞   ∞ [1 − F (x)] dG(x) [1 − F (y)] dG(y) − −∞  ∞ −∞ = [F (x ∧ y) − F (x)F (y)] dG(x) dG(y), −∞

210 = P (X1 < Y1 , X1 < Y2 ) − [P (X1 < Y1 )]2  ∞   ∞   ∞ = G(x ∧ y) dF (x) dF (y) − G(x) dF (x) G(y) dF (y) −∞ −∞ −∞ ∞ = [G(x ∧ y) − G(x)G(y)] dF (x) dF (y). (A.5) −∞

Combining (A.1), (A.3), and (A.5) yields  ∞ ∞  1+ 2  2 0 , 0 , G = J (s, t) dG(s) dG(t) 01 + (1 + )210 −  −∞ −∞  ∞ ∞ = 20 (F, G) − J (s, t) dG(s) dG(t). −∞ −∞

(A.6)



 According to Theorem 2.1 of Zhang (2000), 2 J (s, t)=(1+) 1 +  exp 0 + 0 r(s)  

   0 (t) is the asymptotic 1 +  exp 0 + 0 r(t) A0 (s ∧ t) − A0 (s), A1 (s) A−1 A A1 (t) 

   √  ˜ ˆ covariance function of the process Zn (t) = n 1 +  exp 0 + 0 r(t) G(t) − G(t) . Consequently, it follows from the Continuous Mapping Theorem (Billingsley, 1968, p. 30)

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

1873

that as n → ∞, 

1 0

     Zn G−1 (s) ds → N 0, 2Z 0 , 0 , G

in distribution, where   0 2Z 0 , 0 , G =



=

1 1

0 0 ∞

2

  2 J G−1 (s), G−1 (t) ds dt  ∞ J (s, t) dG(s) dG(t).

−∞ −∞

  This, along with (A.6), implies that 2 0 , 0 , G 20 (F, G), thus establishing (2.3). The proof is complete.  Proof of Theorem 2. It can be shown as in the  proof of Theorem 1 of Qin and Zhang (2003) that under model (3.1) with (, , ) = 0 , 0 , n , one can write     ˜ R(s) − R(s) = Qn G−1 (1 − s) + op n−1/2 ,

s ∈ [1 − b, 1 − a],

(A.7)



 where Qn (t) = − U1 (t) − exp  +  r(t) U2 (t) − −1 U3 (t) with U1 (t) = Fˆ (t) − F (t), U2 (t) = H1 (t) − H2 (t) − G(t), ˆ − H2 (t), U3 (t) = H1 (t) − G(t) and n 1 I (Ti t)  , n0 1 +  exp 0 + 0 r (Ti ) i=1  ⎞ ⎛ j 0 , 0 ⎜ ⎟  1+  j  ⎟ . H2 (t) = A0 (t), A1 (t) A−1 ⎜ ⎝ j 0 , 0 ⎠ n j

H1 (t) =

Furthermore,   it can be shown after some algebra that under model (3.1) with (, , ) = 0 , 0 , n , we have         E [U1 (t)] = E Fˆ (t) − F (t) = I (y t) h y, n − h y, 0 dF (y)  t     1 1 = √  s y, 0 dF (y) + o n−1/2 = √  B(t)[1 + o(1)], n n −∞

1874

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

E

E

√   √  √  √  ˆ nU2 (t) = E nU3 (t) = E n H1 (t) − G(t) − E nH2 (t)    t    exp 0 + 0 r(x)    s x, 0 dG(x) =  −∞ 1 +  exp 0 + 0 r(x)    V0 + o(1) −  (A0 (t), A1 (t)) A−1  V1     V0 =  V0 (t) − (A0 (t), A1 (t)) A−1 + o(1) V1 = m(t) + o(1), √



 √  nQn (t) = − nE U1 (t) − exp 0 + 0 r(t) U2 (t) − −1 U3 (t) 

 = −  B(t) − exp 0 + 0 r(t) m(t) − −1 m(t) + o(1) 

 = exp 0 + 0 r(t) + −1 m(t) −  B(t) + o(1)    ' & 

 V0 = 1+ exp 0 +0 r(t) V0 (t)− (A0 (t), A1 (t)) A−1 V1  − B  (t) + o(1) = (t) + o(1).

(A.8)

Moreover, it can be shown asin the proof of Theorem 1 of Qin and Zhang (2003) that under model (3.1) with (, , ) = 0 , 0 , n as n → ∞, we have Cov

√  √ nQn (s), nQn (t) = K(s, t) + o(1),

(A.9)

where K(s, t) =

1+ [F (s ∧ t) − F (s)F (t)]      + (1 + ) exp 0 + 0 r(s) exp 0 + 0 r(t) [G(s ∧ t) − G(s)G(t)]  



1+ − 1 +  exp 0 + 0 r(s) 1 +  exp 0 + 0 r(t)       −1 A0 (t)  × A0 (s ∧ t) − A0 (s), A1 (s) A . A1 (t)

(A.8) and (A.9), along with the central limit theorem for sample √ means and the Cramer–Wold device, implies that the finite-dimensional distributions of nQn converge weakly to those  of W under model (3.1) with (, , ) = 0 , 0 , n , where W is a Gaussian process with mean function E[W (t)] = (t) and covariance function E[W (s)W (t)] = K(s, t). By employing the tightness criteria in Billingsley (1968, Chapter 3), we can show that the process

√   D √ nQ (t), −∞ t ∞ is tight in D[−∞, ∞]. As a result, we have nQn G−1 −→ W  −1  n G in D[1 − b, 1 − a] by (A.7). It now follows from the Continuous Mapping Theorem

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

1875

(Billingsley, 1968, p. 30) that   √   n A UC − AUC0 =

 1    d ˜ n R(s) − R(s) ds −→ W G−1 (1 − s) ds 0 0   ∼N  , 2 0 , 0 , G , 1√

      d that W˜ n −→ N  / 0 , 0 , G , 1 where 2 0 , 0 , G is given by (A.3).  This implies  under model (3.1) with (, , ) = 0 , 0 , n as n → ∞. Since it can be shown that       ˜ G ˜ is a consistent estimator of 2 0 , 0 , G under 0 , 0 , n as n → ∞, ˜ 2 = 2 ˜ , , it follows from Slutsky’s theorem and the well-known results on the distribution of quadratic  2 2     ˜ ˜ d 2 2 ˜2  forms of normal random variables that  Wn = n AUC − AUC0 / ˜ , , G −→ 1 under model (3.1) with (, , ) = 0 , 0 , n as n → ∞, thus establishing Theorem 2. The proof is completed. 

References Begg, C.B., 1991. Advances in statistical methodology for diagnostic medicine in the 1980’s. Statist. Med. 10, 1887–1895. Billingsley, P., 1968. Convergence of Probability Measures. Wiley, New York. Dorfman, D.D., Berbaum, K.S., Metz, C.E., 1992. Receiver operating characteristic rating analysis: generalization to the population of readers and cases with the jack-knife method. Invest. Radiol. 27, 723–731. Efron, B., Tibshirani, R., 1993. An Introduction to the Bootstrap. Chapman & Hall, New York. Goddard, M.J., Hinberg, I., 1990. Receiver operating characteristic (ROC) curves and non-normal data: an empirical study. Statist. Med. 9, 325–337. Hsieh, F., Turnbull, B.W., 1996. Nonparametric estimation of the receiver operating characteristic curve. Ann. Statist. 24, 25–40. Jiang, Y., Metz, C.E., Nishikawa, R.M., 1996. A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology 201, 745–750. Kay, R., Little, S., 1987. Transformations of the explanatory variables in the logistic regression model for binary data. Biometrika 74, 495–501. Lehmann, E.L., 1999. Elements of Large-Sample Theory. Springer, New York. Li, G., Tiwari, R.C., Wells, M.T., 1999. Semiparametric inference for a quantile comparison function with applications to receiver operating characteristic curves. Biometrika 86, 487–502. Lloyd, C.J., 1998. Using smoothed receiver operating characteristic curves to summarize and compare diagnostic systems. J. Am. Statist. Assoc. 93, 1356–1364. Lloyd, C.J., 2002a. Semi-parametric estimation of ROC curves based on binomial regression modelling. Aust. New Zeal. J. Statist. 44, 193–204. Lloyd, C.J., 2002b. Estimation of a convex ROC curve. Statist. Prob. Lett. 59, 99–111. McClish, D.K., 1990. Determining a range of false-positive rates for which ROC curves differ. Med. Decis. Making 10, 283–287. Metz, C.E., 1986. Statistical analysis of ROC data in evaluating diagnostic performance. In: Herbert, D., Myers, R. (Eds.), Multiple Regression Analysis: Applications in the Health Sciences. American Institute of Physics, New York, pp. 365–384. Metz, C.E., Kronman, H.B., 1980. Statistical significance tests for binormal ROC curves. J. Math. Psychol. 22, 218–243. Metz, C.E., Wang, P., Kronman, H.B., 1984. A new approach for testing the significance of differences between ROC curves measured from correlated data. In: Deconinck, F. (Ed.), Information Processing in Medical Imaging. Nijhoff, The Hague, The Netherlands, pp. 432–445.

1876

B. Zhang / Computational Statistics & Data Analysis 50 (2006) 1855 – 1876

Metz, C.E., Herman, B.A., Roe, C.A., 1998. Statistical comparison of two ROC curve estimates obtained from partially-paired datasets. Med. Decis. Making 18, 110–121. Metz, C.E., Herman, B.A., Shen, J.H., 1998. Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously distributed data. Statist. Med. 17, 1033–1053. Pepe, M.S., 2000a. Receiver operating characteristic methodology. J. Am. Statist. Assoc. 95, 308–311. Pepe, M.S., 2000b. An interpretation for ROC curve and inference using GLM procedures. Biometrics 56, 352–359. Pepe, M.S., 2003. The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford University Press, New York. Qin, J., Zhang, B., 1997.A goodness-of-fit test for logistic regression models based on case-control data. Biometrika 84, 609–618. Qin, J., Zhang, B., 2003. Using logistic regression procedures for estimating receiver operating characteristic curves. Biometrika 90, 585–596. Toledano, A.Y., Gatsonis, C., 1996. Ordinal regression methodology for ROC curves derived from correlated data. Statist. Med. 15, 1807–1826. Venkatraman, E.S., Begg, C.B., 1996. A distribution free procedure for comparing receiver operating characteristic curves from a paired experiment. Biometrika 83, 835–848. Wieand, S., Gail, M.H., James, B.R., James, K.L., 1989. A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika 76, 585–592. Zhang, B., 2000. Quantile estimation under a two-sample semi-parametric model. Bernoulli 6, 491–511. Zhou, X.H., Gatsonis, C.A., 1996. A simple method for comparing correlated ROC curves using incomplete data. Statist. Med. 15, 1687–1693. Zhou, X.H., McClish, D.K., Obuchowski, N.A., 2002. Statistical Methods in Diagnostic Medicine. Wiley, New York. Zou, K.H., Hall, W.J., 2000. Two transformation models for estimating an ROC curve derived from continuous data. J. Appl. Statist. 27, 621–631.