On determining the sensitivity and specificity of a new diagnostic test through comparing its results against a non-gold-standard test

Introduction

Diagnostic tests are among the important means commonly used in clinical medicine. Before a new test can be used in clinical practice, it should be evaluated for clinical validity. Studies assessing the clinical validity of a test (also termed diagnostic accuracy studies) involve determining the test performance indices including the test sensitivity (Se) and specificity (Sp) (1). Other common performance indices are positive and negative predictive values, and likelihood ratios, which can be calculated based on the Se and Sp and the prevalence (pr) of the disease of interest (2, 3). To determine a test performance, its results should be evaluated against another test, the so-called reference standard (4). The reference standard can be a gold-standard test, i.e., a test with a Se and Sp of 1.0 (or 100%). The gold-standard test can thus correctly discriminate those with and without the disease or condition of interest. For a test with binary results, the outcome is clear – positive or negative. For tests with continuous results, however, we need to set a cut-off value to categorize the results into positive or negative (2). Compared to the gold standard, the obtained results can be categorized into true-positive (TP), true-negative (TN), false-positive (FP), and false-negative (FN) results (Table 1a). The tests Se and Sp are defined as follows (5):

Both the Se and Sp follow the binomial distribution. Then, the squared standard errors (SE²) for Se and Sp are:

The prevalence of the disease (π), is then:

Combining Eq. 1 and Eq. 3, we have:

where P(x) designates the probability of x. To evaluate the Se and Sp of a new test, it is common to compare its test results against those obtained from a gold-standard test. Nonetheless, the gold-standard test may not always be available. It either does not exist or is very difficult or expensive to perform for certain disease conditions (6). The question arise is that whether it is possible to calculate the Se and Sp of the new test based on the results obtained from its comparison with a non-perfect reference standard – a well-established (but not a gold-standard) test? This is not a new question, and several solutions has so far been proposed (1). Herein, I wish to propose an analytical method to address the question raised.

Stating the question

Suppose that we have a well-established test, say T₁, with known Se and Sp (measured against a gold-standard test) of Se₁ and Sp₁ (Table 1a). Now, suppose that we have a new test, say T₂, the results of which were compared against T₁ (not against a gold standard), and that it had a Se and Sp (against T₁) of Se_2,1 and Sp_2,1 (Table 1b). We wish to derive the Se and Sp of T₂ (Se₂ and Sp₂), if it would have been tested against the gold standard (e.g., Table 1c).

The proposed solution

When we compare T₂ against T₁, the calculated prevalence, pr, is not really the true prevalence, π, as T₁ is not a gold standard and thus would have FP and FN results. However, we can calculate the true prevalence, π, as follows (7):

Based on Eq. 4 and basic probability rules, we have (Table 1) (8, 9):

and

where T ⁺ and T ^– represent positive and negative test results; and D ⁺ and D ^–, presence and absence of the disease, respectively. P(A|B) denotes the conditional probability of event A given event B.

Based on Eq. 6, we have:

Solving for Se₂, gives:

Based on Eq. 7, we have:

Then:

Equations 9 and 11 are a system of two simultaneous equations. Substituting π from Eq. 5 and solving for Se₂ and Sp₂, yield:

If f is a function of k independent random variables, then the squared SE of f can be calculated as (10, 11):

Assuming that Se₂ is a function of independent random variables pr, Se_2,1, Sp_2,1, and Sp₁ (Eq. 12), using Eq. 13 and employing basic calculus, we have:

In the same way, assuming that Sp₂ is a function of independent random variables pr, Se_2,1, Sp_2,1, and Se₁ (Eq. 12), we have:

The SE for the Se and Sp of the tests can be calculated using Eq. 2.

Discussion

It was shown that the test Se and Sp can be determined with acceptable accuracy even if the gold-standard test is not available. The Se and Sp of the new test (T₂) derived by transforming the values obtained from its comparison with a non-gold-standard test (Se_2,1 and Sp_2,1) are acceptably close to the values if the test would have been compared with the gold-standard (Se₂ and Sp₂). The variances of the calculated Se₂ and Sp₂ (Eqs. 14 and 15) are higher than those you might obtain if you would have compared T₂ directly against the gold standard, instead of T₁. This is attributed to the uncertainty exist in the variables used for the calculation (Eq. 12). To examine the application of the technique proposed let us apply it to an example.

Example

Suppose that in a validity study of 500 (arbitrary chosen) randomly selected people, a diagnostic test (let us call it T₁) was tested against the gold standard (Table 1a), and that the test could correctly identify 85 of 100 diseased people, hence a Se (Se₁) of 0.85, and 360 of 400 disease-free individuals, hence a Sp (Sp₁) of 0.90 (Table 1a). The calculated SE² for the Se₁ and Sp₁ are 1.3 × 10^-3 and 2.3 × 10^-4, respectively (using Eq. 2). Also, suppose that in a validity study on 600 (arbitrary chosen) randomly selected people, the results of a new diagnostic test, T₂, was compared against T₁ (Table 1b). Based on the information provided, the apparent prevalence, pr, is 0.25 (SE² = 3.1 × 10^-4). Using Eq. 5, the true prevalence (π) is:

which is correct when the disease prevalence is measured by a gold-standard test (Table 1a). The Se and Sp (along with their SE²) of T₂ against T₁ (Table 1b), are then:

Plugging in the values in equations 12, 14 and 15, estimations of Se₂ and Sp₂ are 0.95 (SE² = 8.0 × 10^-3; 95% confidence interval (CI): 0.77 to 1.00) and 0.80 (SE² = 5.4 × 10^-4; 95% CI: 0.75 to 0.85), respectively, which are compatible with the results if T₂ would have been compared against the gold-standard test – 0.95 (SE² = 5.9 × 10^-4; 95% CI: 0.90 to 1.00) and 0.80 (SE² = 5.0 × 10^-4; 95% CI: 0.76 to 0.84), respectively (Table 1c). Note that the 95% CI of the calculated Se₂ and Sp₂ when they are derived through comparing the results with T₁ is wider than those if they are directly compared against a gold-standard test.

In conclusion, it seems that this technique is useful, particularly where the gold-standard test is not readily available or is expensive. Further studies are needed to elaborate on the conditions of the validity study where the Se₁ and Sp₁ are estimated, the minimum number of data points examined, the probable effect of the prevalence of the disease or condition of interest on the choice of the reference test, among other things.