Search PubMed⌕ Search

SEARCH · Search PubMed

Results for “ROC Curve”

Search indexed PubMed citations on genomics, clinical trials, systematic reviews and public health. Explore titles, authors and supplied subject terms, then open the PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 recordsLinked to original sources
PubMed1991

Distribution-free confidence bounds for ROC curves.

ROC curves are widely used for the evaluation of diagnostic tests to decide between "healthy" and "diseased" individuals when the measurements are on a continuous scale. These curves are graphical displays of the interdependence between specificity and sensitivity of the test varying with the cut-off point chosen for the decision. Up to now only point estimators derived from the empirical distribution functions are used which may be misleading if they are based on rather small samples. In this paper we propose reasonable confidence bounds for ROC curves and a corresponding point estimator. Our bounds are strongly related to two-sided distribution-free tolerance regions because they are constructed from minimum and maximum coverages which at a given value chi can be guaranteed with a confidence (2 pi*-1). The interpretation of the bounds is that if a cut-off-point is chosen on the basis of the ROC curve then with a nominal confidence of at least (2 pi*-1)2 the real sensitivity and specificity will be within a rectangle.

Algorithms↗
PubMed1992

Evaluation of diagnostic tests using relative operating characteristic (ROC) curves and the differential positive rate. An example using the total serum bile acid concentration and the alanine aminotransferase activity in the diagnosis of canine hepatobiliary diseases.

The value of a diagnostic test depends on most cases on its ability to discriminate between patients with and without a certain disease. One way of evaluating a diagnostic test is to use the relative operating characteristic curve (ROC curve) and the differential positive rate (DPR). The ROC curve displays the relationship between the true positive ratio and the false positive ratio for a range of cutoff values and it can be used to compare various diagnostic tests under equivalent conditions (equal true positive ratios or false positive ratios) and over the entire range of cutoff values. The DPR is the difference between the true positive ratio and the false positive ratio at various cutoff values and it can be used to obtain the cutoff value associated with the highest sensitivity and specificity. The purpose of this study was to describe the evaluation and comparison of diagnostic tests using ROC curves and DPR. Eventually, the positive and negative predictive values were used to assess the differences between the sensitivity and specificity obtained when the upper limit of the reference interval, or the optimal cutoff value indicated by the DPR, was used as cutoff value. To illustrate the methods, the 2 h post-prandial total serum bile acid concentration (PSBA) and the alanine aminotransferase activity (ALAT) in the diagnosis of primary or secondary hepatobiliary diseases in dogs were used. The ROC curves showed, as expected from previous studies, that PSBA was superior to ALAT in diagnosing dogs with hepatobiliary diseases. Using DPR, the optimal cutoff value for PSBA was suggested to be 15.48 mumol/l. Compared to the traditionally used cutoff value of 22.24 mumol/l, no decisive difference in the positive predictive values were observed. However, the cutoff value of 15.48 mumol/l appeared to produce higher negative predictive values compared to a cutoff value of 22.24 mumol/l. Seemingly, ROC curves and DPR are simple methods useful to the evaluation of diagnostic tests and due to the simplicity, there seems to be a great potential for these methods in the evaluation of diagnostic tests in veterinary medicine.

Alanine Transaminase↗
PubMed2004

Application of receiver operating characteristics curve (ROC) analysis when definitive and suspect morphologic flags appear in the new Coulter LH 750 analyzer.

Workflow efficiency is important in every laboratory. Manual assessment of white blood cell counts and differentials that have been rejected by an automated analyzer is one of the most time-consuming tasks in the routine hematology laboratory. In this study, receiver operating characteristics (ROC) curve analysis was used for the first time when anomalous distribution and suspect flag alarms appeared in hemograms carried out with the new Beckman Coulter LH 750 analyzer. This article is our second about the LH 750 analyzer published in this journal; we increased the number of cases and describe the novel application of statistical analysis of ROC curves. In processing of specimens from patients with 3% to 6% immature granulocytes (myelocytes + metamyelocytes + bands ), the suspect flag Imm Ne 1 (immature granulocytes) showed 77% diagnostic efficiency with a maximum area under curve (AUC) of 0.71 and a 95% confidence interval (CI) of 0.597 to 0.831 without significant differences between the 3 available levels of alarms in the analyzer (L1, L2, L2). In processing of specimens from patients with >6% immature granulocytes, the Imm Ne 1 flag showed superior diagnostic efficiency of 98% with a maximum AUC of 0.80 and a CI of 0.713 to 0.879. The suspect flag Imm Ne 2 in processing of specimens from patients with >6% of immature granulocytes showed diagnostic efficiency of 92% with a maximum AUC of 0.77 and a CI of 0.665 to 0.871, finding a significant positive difference in level L3 regarding sensitivity in comparison with the other 2 levels of the analyzer (L1, L2). For specimens from patients with >2% blasts, the suspect Blasts alarm showed a diagnostic efficiency of 94%, an AUC of 0.91, and a CI of 0.775 to 1.043; positive differences were observed between the levels L2/L3 and L1. In processing of specimens with variant lymphocytes (large, granular, prolymphocytes, cleaved, chronic lymphocytic leukemia type, and so forth) >10% (x = 14%), the suspect alarm Var Lym (variant lymphocytes) showed a low diagnostic sensitivity of 20% with a maximum AUC of 0.59 and a CI of 0.300 to 0.870 without significant differences between the 3 available levels (L1, L2, L3). However, in processing of specimens presenting values >10% reactive or activated lymphocytes (x = 23%), typical for patients with infectious mononucleosis, the Var Lym flag showed a superior sensitivity of 75% with a diagnostic efficiency of 92% and an AUC of 0.84 with a CI of 0.587 to 1.089. Finally, the laboratory can easily program definitive abnormal morphological flags of distribution (granulocytosis, eosinophilia, monocytosis, and so forth) on the basis of its patient population. In this study we were able to carry out comparisons of AUC and to choose the values for the automated counts in percentage, absolute value, or both. Therefore we were able to define the reliability and impact on the alarm routine to optimize the performance of the user-adjustable definitive alarms for anomalous distribution.

Adult↗
PubMed1992

Variation in restorative treatment decisions: application of Receiver Operating Characteristic curve (ROC) analysis.

It has been evident for many years that dentists, when planning treatment for patients, do not act in a standard manner, and previous research has shown there to be wide variations in treatment planning amongst groups of dentists. Signal detection theory and Receiver Operating Characteristic (ROC) analysis allows measurement of an observer's ability to detect a lesion, while at the same time allowing examination of how a lesion, once perceived, is judged to be in need of treatment. An ROC curve is constructed by plotting the sensitivity (or true positive rate) of decisions made, against the false positive rate (equivalent to 1-specificity) when various decision attitudes, from interventionist to non-interventionist, are held. Fifteen pairs of simulated bitewing radiographs were shown to 20 dentists, who were asked to specify, for each approximal lesion, whether or not they would place a conventional restoration. The 7200 decisions made by the dentists were validated by sectioning and microscopically examining the teeth. The mean sensitivity of the dentists' decisions, when the strictest operating thresholds were held and caries into dentine was the validating criterion, was 0.26 and the mean specificity was 0.96. ROC analysis shows that when operating at the strictest threshold, the dentists were implying that specificity was weighted as being 2.7 times more important than sensitivity. ROC analysis leads to insight into how dentists differentially weight the true and false, positive and negative, outcomes of their decisions and thus allows explanation of why two dentists would rarely make exactly the same treatment plan for one patient, and also why different treatments might be offered to two patients exhibiting the same levels of disease.

Decision Support Techniques↗
PubMed2006

Empirical likelihood inference for the area under the ROC curve.

For a continuous-scale diagnostic test, the most commonly used summary index of the receiver operating characteristic curve (ROC) is the area under the curve (AUC) that measures the accuracy of the diagnostic test. In this article, we propose an empirical likelihood (EL) approach for the inference on the AUC. First we define an EL ratio for the AUC and show that its limiting distribution is a scaled chi-square distribution. We then obtain an EL-based confidence interval for the AUC using the scaled chi-square distribution. This EL inference for the AUC can be extended to stratified samples, and the resulting limiting distribution is a weighted sum of independent chi-square distributions. Additionally we conduct simulation studies to compare the relative performance of the proposed EL-based interval with the existing normal approximation-based intervals and bootstrap intervals for the AUC.

Biometry↗
PubMedDate not supplied

Integrating stratum-specific likelihood ratios with the analysis of ROC curves.

Data used to construct receiver operating characteristic (ROC) curves and to calculate the area under the curve (ROC AUC) can be used to derive stratum-specific likelihood ratios (SSLRs) with their 95% confidence intervals (95% CIs). The purpose of this study was to determine whether useful information can be obtained by adding SSLRs to the analysis of ROC curves. The authors analyzed four previously reported sets of data: 1) serum creatine kinase (SCK) for diagnosing acute myocardial infarction (AMI) in the coronary care unit (CCU); 2) SCK in the evaluation of chest pain in the emergency center (EC); 3) four predictor variables in the diagnosis of strep throat; and 4) the ordinal assessment of computed tomographic (CT) images. Use of SCK in the CCU produced four strata that had posttest probabilities that were highly discriminating, whereas SCK in the EC resulted in only two strata with limited discriminating ability. In either study the cutpoint at which the SSLR changed from less than to greater than 1.0 was higher than the reported upper normal for the test, thereby quantitating spectrum bias. The maximum number of strata of predictor signs and symptoms for strep throat was three rather than the five used in previous studies. With a larger sample size or pooling, four strata could probably be developed. With CT images, "definitely normal," "probably normal," and "questionable" were collapsed to one negative stratum. "Probably abnormal" became the true "questionable" stratum and "definitely abnormal" was the only positive stratum. The authors conclude that additional useful information is obtained by deriving stratum-specific likelihood ratios as part of the analysis of an ROC curve.

Humans↗
PubMedDate not supplied

Analyzing a portion of the ROC curve.

The area under the ROC curve is a common index summarizing the information contained in the curve. When comparing two ROC curves, though, problems arise when interest does not lie in the entire range of false-positive rates (and hence the entire area). Numerical integration is suggested for evaluating the area under a portion of the ROC curve. Variance estimates are derived. The method is applicable for either continuous or rating scale binormal data, from independent or dependent samples. An example is presented which looks at rating scale data of computed tomographic scans of the head with and without concomitant use of clinical history. The areas under the two ROC curves over an a priori range of false-positive rates are examined, as well as the areas under the two curves at a specific point.

Computer Simulation↗
PubMed1997

Sample size determination for diagnostic accuracy studies involving binormal ROC curve indices.

Receiver operating characteristic (ROC) curves and their associated indices are valuable tools for the assessment of the accuracy of diagnostic tests. The area under the ROC curve is a popular summary measure of the accuracy of a test. The full area under the ROC curve, however, has been criticized because it gives equal weight to all false positive error rates. Alternative indices include the area under the ROC curve in a particular range of false positive rates ('partial' area) and the sensitivity of the test for a single fixed false positive rate (FPR). We present a unified approach for computing sample size for binormal ROC curves and their indices. Our method uses Taylor series expansions to derive approximate large-sample estimates of the variance and covariance of binormal ROC curve parameters. Several examples from diagnostic radiology illustrate the proposed method.

Diagnostic Tests, Routine↗
PubMed2000

ROC surface: a generalization of ROC curve analysis.

Receiver operating characteristic (ROC) curve analysis is widely used in biomedical research to assess the performance of diagnostic tests. Much of the work has been directed at developing accurate indices to describe ROC curves and appropriate statistics to test differences between them. The analysis, however, is largely built on the assumption that the test results are dichotomous. We generalize the ROC curve analysis to allow for tests to have more than two outcomes. The generalized ROC curve constitutes a surface. We propose to use the volume under the surface to measure the accuracy of a diagnostic test.

Algorithms↗
PubMed2006

A permutation test for comparing ROC curves in multireader studies a multi-reader ROC, permutation test.

RATIONALE AND OBJECTIVES: The aim of the study is to develop a permutation test to compare receiver operating characteristic (ROC) curves of two diagnostic modalities in a multireader paired design. MATERIALS AND METHODS: A statistical test for comparing two diagnostic modalities is developed based on all possible exchanges of the set of reader-ratings between the two modalities. An exact permutation test is formed by determining the frequency of the most extreme values of the statistic estimating the average difference in the areas under the ROC curves (AUCs). An asymptotic version of the test is constructed by obtaining the exact permutation variance and appealing to the asymptotic normality of the nonparametric estimator of the average difference in areas. Computer simulations were conducted to validate the type I error for small sample sizes. RESULTS: The new test provides a permutation approach for comparing ROC curves in a multireader paired-design setting in which effects of the readers are considered to be fixed. The type I error of the asymptotic test is close to the true value, even for samples as small as 20 normal and 20 abnormal cases. The test is designed to be sensitive to alternatives in which the AUCs of the two diagnostic modalities differ. CONCLUSIONS: The proposed test provides a powerful method for comparing two diagnostic modalities in a multireader paired-study design when the primary interest is to detect difference in average AUCs.

Analysis of Variance↗
PubMed1997

Smooth non-parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests.

Receiver operating characteristic (ROC) curves have seen increasing use to assess the accuracy of diagnostic tests that yield continuous test results. We can calculate a fully non-parametric ROC curve from the empirical false positive rate (FPR) and true positive rate (TPR) at each possible decision threshold, but it may be quite jagged. We can obtain a smooth ROC curve by directly fitting a parameteric model, for example, binormal or bilogistic, to the actual test results, but substantial lack-of-fit may result if the distributional assumptions are not valid. A recently proposed alternative algorithm 'LABROC4' is insensitive to such departures, but is not fully flexible in that the form of the resulting ROC curve is restricted to be binormal. We propose a smooth non-parametric ROC curve derived from kernel density estimates of the two test result distributions. We obtain pointwise standard errors for the estimated TPR at each FPR and use them to construct pointwise confidence intervals for the ROC curve, using a logit transformation. We also obtain standard errors for the estimated TPR and FPR at given thresholds and use them to construct confidence rectangles in (FPR, TPR)-space that correspond to a specific threshold. We adapt existing methods for the unsmoothed non-parametric ROC curve to obtain the area under the ROC curve and its standard error; we also give partial areas and corresponding standard errors. We have created a FORTRAN algorithm 'ROC-&-ROL', available upon request. We compare ROC curves and areas obtained by applying our methods and its competitors to two data sets, one of which is fit well by parametric methods and LABROC4, the other of which is not.

Algorithms↗
PubMed2004

Adjusting the generalized ROC curve for covariates.

Receiver operating characteristic (ROC) curves and in particular the area under the curve (AUC), are widely used to examine the effectiveness of diagnostic markers. Diagnostic markers and their corresponding ROC curves can be strongly influenced by covariate variables. When several diagnostic markers are available, they can be combined by a best linear combination such that the area under the ROC curve of the combination is maximized among all possible linear combinations. In this paper we discuss covariate effects on this linear combination assuming that the multiple markers, possibly transformed, follow a multivariate normal distribution. The ROC curve of this linear combination when markers are adjusted for covariates is estimated and approximate confidence intervals for the corresponding AUC are derived. An example of two biomarkers of coronary heart disease for which covariate information on age and gender is available is used to illustrate this methodology.

Adult↗
PubMed1985

An evaluation of methods for estimating the area under the receiver operating characteristic (ROC) curve.

The area under the receiver operating characteristic (ROC) curve serves as one means for evaluating the performance of diagnostic and predictive test systems. The most commonly used method for estimating the area under an ROC curve utilizes the maximum-likelihood-estimation technique, and a nonparametric method to calculate the area under an ROC curve was recently described. We compared the performance of these two methods. The results for the area under the ROC curve and the standard error of the estimate as calculated by each of the two methods exhibited high correlation. Generally, the nonparametric method yields lower area estimates than the maximum-likelihood-estimation technique. However, these differences generally were small, particularly with ROC curves derived from five or more cutoff points. Consistent results of hypothesis testing of the significance of differences between two ROC curves will be similar, regardless of which method is used, as long as one uses the same estimation technique on the two curves and as long as the two ROC curves being compared are of similar shape.

Computers↗
PubMed1997

Generating ROC curves for artificial neural networks.

Receiver operating characteristic (ROC) analysis is an established method of measuring diagnostic performance in medical imaging studies. Traditionally, artificial neural networks (ANN's) have been applied as a classifier to find one "best" detection rate. Recently researchers have begun to report ROC curve results for ANN classifiers. The current standard method of generating ROC curves for an ANN is to vary the output node threshold for classification. In this work, we propose a different technique for generating ROC curves for a two-class ANN classifier. We show that this new technique generates better ROC curves in the sense of having greater area under the ROC curve (AUC), and in the sense of being composed of a better distribution of operating points.

Diagnostic Imaging↗
PubMed2001

A constrained formulation for the receiver operating characteristic (ROC) curve based on probability summation.

We propose a principled formulation of the ROC curve that is constrained in a realistic way by the mechanism of probability summation. The constrained and conventional ROC formulations were fitted to 150 separate sets of rating data taken from previous observer studies of 250 or 529 chest radiographs. A total of 20 different readers had used either discrete or continuous rating scales to evaluate those chest cases for likelihood of separate specified abnormalities: interstitial disease, pulmonary nodule, pneumothorax, alveolar infiltrate, or rib fracture. Both ROC formulations were fitted separately to every set of rating data using maximum-likelihood statistical procedures that specified each ROC curve by normally distributed latent variables with two scaling parameters, and estimated the area below the ROC curve (Az) with its standard error. The conventional and constrained binormal formulations usually fitted ROC curves that were nearly indistinguishable in form and in Az. But when fitted to asymmetric rating data that contained few false-positive cases, the conventional ROC curves often rose steeply, then flattened and extrapolated into an unrealistic upward "hook" at the higher false-positive rates. For those sets of rating data, the constrained ROC curves (without hooks) estimated larger values for Az with smaller standard errors. The constrained ROC formulation describes observers' ratings of cases at least as well as the conventional ROC, and always guarantees a realistic fitted curve for observer performance. Its estimated parameters are easy to interpret, and may also be used to predict observer accuracy in localizing the image abnormalities.

Databases as Topic↗
PubMedDate not supplied

Receiver operating characteristic (ROC) curve: practical review for radiologists.

The receiver operating characteristic (ROC) curve, which is defined as a plot of test sensitivity as they coordinate versus its 1-specificity or false positive rate (FPR) as the x coordinate, is an effective method of evaluating the performance of diagnostic tests. The purpose of this article is to provide a nonmathematical introduction to ROC analysis. Important concepts involved in the correct use and interpretation of this analysis, such as smooth and empirical ROC curves, parametric and nonparametric methods, the area under the ROC curve and its 95% confidence interval, the sensitivity at a particular FPR, and the use of a partial area under the ROC curve are discussed. Various considerations concerning the collection of data in radiological ROC studies are briefly discussed. An introduction to the software frequently used for performing ROC analyses is also presented.

Area Under Curve↗
PubMed2000

Time-dependent ROC curves for censored survival data and a diagnostic marker.

ROC curves are a popular method for displaying sensitivity and specificity of a continuous diagnostic marker, X, for a binary disease variable, D. However, many disease outcomes are time dependent, D(t), and ROC curves that vary as a function of time may be more appropriate. A common example of a time-dependent variable is vital status, where D(t) = 1 if a patient has died prior to time t and zero otherwise. We propose summarizing the discrimination potential of a marker X, measured at baseline (t = 0), by calculating ROC curves for cumulative disease or death incidence by time t, which we denote as ROC(t). A typical complexity with survival data is that observations may be censored. Two ROC curve estimators are proposed that can accommodate censored data. A simple estimator is based on using the Kaplan-Meier estimator for each possible subset X > c. However, this estimator does not guarantee the necessary condition that sensitivity and specificity are monotone in X. An alternative estimator that does guarantee monotonicity is based on a nearest neighbor estimator for the bivariate distribution function of (X, T), where T represents survival time (Akritas, M. J., 1994, Annals of Statistics 22, 1299-1327). We present an example where ROC(t) is used to compare a standard and a modified flow cytometry measurement for predicting survival after detection of breast cancer and an example where the ROC(t) curve displays the impact of modifying eligibility criteria for sample size and power in HIV prevention trials.

Breast Neoplasms↗
PubMedDate not supplied

The area under an ROC curve with limited information.

The area under the receiver operating characteristic (ROC) curve of a diagnostic test can be used as a summary measure for its discriminative ability. If only a single point of an ROC curve is available, then the entire form of the ROC curve is unknown and the area under it cannot be calculated. Assuming that the unknown ROC curve is either monotone or concave, lower and upper bounds are derived for the area. From these bounds, the minmax approximations are obtained. Compared to only assuming monotonicity, assuming that the unknown ROC curve is concave renders a higher minmax approximation for the area under it, with tighter bounds.

Area Under Curve↗