Xavier Robin – Tag – pROC

pROC 1.5.3 released

2012-08-31T11:48:50+02:00

I just released a minor revision of pROC, version 1.5.3.

This version fixes the following bugs:

AUC specification was lost when roc.test, cov or var was passed an auc' object.
Incorrect computation of "accuracy" in coords.

As usual, you can find the new version on ExPASy and on the CRAN (please allow up to a few days before it is available for Windows). To update, type update.packages() or install.packages("pROC") if you want to update pROC only.

pROC 1.5.1 released

2012-03-09T11:03:57+01:00

I sent a new minor version of pROC (1.5.1) to the CRAN.

This version fixes a bug that would make slow down the loading of pROC on computers with many packages. Thanks to Prof Brian Ripley and Glenn Lawyer for the report.

The new version is available on ExPASy and on the CRAN. To update, type update.packages() or install.packages("pROC") if you want to update pROC only.

pROC 1.5 released

2011-12-12T09:27:08+01:00

pROC's steady progression goes on with version 1.5. It is avalable for R only. S+ users will need to wait for the upcoming 1.6 release which will introduce power / sample size computations.

This version introduces four new notable features:

Variance and covariance
Univariate Log-Concave Density Estimation smoothing
Improvements to the plotting function
New return values in coords

Variance and covariance

It is now possible to compute the variance of a ROC curve, and the covariance of two paired ROC curves.

library(pROC)
data(aSAH)
rocobj <- roc(aSAH$outcome, aSAH$s100b)

var(roc1)
var(roc2)
cov(roc1, roc2)

Two methods are available: bootstrap, DeLong¹. The bootstrap is the most versatile method. DeLong is faster but works for full AUC only. For more details, see ?var.roc and ?cov.roc.

Univariate Log-Concave Density Estimation smoothing

Until now, three methods were available to smooth a ROC curve: binormal, density and fitdistr (to fit a distribution with MASS). Now, two new methods are available: logcondens and logcondens.smooth. They are based on Duembgen and Rufibach (2011)². You first need to install the logcondens package:

install.packages("logcondens")

It doesn't need to be loaded.

plot(rocobj)
rs <- smooth(rocobj, method="binormal")
plot(rs, add=TRUE, col="green")
rs2 <- smooth(rocobj, method="density")
plot(rs2, add=TRUE, col="blue")
rs3 <- smooth(rocobj, method="fitdistr", density="lognormal")
plot(rs3, add=TRUE, col="magenta")
rs4 <- smooth(rocobj, method="logcondens")
plot(rs4, add=TRUE, col="brown")
rs5 <- smooth(rocobj, method="logcondens.smooth")
plot(rs5, add=TRUE, col="orange")
legend("bottomright", legend=c("Empirical", "Binormal", "Density", "Log-normal",
                               "Log-concave density", "Smoothed log-concave density"),
       col=c("black", "green", "blue", "magenta", "brown", "orange"), lwd=2)

Improvements to the plotting function

Several users have been bothered by the fact than in pROC (R version), the sensitivity is plotted as decreasing specificity. Most other software plot increasing 1 – specificity on the X axis. The reason is purely historical: only few statistical software can plot an axis in decreasing direction. For instance S+ cannot do it, and pROC's ROC curve are plotted as 1 – specificity there. However it makes absolutely no difference on the ROC curve itself. As it was possible, I decided plot the modern version on R rather than stick to obsolete conventions.

For those who are disturbed and prefer to stick to obsolete conventions, pROC 1.5 comes with a way to plot increasing 1 – specificity in the R version with the legacy.axes argument.

plot(rocobj, legacy.axes=TRUE)

Note that it makes no difference to the coordinates of the plot, and if you want to add some text you still have to think in the "new" way. Consequently, the following will always be plotted to the top left corner of the curve, whatever legacy.axes you specified:

text(1, 1, auc(rocobj), adj=c(0, 1))

New return values in coords

The retargument of the coords function now accepts several new values:

"accuracy": (sensivity + specificity) / 2
"tn": true positives count
"tp": true negatives count
"fn": false negtives count (positive observations classified as negative by predictor)
"fp": false positives count (negative observations classified as positive by predictor)
"npv": negative predictive value, or tn / (tn + fn)
"ppv": positive predictive value, or tp / (tp + fp)

In addition, sensitivity, specificity, npv and ppv can be prefixed with 1- in order to get the opposite value. Finally two additional values are recognized:

"npe": converted to 1-npv
"ppe": converted to 1-ppv

Here is an example. We take the best threshold of the ROC curve rocobj and display all the parameters of this threshold:

coords(rocobj, "best", ret=c("threshold", "specificity", "sensitivity", "accuracy", "tn", "tp", "fn", "fp", "npv", "ppv", "1-specificity", "1-sensitivity", "1-npv", "1-ppv"))

Coords also accepts a new argument: drop to control the dimension of the return value. If drop is FALSE, a matrix will always be returned, even if it contains only one column. This is especially useful to make scripts more reliable.

Conclusion

Here is the full change log:

New cov and var functions
coords accepts new ret values: "accuracy", "tn", "tp", "fn", "fp", "npv", "ppv", "1-specificity", "1-sensitivity", "1-npv", "1-ppv", "npe" and "ppe"
New legacy.axes argument to plot 1-specificity rather than specificity
New axes argument to turn off the plotting of the axis
New logcondens and logcondens.smooth (Univariate Log-Concave Density Estimation) smoothing methods
New function has.partial.auc to determine if an AUC is full or partial
New argument drop for coords
auc and multiclass.auc objects now also have secondary class numeric
Updated load call
Delong's CI reversed in ROC curves with direction=">"
Delong's CI AUC returned values > 1 or < 0 in some rare cases
Minor improvements in documentation

As usual, you can find the new version on ExPASy and on the CRAN. To update, type update.packages() or install.packages("pROC") if you want to update pROC only.

1. Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson (1988) “Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach”. Biometrics 44, 837–845.
2. Lutz Duembgen, Kaspar Rufibach (2011) “logcondens: Computations Related to Univariate Log-Concave Density Estimation”. Journal of Statistical Software, 39, 1–28. URL: jstatsoft.org/v39/i06.

Thoughts about the S+ GUI

2011-11-19T19:37:29+01:00

For more than a year and a half, I have been releasing pROC version in two flavors: R and S+. The S+ version has a great advantage (or disadvantage for some) over R: it has a GUI, a graphical user interface. This is especially useful for users who do not want to learn programming R and who prefer a clic-and-point interface.

However, this dual-version comes with a cost. Here are some thoughts about it.

I have to maintain two different version. Fortunately R and S+ are similar enough that most code works in both environments. There are a few inconsistencies, especially the way the variables are scoped (see this document by John Fox for more details). I need to be careful when sapplying: any variable accessed inside the function must be passed as argument in S+. Functions such as try and eval also behave quite differently. Overall, good tests in the examples section of the documentation are crucial to detect potential errors.
The only exception is the plot function. Graphics have huge differences between R and S+. Any modification of plot is really difficult.
S+ is much slower than R. Bootstrap operations take much longer to complete. Even DeLong computations, which are nearly instant in R, take a noticeable time to complete in S+.
Specifically about the GUI now, writing S+ GUI code is awfully painful. For example there is little control over the placement of the buttons. The sample size window I'm currently developing is really ugly, and there is nothing I can do to improve it. The events are quite unpredictable, especially when the user rolls-back a window it is often in a broken status.
The previous point stems from the lack documentation of the S+ GUI. It comes with a minimal introduction, that is just enough to start. But there are far too many undocumented features I had to discover myself by guessing-and-error. For instance, how to catch a user roll-back? I still haven't found how to cancel an analysis when I detect an error in the user's submission, or how to handle colors properly. For a professional-grade software, this lack of documentation is unexpected, to say the least.

So, would I start a new project with a GUI in S+? Probably not. I would first investigate the R integration plugin of SPSS. There are also some GUI for R, such as Rcmdr. I can't tell if they are a manageable option, as I have never tested it. Another option would be to create a web interface, but R doesn't have CGI and it would require interfacing with another language such as perl. Anyway, I would not recommend anybody to start an S+ GUI project.

What to expect of pROC in the future? Let me say I don't expect to stop its development soon. The version 1.5 of pROC, which will be released very soon, has a new sample size computation function, and a dedicated window in S+. In the long term (> 1 year) however, the situation might change. I will leave the University of Geneva in just over a year; will I still have access to S+? This problem is typical from proprietary software and its licensing. I won't pay for it, and I don't know if TIBCO provides licenses for application developers.

pROC now on GitHub

2011-09-13T12:19:38+02:00

You can now follow pROC development on GitHub!

Among the upcoming features: sample size computation, variance and covariance functions, and much more… stay tuned!

pROC 1.4.4 released

2011-08-10T09:22:57+02:00

I just released pROC version 1.4.4.

This version fixes a critical bug in the computation of one-tailed ROC tests p values. Please update to this version if you perform one-tailed tests.

library(pROC)
data(aSAH)
rocs100 <- roc(aSAH$outcome, aSAH$s100)
rocwfns <- roc(aSAH$outcome, aSAH$wfns)
roc.test(rocs100, rocwfns, alternative="greater")
roc.test(rocs100, rocwfns, alternative="less")

Thanks to Lisa Koch for pointing out the problem.

Here is the full change log:

Fixed alternative for one-tailed tests.
Removed COPYING file to fix a warning in r-devel.

As usual, you can find the new version on ExPASy and on the CRAN. To update, type update.packages() or install.packages("pROC") if you want to update pROC only.

pROC 1.4.3 released

2011-03-19T09:42:12+01:00

Following the publication of our paper, I release a new version. The only change is the updated citation. To see it, type:

library(pROC)
citation("pROC")

Here is the full change log:

Updated citation.

As always, you can find the new version on ExPASy and on the CRAN. To update, type update.packages() or install.packages("pROC") if you want to update pROC only.

pROC paper published

2011-03-17T15:08:36+01:00

The paper is now published online.

The full reference is: Robin X., Turck N., Hainard A., Tiberti N., Lisacek F., Sanchez J.-C., Müller M. (2011). pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 12, p 77. PMID: 21414208. DOI: 10.1186/1471-2105-12-77.

2011-03-19 follow-up: the paper is now on pubmed.

pROC paper accepted

2011-03-15T09:12:01+01:00

Paper accepted! Hopefully it should be available online in a few days.

pROC 1.4.2 released

2011-03-03T13:06:18+01:00

I just released the version 1.4.2 of pROC. It is a minor version with only a few bug fixes

Here is the full change log:

Fixed bootstrap roc.test generating NAs when smooth.rocs were used with reuse.auc=FALSE (thanks to Buddy for the report).
Documented a warning that was missing in roc.test.
Updated citation.

As usual, you can find the new version on ExPASy and on the CRAN. To update, type update.packages() or install.packages("pROC") if you want to update pROC only.