Xavier Robin – Tag – pROC

pROC 1.19.0

2025-07-30T18:56:51+02:00

pROC version 1.19.0 was just released and will be available on CRAN very soon.

Besides minor changes and fixes in the coords and ci.coords functions, the main updates in this version focus on the core of the package, aiming to make it more modern, efficient, and easier to maintain. Several features that were difficult to maintain have been deprecated.

The dependency on the retired plyr package has been removed (thanks to Michael Chirico for his contributions). Unfortunately, as a side effect, progress bars and parallel processing have been removed. Trying to set the progress and parallel arguments will now trigger a warning, and the arguments will be ignored.
As a followup from the changes to the output of the coords function in version 1.16.0, the transpose, as.list, as.matrix and drop arguments have been deprecated. The coords function currently has multiple exit points, depending on arguments and inputs, and can return retults in mutliple formats. This makes it difficult to use and to maintain. Going forward, coords will only retrun data in a single, tidy data.frame format, compatible following modern R coding practices. For compatibility reasons, deprecated arguments are still available, but setting them to non default values will trigger warnings. They will be removed in a future release. If your code still uses these arguments, please update it accordingly.
Finally, following years of performance improvements, an in an effort to simplify the codebase, all ROC computation algorithms other than 2 have been removed as they no longer provided meaningful performance advantages. The algorithm argument to roc has been deprecated. Setting it to a non-default value has no effect and triggers a warning. The fun.sesp value of roc objects is also deprecated. Calling it triggers a warning. Both will be removed in a future release of pROC.

Here is the full changelog:

ci.coords can now take the same input values as coords (issue #90)
ci.coords can be plotted
Added "lr_pos" and "lr_neg" to coords (issue #102)
coords with partial.auc now interpolates bounds when needed
Added ignore.partial.auc argument to coords
Deprecated transpose, as.list, as.matrix and drop in coords
Deprecated the algorithm argument to roc and fun.sesp value
Deprecated the progress and parallel argument for bootstrap operations.
Removed dependencies on doParallel and retired package plyr (thanks to Michael Chirico, pr #134, #135, #136, #137, #138, #139 and #140).

You can update your installation by simply typing:

install.packages("pROC")

Update: pROC 1.19.0 was rejected from CRAN. A patch revision 1.19.0.1 was created to workaround an issue with a reverse dependency but provides no meaningful change:

Move fun.sesp definition to work around LudvigOlsen/cvms#44.

pROC 1.18.5

2023-11-02T17:01:20+01:00

pROC 1.18.5 is now available on CRAN. It's a minor bugfix release:

Fixed formula input when given as variable and combined with with (issue #111)
Fixed formula containing variables with spaces (issue #120)
Fixed broken grouping when colour argument was given in ggroc (issue #121)

You can update your installation by simply typing:

install.packages("pROC")

pROC 1.18.0

2021-09-06T18:34:01+02:00

pROC version 1.18.0 is now available on CRAN now. Only a few changes were implemented in this release:

Add CI of the estimate for roc.test (DeLong, paired only for now) (code contributed by Zane Billings) (issue #95).
Fix documentation and alternative hypothesis for Venkatraman test (issue #92).

You can update your installation by simply typing:

install.packages("pROC")

pROC 1.17.0.1

2021-01-13T16:19:16+01:00

pROC version 1.17.0.1 is available on CRAN now. Besides several bug fixes and small changes, it introduces more values in input of coords.

Here is an example:

library(pROC)
data(aSAH)
rocobj <- roc(aSAH$outcome, aSAH$s100b)
coords(rocobj, x = seq(0, 1, .1), input="recall", ret="precision")
#    precision
# 1        NaN
# 2  1.0000000
# 3  1.0000000
# 4  0.8601399
# 5  0.6721311
# 6  0.6307692
# 7  0.6373057
# 8  0.4803347
# 9  0.4517906
# 10 0.3997833
# 11 0.3628319

Getting the update

The update his available on CRAN now. You can update your installation by simply typing:

install.packages("pROC")

Here is the full changelog:

1.17.0.1 (2020-01-07):

Fix CRAN incoming checks as requested by CRAN.

1.17.0 (2020-12-29)

Accept more values in input of coords (issue #67).
Accept kappa for the power.roc.test of two ROC curves (issue #82).
The input argument to coords for smooth.roc curves no longer has a default.
The x argument to coords for smooth.roc can now be set to all (also the default).
Fix bootstrap roc.test and cov with smooth.roc curves.
The ggroc function can now plot smooth.roc curves (issue #86).
Remove warnings with warnPartialMatchDollar option (issue #87).
Make tests depending on vdiffr conditional (issue #88).

pROC 1.16.1

2020-01-14T08:52:57+01:00

pROC version 1.16.1 is a minor release that disables a timing-dependent test based on the microbenchmark package that can sometimes cause random failures on CRAN. This version contains no user-visible changes. Users don't need to install this update.

pROC 1.16.0

2020-01-12T21:46:00+01:00

pROC version 1.16.0 is available on CRAN now. Besides several bug fixes, the main change is the switch of the default value of the transpose argument to the coords function from TRUE to FALSE. As announced earlier, this is a backward incompatible change that will break any script that did not previously set the transpose argument and for now comes with a warning to make debugging easier. Scripts that set transpose explicitly are not unaffected.

New return values of `coords` and `ci.coords`

With transpose = FALSE, the coords returns a tidy data.frame suitable for use in pipelines:

data(aSAH)
rocobj <- roc(aSAH$outcome, aSAH$s100b)
coords(rocobj, c(0.05, 0.2, 0.5), transpose = FALSE)
#      threshold specificity sensitivity
# 0.05      0.05  0.06944444   0.9756098
# 0.2       0.20  0.80555556   0.6341463
# 0.5       0.50  0.97222222   0.2926829

The function doesn't drop dimensions, so the result is always a data.frame, even if it has only one row and/or one column.

If speed is of utmost importance, you can get the results as a non-transposed matrix instead:

coords(rocobj, c(0.05, 0.2, 0.5), transpose = FALSE, as.matrix = TRUE)
#      threshold specificity sensitivity
# [1,]      0.05  0.06944444   0.9756098
# [2,]      0.20  0.80555556   0.6341463
# [3,]      0.50  0.97222222   0.2926829

In some scenarios this can be a tiny bit faster, and is used internally in ci.coords.

Type help(coords_transpose) for additional information.

`ci.coords`

The ci.coords function now returns a list-like object:

ciobj <- ci.coords(rocobj, c(0.05, 0.2, 0.5))
ciobj$accuracy
#        2.5%       50%     97.5%
# 1 0.3628319 0.3982301 0.4424779
# 2 0.6637168 0.7433628 0.8141593
# 3 0.6725664 0.7256637 0.7787611

The print function prints a table with all the results, however this table is generated on the fly and not available directly.

ciobj
# 95% CI (2000 stratified bootstrap replicates):
#      threshold sensitivity.low sensitivity.median sensitivity.high
# 0.05      0.05          0.9268             0.9756           1.0000
# 0.2       0.20          0.4878             0.6341           0.7805
# 0.5       0.50          0.1707             0.2927           0.4390
#      specificity.low specificity.median specificity.high accuracy.low
# 0.05         0.01389            0.06944           0.1250       0.3628
# 0.2          0.70830            0.80560           0.8889       0.6637
# 0.5          0.93060            0.97220           1.0000       0.6726
#      accuracy.median accuracy.high
# 0.05          0.3982        0.4425
# 0.2           0.7434        0.8142
# 0.5           0.7257        0.7788

The following code snippet can be used to obtain all the information calculated by the function:

for (ret in attr(ciobj, "ret")) {
	print(ciobj[[ret]])
}
#        2.5%       50%     97.5%
# 1 0.9268293 0.9756098 1.0000000
# 2 0.4878049 0.6341463 0.7804878
# 3 0.1707317 0.2926829 0.4390244
#         2.5%        50%     97.5%
# 1 0.01388889 0.06944444 0.1250000
# 2 0.70833333 0.80555556 0.8888889
# 3 0.93055556 0.97222222 1.0000000
#        2.5%       50%     97.5%
# 1 0.3628319 0.3982301 0.4424779
# 2 0.6637168 0.7433628 0.8141593
# 3 0.6725664 0.7256637 0.7787611

Getting the update

The update his available on CRAN now. You can update your installation by simply typing:

install.packages("pROC")

Here is the full changelog:

BACKWARD INCOMPATIBLE CHANGE: transpose argument to coords switched to FALSE by default (issue #54).
BACKWARD INCOMPATIBLE CHANGE: ci.coords return value is now of list type and easier to use.
Fix one-sided DeLong test for curves with direction=">" (issue #64).
Fix an error in ci.coords due to expected NA values in some coords (like "precision") (issue #65).
Ordrered predictors are converted to numeric in a more robust way (issue #63).
Cleaned up power.roc.test code (issue #50).
Fix pairing with roc.formula and warn if na.action is not set to "na.pass" or "na.fail" (issue #68).
Fix ci.coords not working with smooth.roc curves.

pROC 1.15.3

2019-07-22T09:07:57+02:00

A new version of pROC, 1.15.3, has been released and is now available on CRAN. It is a minor bugfix release. Versions 1.15.1 and 1.15.2 were rejected from CRAN.

Here is the full changelog:

Fix -Inf threshold in coords for curves with direction = ">" (issue 60).
Keep list order in ggroc (issue 58).
Fix erroneous error in ci.coords with ret="threshold" (issue 57).
Restore lazy loading of the data and fix an R CMD check warning "Variables with usage in documentation object 'aSAH' not in code".
Fix vdiffr unit tests with ggplot2 3.2.0 (issue 53).

pROC 1.15.0

2019-06-01T09:33:08+02:00

The latest version of pROC, 1.15.0 has just been released. It features significant speed improvements, many bug fixes, new methods for use in dplyr pipelines, increased verbosity, and prepares the way for some backwards-incompatible changes upcoming in pROC 1.16.0.

Verbosity

Since its initial release, pROC has been detecting the levels of the positive and negative classes (cases and controls), as well as the direction of the comparison, that is whether values are higher in case or in control observations. Until now it has been doing so silently, but this has lead to several issues and misunderstandings in the past. In particular, because of the detection of direction, ROC curves in pROC will nearly always have an AUC higher than 0.5, which can at times hide problems with certain classifiers, or cause bias in resampling operations such as bootstrapping or cross-validation.

In order to increase transparency, pROC 1.15.0 now prints a message on the command line when it auto-detects one of these two arguments.

	> roc(aSAH$outcome, aSAH$ndka)
	Setting levels: control = Good, case = Poor
	Setting direction: controls < cases

	Call:
	roc.default(response = aSAH$outcome, predictor = aSAH$ndka)

	Data: aSAH$ndka in 72 controls (aSAH$outcome Good) < 41 cases (aSAH$outcome Poor).
	Area under the curve: 0.612

If you run pROC repeatedly in loops, you may want to turn off these diagnostic messsages. The recommended way is to explicitly specify them explicitly:

	roc(aSAH$outcome, aSAH$ndka, levels = c("Good", "Poor"), direction = "<")

Alternatively you can pass quiet = TRUE to the ROC function to silenty ignore them.

	roc(aSAH$outcome, aSAH$ndka, quiet = TRUE)

As mentioned earlier this last option should be avoided when you are resampling, such as in bootstrap or cross-validation, as this could silently hide some biases due to changing directions.

Speed

Several bottlenecks have been removed, yielding significant speedups in the roc function with algorithm = 2 (see issue 44), as well as in the coords function which is now vectorized much more efficiently (see issue 52) and scales much better with the number of coordinates to calculate. With these improvements pROC is now as fast as other ROC R packages such as ROCR.

With Big Data becoming more and more prevalent, every speed up matters and making pROC faster has very high priority. If you think that a particular computation is abnormally slow, for instance with a particular combination of arguments, feel free to submit a bug report.

As a consequence, algorithm = 2 is now used by default for numeric predictors, and is automatically selected by the new algorithm = 6 meta algorithm. algorithm = 3 remains slightly faster with very low numbers of thresholds (below 50) and is still the default with ordered factor predictors.

Pipelines

The roc function can be used in pipelines, for instance with dplyr or magrittr. This is still a highly experimental feature and will change significantly in future versions (see issue 54 for instance). Here is an example of usage:

library(dplyr)
aSAH %>% 
    filter(gender == "Female") %>% 
    roc(outcome, s100b)

The roc.data.frame method supports both standard and non-standard evaluation (NSE), and the roc_ function supports standard evaluation only. By default it returns the roc object, which can then be piped to the coords function to extract coordinates that can be used in further pipelines

aSAH %>%
    filter(gender == "Female") %>%
    roc(outcome, s100b) %>%
	coords(transpose=FALSE) %>%
    filter(sensitivity > 0.6,
           specificity > 0.6)

More details and use cases are available in the ?roc help page.

Transposing coordinates

Since the initial release of pROC, the coords function has been returning a matrix with thresholds in columns, and the coordinate variables in rows.

data(aSAH)
rocobj <- roc(aSAH$outcome, aSAH$s100b)
coords(rocobj, c(0.05, 0.2, 0.5))
#                   0.05       0.2       0.5
# threshold   0.05000000 0.2000000 0.5000000
# specificity 0.06944444 0.8055556 0.9722222
# sensitivity 0.97560976 0.6341463 0.2926829

This format doesn't conform to the grammar of the tidyverse, outlined by Hadley Wickham in his Tidy Data 2014 paper, which has become prevalent in modern R language. In addition, the dropping of dimensions by default makes it difficult to guess what type of data coords is going to return.

	coords(rocobj, "best")
	#   threshold specificity sensitivity 
	#   0.2050000   0.8055556   0.6341463 
	# A numeric vector

Although it is possible to pass drop = FALSE, the fact that it is not the default makes the behaviour unintuitive. In an upcoming version of pROC, this will be changed and coords will return a data.frame with the thresholds in rows and measurement in colums by default.

Changes in 1.15

Addition of the transpose argument.
Display a warning if transpose is missing. Pass transpose explicitly to silence the warning.
Deprecation of as.list.

With transpose = FALSE, the output is a tidy data.frame suitable for use in pipelines:

 coords(rocobj, c(0.05, 0.2, 0.5), transpose = FALSE)
#      threshold specificity sensitivity
# 0.05      0.05  0.06944444   0.9756098
# 0.2       0.20  0.80555556   0.6341463
# 0.5       0.50  0.97222222   0.2926829

It is recommended that new developments set transpose = FALSE explicitly. Currently these changes are neutral to the API and do not affect functionality outside of a warning.

Upcoming backwards incompatible changes in future version (1.16)

The next version of pROC will change the default transpose to FALSE. This is a backward incompatible change that will break any script that did not previously set transpose and will initially come with a warning to make debugging easier. Scripts that set transpose explicitly will be unaffected.

Recommendations

If you are writing a script calling the coords function, set transpose = FALSE to silence the warning and make sure your script keeps running smoothly once the default transpose is changed to FALSE. It is also possible to set transpose = TRUE to keep the current behavior, however is likely to be deprecated in the long term, and ultimately dropped.

New `coords` return values

The coords function can now return two new values, "youden" and "closest.topleft". They can be returned regardless of whether input = "best" and of the value of the best.method argument, although they will not be re-calculated if possible. They follow the best.weights argument as expected. See issue 48 for more information.

Bug fixes

Several small bugs have been fixed in this version of pROC. Most of them were identified thanks to an increased unit test coverage. 65% of the code is now unit tested, up from 46% a year ago. The main weak points remain the testing of all bootstrapping and resampling operations. If you notice any unexpected or wrong behavior in those, or in any other function, feel free to submit a bug report.

Getting the update

The update his available on CRAN now. You can update your installation by simply typing:

install.packages("pROC")

Here is the full changelog:

roc now prints messages when autodetecting levels and direction by default. Turn off with quiet = TRUE or set these values explicitly.
Speedup with algorithm = 2 (issue 44) and in coords (issue 52).
New algorithm = 6 (used by default) uses algorithm = 2 for numeric data, and algorithm = 3 for ordered vectors.
New roc.data.frame method and roc_ function for use in pipelines.
coords can now returns "youden" and "closest.topleft" values (issue 48).
New transpose argument for coords, TRUE by default (issue 54).
Use text instead of Tcl/Tk progress bar by default (issue 51).
Fix method = "density" smoothing when called directly from roc (issue 49).
Renamed roc argument n to smooth.n.
Fixed 'are.paired' ignoring smoothing arguments of roc2 with return.paired.rocs.
New ret option "all" in coords (issue 47)
drop in coords now drops the dimension of ret too (issue 43)

pROC 1.14.0

2019-03-13T10:22:42+01:00

pROC 1.14.0 was released with many bug fixes and some new features.

Multiclass ROC

The multiclass.roc function can now take a multivariate input with columns corresponding to scores of the different classes. The columns must be named with the corresponding class labels. Thanks Matthias Döring for the contribution.

Let's see how to use it in practice with the iris dataset. Let's first split the dataset into a training and test sets:

data(iris)
iris.sample <- sample(1:150)
iris.train <- iris[iris.sample[1:75],]
iris.test <- iris[iris.sample[76:150],]

We'll use the nnet package to generate some predictions. We use the type="prob" to the predict function to get class probabilities.

library("nnet")
mn.net <- nnet::multinom(Species ~ ., iris.train)

iris.predictions <- predict(mn.net, newdata=iris.test, type="prob")
head(iris.predictions)

	          setosa   versicolor    virginica
	63  2.877502e-21 1.000000e+00 6.647660e-19
	134 1.726936e-27 9.999346e-01 6.543642e-05
	150 1.074627e-28 7.914019e-03 9.920860e-01
	120 6.687744e-34 9.986586e-01 1.341419e-03
	6   1.000000e+00 1.845491e-24 6.590050e-72
	129 4.094873e-45 1.779882e-15 1.000000e+00

Notice the column names, identical to the class labels. Now we can use the multiclass.roc function directly:

multiclass.roc(iris.test$Species, iris.predictions)

Many modelling functions have similar interfaces, where the output of predict can be changed with an extra argument. Check their documentation to find out how to get the required data.

Multiple aesthetics for `ggroc`

It is now possible to pass several aesthetics to ggroc. So for instance you can map a curve to both colour and linetype:

roc.list <- roc(outcome ~ s100b + ndka + wfns, data = aSAH)
ggroc(roc.list, aes=c("linetype", "color"))

Mapping 3 ROC curves to 2 aesthetics with ggroc.

Getting the update

The update his available on CRAN now. You can update your installation by simply typing:

install.packages("pROC")

Here is the full changelog:

The multiclass.roc function now accepts multivariate decision values (code contributed by Matthias Döring).
ggroc supports multiple aesthetics.
Make ggplot2 dependency optional.
Suggested packages can be installed interactively when required.
Passing both cases and controls or response and predictor arguments is now an error.
Many small bug fixes.

pROC 1.13.0

2018-09-24T20:09:07+02:00

pROC 1.13.0 was just released with bug fixes and a new feature.

Infinite values in predictor

Following the release of pROC 1.12, it quickly became clear with issue #30 that infinite values were handled differently by the different algorithms of pROC. The problem with these values is that they cannot be thresholded. An Inf will always be greater than any value. This means that in some cases, it may not be possible to reach 0 or 100% specificity or sensitivity. This also revealed that threshold-agnostic algorithms such as algorithm="2" or the DeLong theta calculations would happily reach 0 or 100% specificity or sensitivity in those case, although those values are unattainable.

Starting with 1.13.0, when pROC's roc function finds any infinite value in the predictor argument, or in controls or cases, it will return NaN (not a number).

Numerical accuracy

The handling of near ties close to + or - Infinity or 0 has been improved by calculating the threshold (which is the mean between two consecutive values) differently depending on the mean value itself. This allows preserving as much precision close to 0 without maxing out large absolute values.

New argument for ggroc

ggroc can now take a new value for the aes argument, aes="group". Consistent with ggplot2, it allows to curves with identical aesthetics to be split in different groups. This is especially useful for instance in facetted plots.

library(pROC)
data(aSAH)
roc.list <- roc(outcome ~ s100b + ndka + wfns, data = aSAH)
g.list <- ggroc(roc.list)
g.group <- ggroc(roc.list, aes="group")
g.group + facet_grid(.~name)

Facetting of 3 ROC curves with ggroc.

Getting the update

The update has just been accepted on CRAN and should be online soon. Once it is out, update your installation by simply typing:

install.packages("pROC")

The full changelog is:

roc now returns NaN when predictor contains infinite values ( issue #30).
Better handling of near-ties near +-Infinity and 0.
ggroc supports aes="group" to allow curves with identical aesthetics.

Xavier Robin – Tag – pROC

pROC 1.19.0

pROC 1.18.5

pROC 1.18.0

pROC 1.17.0.1

Getting the update

pROC 1.16.1

pROC 1.16.0

New return values of coords and ci.coords

ci.coords

Getting the update

pROC 1.15.3

pROC 1.15.0

Verbosity

Speed

Pipelines

Transposing coordinates

Changes in 1.15

Upcoming backwards incompatible changes in future version (1.16)

Recommendations

New coords return values

Bug fixes

Getting the update

pROC 1.14.0

Multiclass ROC

Multiple aesthetics for ggroc

Getting the update

pROC 1.13.0

Infinite values in predictor

Numerical accuracy

New argument for ggroc

Getting the update

New return values of `coords` and `ci.coords`

`ci.coords`

New `coords` return values

Multiple aesthetics for `ggroc`