# pROC 1.12.1

A major regression slipped into yesterday's release 1.12.0 of pROC. The fix of issue #25 allocated a matrix which could quickly become very large for bigger datasets, and cause significant slow downs as the machine allocated the memory, or even crash pROC with an error message such as `cannot allocate vector of size ...`

for larger datasets. This bug (issue #29) is fixed (and now automatically tested for) in pROC 1.12.1.

Please update your installation by simply typing:

install.packages("pROC")

Xavier Robin

Published Sunday, May 6, 2018 15:44 CEST

Permalink: /blog/2018/05/06/proc-1.12.1

Tags:

Comments: 0

# pROC 1.12.0

I just released pROC 1.12.0, which fixes several bugs and should significantly improve the performance on large datasets by selecting the best algorithm automatically.

## Issue 25

GitHub issue #25 identified two distinct bugs causing `ci.auc`

, `var`

and other functions to fail when calculating DeLong placements with the following error message, with different variation of numbers:

Error in delongPlacements(roc) : A problem occured while calculating DeLong's theta: got 0.50057161522129678399 instead of 0.50032663726931247972. This is a bug in pROC, please report it to the maintainer.

pROC calculates the AUC with the trapezoidal method. This is the AUC obtained when calling `auc(roc)`

. When using the DeLong method (for `roc.test`

, `var`

etc.), the AUC is also calculated with an other method (similar to the Wilcoxon/Mann-Whitney statistic). These two values should be identical, at least down to something close to the floating point precision of the hardware, typically below 10^-8. To be on the safe side, pROC checks this assumption after calculating the DeLong AUC.

The first sub-issue caused by a broken conversion of the roc curve from percent to fractions for internal calculations, followed by a broken check that the DeLong code produced the correct AUC. In combination, these two bugs caused pROC to stop with an error message in the specific case where `percent=TRUE`

and `direction=">"`

. The check was introduced in pROC version 1.9.1. The bug in the conversion from percent to fraction was present in earlier versions, however it never affected calculations, which is why it was left unnoticed until the check was added. Both bugs are now fixed.

The second sub-issue was impacting the calculation of the thresholds. When two predictor values were too close, their mean could not be represented exactly in the IEEE 754 arithmetic and the result would rounded back to one or the other value, pretty much arbitrarily depending on the implementation.

a <- c(0.65354946699793847742, 0.65354946699793858844) > print(mean(a), digits = 20) [1] 0.65354946699793847742 > mean(a) == a [1] TRUE FALSE

Because pROC calculates the ROC thresholds as the mean between consecutive predictor observations, this would cause some comparisons to be incorrect when calculating sensitivity and specificity. As a consequence, erroneous sensitivities, sensitivities and AUCs may have been reported in the past. The issue was fixed by carefully selecting the correct value for threshold in case the mean was identical to a predictor value.

## Other bug fixes

- GitHub issue #27 caused
`ci.auc`

to return`NaN`

when cases or controls contained only a single observation. The function has been fixed and now returns`NA`

as expected. `power.roc.curve`

failed with ROC curves having`percent=TRUE`

. This issue was identified when adding testthat unit tests for the function.`ci(..., of="coords")`

returned the`ci`

function instead of calculating the CI.- C++ code now check for user interrupts regularly with
`Rcpp::checkUserInterrupt()`

so that very long runs can be aborted from R. - A better error message (instead of a useless internal garbage message error) is now displayed when attempting to return
`threshold`

with`ci.coords`

. An empirical ROC curve, like one produced by pROC, is is made of discrete points that concentrate all the possible thresholds. Lines are added to join the points visually on the plot, however they do not contain any actual threshold. Returning thresholds at an arbitrary sensitivity or specificity requires either to be very lucky to have a point at the exact desired value, or to interpolate thresholds between the points. Interpolating is more tricky than it sounds and is very sensitive to the method of calculation of the threshold (very different results will be returned by pROC that uses the mean between consecutive predictor, than by some other packages which use the values directly).

## New algorithm

A new "meta" algorithm is introduced in the `roc`

function. `algorithm = 5`

causes pROC to automatically select algorithm 2 or 3, based on the number of threshold of the ROC curve. Algorithm 3 has a time complexity of O(N^2). It behaves very well when the number of thresholds remains low. However its square term can cause a catastrophic slowdown with very large datasets where the predictor takes nearly continuous values, and the ROC curve contains many thresholds (typically this will become very obvious above 10000 thresholds). Algorithm 2 has an algorithmic complexity of O(N), and shows a much better performance with large number of data points. However it comes with a rather large pre-factor which makes it very inefficient in most small- to normal-sized datasets. The decisive factor is the number of thresholds, and pROC will select algorithm 2 in curves with more than about 1500 thresholds, 3 otherwise. Algorithm 5 is now the default algorithm in pROC which should significantly improve the performance with large datasets, without impacting the speed in most cases.

## Getting the update

The update has just been submitted to the CRAN and should be online soon. Once it is out, update your installation by simply typing:

install.packages("pROC")

The full changelog is:

- Fix bug that crashed DeLong calculations when predictor had near-ties close to the floating point precision limit that were rounded back to a predictor value (issue #25).
- Fix bug that crashed
`ci.auc`

and`var`

if`direction`

was`">"`

and`percent=TRUE`

(issue #25). - Fix bug causing
`ci`

to return`NaN`

values with`method="delong"`

when cases or controls had a single observation (issue #27). - Fix
`power.roc.curve`

failed with curves having`percent=TRUE`

. - Fix
`ci(..., of="coords")`

returned the`ci`

function instead of the CI. - C++ code now check for user interrupts regularly with
`Rcpp::checkUserInterrupt()`

. - Better error message for
`ci.coords`

attempting to return`threshold`

. - New algorithm = 5 (used by default) chooses the algorithm based on the number of thresholds to avoid worst case with algorithm = 3.

Xavier Robin

Published Saturday, May 5, 2018 13:50 CEST

Permalink: /blog/2018/05/05/proc-1.12.0

Tags:
pROC

Comments: 0

# pROC 1.11.0

pROC 1.11.0 is now on CRAN! This is a minor update that mostly fixes notes in CRAN checks. It also adds support for the `legacy.axes`

argument to change the axis labeling in `ggroc`

.

The full changelog is:

- Added argument
`legacy.axes`

to`ggroc`

- Fix NOTE about "apparent S3 methods exported but not registered" in
`R CMD check`

Xavier Robin

Published Sunday, March 25, 2018 15:04 CEST

Permalink: /blog/2018/03/25/proc-1.11.0

Tags:
pROC

Comments: 0

# pROC 1.10.0

A new update of pROC is now available on CRAN: version 1.10.0.

## ggplot2 support (Experimental)

A new function was introduced: `ggroc`

. Given a `roc`

object, or a (optionally named) `list`

of `roc`

objects, it returns a ggplot object, that can then be printed, with optional aesthetics, themes etc. Here is a basic example:

library(pROC) # Create a basic roc object data(aSAH) rocobj <- roc(aSAH$outcome, aSAH$s100b) rocobj2 <- roc(aSAH$outcome, aSAH$wfns) library(ggplot2) # Multiple curves: gg2 <- ggroc(list(s100b=rocobj, wfns=rocobj2)) gg2

Basic ggplot with two ROC curves.

The usual ggplot syntax applies, so you can add themes, labels, etc. Note the `aes`

argument, which control the aesthetics for geom_line to map to the different ROC curves supplied. Here we use `"linetype"`

instead of the default color:

# with additional aesthetics: gg2b <- ggroc(list(s100b=rocobj, wfns=rocobj2), aes="linetype", color="red") # You can then your own theme, etc. gg2b + theme_minimal() + ggtitle("My ROC curve")

Basic ggplot with two ROC curves.

This functionality is currently experimental and subject to change. Please report bugs and feedback on pROC's GitHub issue tracker.

## Precision and recall in `coords`

The `coords`

function supports two new `ret`

values: `"precision"`

and `"recall"`

:

library(pROC) # Create a basic roc object data(aSAH) rocobj <- roc(aSAH$outcome, aSAH$s100b) coords(rocobj, "best", ret = c("threshold", "sensitivity", "specificity", "precision", "recall")) threshold sensitivity specificity precision recall 0.2050000 0.6341463 0.8055556 0.6500000 0.6341463

It makes it very easy to get a Precision-Recall (PR) plot:

plot(precision ~ recall, t(coords(rocobj, "all", ret = c("recall", "precision"))), type="l", main = "PR plot of S100B")

A simple PR plot.

## Automated testing

Several functions are now covered with tests (powered by the testthat package) to ensure correct behavior. This allowed me to find and fix a few glitches. It will also make it easier to refactor code in the future.

The tests are automatically run by `R CMD check`

. Additional tests that are too slow to be enabled by defauld can be activated with the `RUN_SLOW_TESTS`

environment variable.

export RUN_SLOW_TESTS=true R CMD check pROC

Test results can be seen on Travis CI, and the coverage of the tests can be seen on Codecov. Currently 30% of the code is tested. This includes most functionality, with the exception of bootstrapping and smoothing which I plan to implement in the future.

## Obtaining the update

To update your installation, simply type:

install.packages("pROC")

Here is the full changelog:

- Basic ggplot2 support (one and multiple ROC curves)
- Implement
`precision`

and`recall`

for`coords`

- Fix: properly handle NAs in cases when passing cases/controls to
`roc`

(thanks Thomas König for the report) - Fix various minor bugs detected with new unit tests

Xavier Robin

Published Sunday, June 11, 2017 08:03 CEST

Permalink: /blog/2017/06/11/proc-1.10.0

Tags:
pROC

Comments: 0

# Php's htmlspecialchars stupid behavior

Can php's htmlspecialchars delete your data? The answer, unfortunately, is yes.

I just updated a database server with a web interface from PHP 5.3 (in Ubuntu 12.04) to PHP 7 (Ubuntu 16.04.2). It went pretty smoothly, but after a couple of weeks, users started reporting missing data in some fields where they were expecting some. After some investigation, it turns out the curlpit is the `htmlspecialchars`

function, which changed behaviour with the update. Given the following script:

<?php $string = "An e acute character: \xE9\n"; echo htmlspecialchars($string); ?>

In PHP 5.3, it would output:

An e acute character: �

Now with PHP >= 5.4, here's the output:

Yep, that's correct: the output is empty. PHP just discarded the whole string. Without even a warning!

While this is documented in the manual, this is the most stupid and destructive design I have seen in a long while. Data loss guaranteed when the user saves the page without realizing some fields are accidentally empty! How can anyone be so brain dead and design and implement such a behaviour? Without even a warning!

It turns out one has to define the encoding for the function to work with non-UTF8 characters:

htmlspecialchars($string, ENT_COMPAT,'ISO-8859-1', true);

As this is a legacy application dating back more than 15 years, I fully expect some strings to be broken beyond repair. Thus I wrote the following function to replace all the calls to `htmlspecialchars`

:

function safe_htmlspecialchars($string) { $htmlstring = htmlspecialchars($string, ENT_COMPAT,'ISO-8859-1', true); if (strlen($string) > 0 && strlen($htmlstring) == 0) { trigger_error ("htmlspecialchars failed to convert data", E_USER_ERROR); } }

Displaying an error in case of doubt is the only sensible behaviour here, and should be the default.

Moral of the story: I'm never using PHP in a new project again. And neither should you, if you value your data more than the PHP developers who clearly don't.

Xavier Robin

Published Sunday, February 26, 2017 14:25 CET

Permalink: /blog/2017/02/26/php-s-htmlspecialchars-stupid-behavior

Tags:
Programming

Comments: 0

# pROC 1.9.1

After nearly two years since the previous release, pROC 1.9.1 is finally available on CRAN. Here is a list of the main changes:

`subset`

and`na.action`

arguments now handled properly in`roc.formula`

. This means you can now do something like this:data(aSAH) roc(outcome ~ s100b, data=aSAH, subset=(gender == "Male")) roc(outcome ~ s100b, data=aSAH, subset=(gender == "Female"))

Thanks Terry Therneau for the report.- Added policies to handle the case where a ROC curve has multiple "best" threshold in
`ci.coords`

. The following policies are available:- "stop" will abort the processing and throw an error (with
`stop`

). This is the default. - "omit" will ignore the sample (as in
`NULL`

). This can lead to a reduced effective number of usable sample in the final statistic. - "random" will select one of the threshold randomly.

data(aSAH) ci.coords(aSAH$outcome, aSAH$s100b, x="best", input = "threshold", ret=c("specificity", "ppv", "tp"), best.policy = "random")

Thanks Nicola Toschi for the report. - "stop" will abort the processing and throw an error (with
- Support
`xlim`

and`ylim`

gracefully in`plot.roc`

. - Improved validation of input class
`levels`

and`direction`

; A message can be printed when auto-detecting, use the`quiet`

argument to turn on. - Removed extraneous
`name`

attribute on the`p.value`

(thanks Paweł Kleka for the report). - Faster DeLong algorithm (code contributed by Stefan Siegert). The code is based on the algorithm by Xu Sun and Weichao Xu (2014) that has an O(N log N) complexity instead of O(N
^{2}).

The DeLong algorithm is now always faster than bootstrapping, even in the previous edge case of ROC curve with large number of samples and few thresholds where bootstrapping used to be faster. Here is a quick example with 200000 data points:library(pROC) n <- 200000 a <- as.numeric(cut(rnorm(n), c(-Inf, -1, 0, 1, Inf))) b <- round(runif(n)) r <- roc(b, a, algorithm = 3) # With Bootstrap > system.time(var(r, method = "b", progress = "none")) utilisateur système écoulé 25.896 0.136 26.027 # With old DeLong algorithm > system.time(var(r, method = "d")) utilisateur système écoulé 47.352 0.008 47.353 # With new DeLong algorithm > system.time(var(r, method = "d")) utilisateur système écoulé 0.016 0.008 0.023

## Obtaining the update

To update your installation, simply type:

install.packages("pROC")

## References

Xu Sun and Weichao Xu (2014) "Fast Implementation of DeLongs Algorithm for Comparing
the Areas Under Correlated Receiver Operating Characteristic Curves". *IEEE Signal
Processing Letters*, **21**, 1389-1393. DOI: 10.1109/LSP.2014.2337313.

Xavier Robin

Published Monday, February 6, 2017 09:08 CET

Permalink: /blog/2017/02/06/proc-1.9.1

Tags:
pROC

Comments: 0

# pROC 1.8 is coming with some potential backward-incompatible changes in the namespace

The last significant update of pROC, 1.7, was released a year ago, followed by some minor bug fix updates. In the meantime, the policies of the CRAN repository evolved, and are requiring a significant update of pROC.

Specifically, S3 methods in pROC have always been exported, which means that you could call `auc.roc`

or `roc.formula`

directly. This is not allowed any longer, and methods must now to be registered as such with `S3method()`

calls in the `NAMESPACE`

file. The upcoming version of pROC (1.8) will therefore feature a major cleanup of the namespace.

In practice, this could potentially break some of your code. Specifically, direct call to S3 methods will not work any longer. For instance, the following is incorrect:

rocobj <- roc(...) smooth.roc(rocobj)

Although not documented, it used to work but that will no longer be the case. Instead, you should call the generic function that will dispatch to the proper method:

smooth(rocobj)

Other examples include for instance:

# Incorrect: auc.roc(rocobj) # Correct: auc(rocobj) # Incorrect: var.roc(rocobj) # Correct: var(rocobj)

Please make sure you replace any call to a method with the generic. In doubt, consult the *Usage* section of pROC's manual.

Xavier Robin

Published Monday, February 23, 2015 23:13 CET

Permalink: /blog/2015/02/23/proc-1.8-is-coming-with-some-potential-backward-incompatible-changes-in-the-namespace

Tags:
pROC

Comments: 0

# pROC 1.7.3 bugfix release

pROC 1.7.3 was pushed to the CRAN a few minutes ago. It is a bugfix release that solves two issues with smoothing, the first of which is a significant numeric issue:

- Fixed AUC of binomial-smoothed ROC off by 100^2 (thanks Bao-Li Chang for the report)
- Fix print of logcondens-smoothed ROC

It should be available for update from CRAN in a few hours / days, depending on your operating system.

Xavier Robin

Published Thursday, June 12, 2014 20:34 CEST

Permalink: /blog/2014/06/12/proc-1.7.3

Tags:
pROC

Comments: 0

# pROC 1.7.2

pROC 1.7.2 was published this morning. It is a bugfix release that primarily solves various issues with `coords`

and `ci.coords`

. It also warns when computing confidence intervals / roc tests of a ROC curves with AUC == 1 (the CI will always be 1-1 / p value 0) as this can potentially be misleading.

- Fixed bug where
`ci.coords`

with`x="best"`

would fail if one or more resampled ROC curve had multiple "best" thresholds (thanks Berend Terluin for the report) - Fixed bug in
`ci.coords`

: passing more than one value in`x`

now works - Fixed typo in documentation of
`direction`

argument to`roc`

(thanks Le Kang for the report) - Add a warning when computing statistics of ROC curve with AUC = 1
- Require latest version of Rcpp to avoid weird errors (thanks Tom Liptrot for the report)

Xavier Robin

Published Sunday, April 6, 2014 08:49 CEST

Permalink: /blog/2014/04/06/proc-1.7.2

Tags:
pROC

Comments: 0

# pROC 1.7 released

pROC 1.7 was released. It provides additional speed improvements with the DeLong calculations now implemented with Rcpp, improved behaviour with math operations, and various bug fixes. It is now possible to pass multiple predictors in a formula: a list of ROC curves is returned. In details:

- Faster algorithm for DeLong
`roc.test`

,`power.roc.test`

,`ci.auc`

,`var`

and`cov`

function (no large matrix allocation) - Handling Math and Operations correctly on
`auc`

and`ci`

objects (see`?groupGeneric.pROC`

) - The
`formula`

for`roc.formula`

can now provide several predictors and a list of ROC curves will be returned - Fixed documentation of
`ci.coords`

with examples - Fixed binormal AUC computed with triangulation despite the claim in the documentation
- Fixed unstated requirement on Rcpp >= 0.10.5

pROC 1.7.1 is an quick fix release to get the package on CRAN.

- Close SOCK cluster on Windows with parallel=TRUE
- Fixed really use algorithm 1 when microbenchmark fails

Xavier Robin

Published Thursday, February 20, 2014 21:48 CET

Permalink: /blog/2014/02/20/proc-1.7-released

Tags:
pROC

Comments: 0