Introduction to Lift Chart, ROC Curve and Word Cloud

Connecting to DataRobot

First, let’s load some useful libraries:

library(datarobot)

## Authenticating with config at: /Users/peter.hurford/.config/datarobot/drconfig.yaml

## second argument must be a list

library(knitr)

To access the DataRobot modeling engine, it is necessary to establish an authenticated connection, which can be done in one of two ways. In both cases, the necessary information is an endpoint, the URL address of the specific DataRobot server being used and a token, a previously validated access token.

token is unique for each DataRobot modeling engine account and can be accessed using the DataRobot webapp in the account profile section.

endpoint depends on DataRobot modeling engine installation (cloud-based vs. on-premise) you are using. Contact your DataRobot admin for information on which endpoint to use if you do not know. The endpoint for DataRobot cloud accounts is https://app.datarobot.com/api/v2.

The first access method uses a YAML configuration file with these two elements - labeled token and endpoint - located at $HOME/.config/datarobot/drconfig.yaml. If this file exists when the datarobot package is loaded, a connection to the DataRobot modeling engine is automatically established during library(datarobot). It is also possible to establish a connection using this YAML file via the ConnectToDataRobot function, by specifying the configPath parameter.

The second method of establishing a connection to the DataRobot modeling engine is to call the function ConnectToDataRobot with the endpoint and token parameters.

ConnectToDataRobot(endpoint = "http://<YOUR DR SERVER>/api/v2", token = "<YOUR API TOKEN>")

Data

We will be using the Lending Club dataset, a sample dataset related to credit scoring open-sourced by LendingClub (https://www.lendingclub.com/). We can create a project with this dataset like this:

project <- SetupProject(dataSource = "path/to/lendingclub.csv", projectName = "AdvancedModelInsightsVignette")
SetTarget(project = project, target = "is_bad")
UpdateProject(project = project$projectId, workerCount = 10) # increase the number of workers used by this project

Once the modeling process has completed, the GetAllModels function returns an S3 object of class listOfModels that characterizes all of the models in a specified DataRobot project. It is important to use WaitforAutopilot before calling GetAllModels, as the function will return only a partial list (and a warning) if the autopilot is not yet complete.

WaitForAutopilot(project = project)
results <- as.data.frame(GetAllModels(project))
saveRDS(results, "resultsModelInsights.rds")
kable(head(results), longtable=TRUE, booktabs=TRUE, row.names = TRUE)

	modelType	expandedModel	modelId	blueprintId	featurelistName	featurelistId	samplePct	validationMetric
1	ENET Blender	ENET Blender	598bd864acf4c3074f48bcfc	4bcba7bb67f871eb68d6630b983964f9	Informative Features	598bd636e6292976d5111993	64	0.33815
2	Gradient Boosted Trees Classifier with Early Stopping	Gradient Boosted Trees Classifier with Early Stopping::Tree-based Algorithm Preprocessing v1	598bd74f456a01bbf282fd55	49d909a38fccf878347cb2f54e54455a	Informative Features	598bd636e6292976d5111993	64	0.33865
3	ENET Blender	ENET Blender	598bd864acf4c3074f48bcfe	9249cf3f02a346719a81441452ff6479	Informative Features	598bd636e6292976d5111993	64	0.33901
4	AVG Blender	AVG Blender	598bd863acf4c3074f48bcf8	1eda640fca58fd959981b00481a0ee9a	Informative Features	598bd636e6292976d5111993	64	0.33908
5	Advanced AVG Blender	Advanced AVG Blender	598bd864acf4c3074f48bcfa	9b0635eb46dacac9001aaf7735835c50	Informative Features	598bd636e6292976d5111993	64	0.33937
6	eXtreme Gradient Boosted Trees Classifier with Early Stopping	eXtreme Gradient Boosted Trees Classifier with Early Stopping::Tree-based Algorithm Preprocessing v1	598bd74f456a01bbf282fd59	c12c706e449b1e23a6ec0092b568dc53	Informative Features	598bd636e6292976d5111993	64	0.34132

Lift Chart

Lift chart data can be retreived for a specific data partition (validation, cross-validation, or holdout) or for all the data partitions using GetLiftChart and GetAllLiftCharts. To retreive the data for holdout partition, it needs to be unlocked first.

Let’s retreive the validation partition data for top model using GetLiftChart. The GetLiftChart function returns data for validation partition by default. We can retreive data for specific data partition by passing value to source parameter in GetLiftChart.

project <- GetProject("598b5182962d747b64e6c4f5")
allModels <- GetAllModels(project)
saveRDS(allModels, "modelsModelInsights.rds")
modelFrame <- as.data.frame(allModels)
metric <- modelFrame$validationMetric
if (project$metric %in% c('AUC', 'Gini Norm')) {
  bestIndex <- which.max(metric)
} else {
  bestIndex <- which.min(metric)
}
bestModel <- allModels[[bestIndex]]
bestModel$modelType

[1] “ENET Blender”

This selects an ENET Blender model.

The lift chart data we retrieve from the server includes the mean of the model prediction and the mean of the actual target values, sorted by the prediction values in ascending order and split into up to 60 bins.

lc <- GetLiftChart(bestModel)
saveRDS(lc, "liftChartModelInsights.rds")
head(lc)

  actual  predicted binWeight

1 0.00000000 0.02292314 27 2 0.00000000 0.03038243 27 3 0.03846154 0.03576951 26 4 0.00000000 0.04002537 27 5 0.03703704 0.04346634 27 6 0.00000000 0.04579174 26

ValidationLiftChart <- GetLiftChart(bestModel, source = "validation")
dr_dark_blue <- "#08233F"
dr_blue <- "#1F77B4"
dr_orange <- "#FF7F0E"

# Function to plot lift chart
library(data.table)
LiftChartPlot <- function(ValidationLiftChart, bins = 10) {
  if (60 %% bins == 0) {
    ValidationLiftChart$bins <- rep(seq(bins), each = 60 / bins)
    ValidationLiftChart <- data.table(ValidationLiftChart)
    ValidationLiftChart[, actual := mean(actual), by = bins]
    ValidationLiftChart[, predicted := mean(predicted), by = bins]
    unique(ValidationLiftChart[, -"binWeight"])
  } else {
    "Please provide bins less than 60 and divisor of 60"
  }
}
LiftChartData <- LiftChartPlot(ValidationLiftChart)
saveRDS(LiftChartData, "LiftChartDataVal.rds")
par(bg = dr_dark_blue)
plot(LiftChartData$Actual, col = dr_orange, pch = 20, type = "b", main = "Lift Chart", xlab = "Bins", ylab = "Value")
lines(LiftChartData$Predicted, col = dr_blue, pch = 20, type = "b")

All the available lift chart data can be retreived using GetAllLiftCharts. Here is an example retrieving data for all the available partitions, followed by plotting the cross validation partition:

AllLiftChart <- GetAllLiftCharts(bestModel)
LiftChartData <- LiftChartPlot(AllLiftChart[["crossValidation"]])
saveRDS(LiftChartData, "LiftChartDataCV.rds")
par(bg = dr_dark_blue)
plot(LiftChartData$Actual, col = dr_orange, pch = 20, type = "b", main = "Lift Chart", xlab = "Bins", ylab = "Value")
lines(LiftChartData$Predicted, col = dr_blue, pch = 20, type = "b")

We can also plot the lift chart using ggplot2:

library(ggplot2)
lc$actual <- lc$actual / lc$binWeight
lc$predicted <- lc$predicted / lc$binWeight
lc <- lc[order(lc$predicted),]
lc$id <- seq(nrow(lc))
lc$binWeight <- NULL
lc <- data.frame(value = c(lc$actual, lc$predicted),
                 variable = c(rep("Actual", length(lc$actual)),
                              rep("Predicted", length(lc$predicted))),
                 id = rep(seq_along(lc$actual), 2))
ggplot(lc) + geom_line(aes(x = id, y = value, color = variable))

ROC Curve Data

The receiver operating characteristic curve, or ROC curve, is a graphical plot that illustrates the performance of a binary classifier system as its discrimination threshold is varied. The curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings.

ROC curve data can be generated for a specific data partition (validation, cross validation, or holdout) or for all the data partition using GetRocCurve and GetAllRocCurves.

To retrieve ROC curve information use GetRocCurve:

roc <- GetRocCurve(bestModel)
saveRDS(roc, "ROCCurveModelInsights.rds")
head(roc)

$source [1] “validation”

$negativeClassPredictions [1] 0.07813619 0.05756771 0.06973542 0.12628534 0.14795599 0.07071595 [7] 0.08506275 0.11053543 0.10947159 0.07039247 0.10654207 0.33836080 [13] 0.09876669 0.13912130 0.06564848 0.13607946 0.12776674 0.03550797 [19] 0.14991779 0.08946280 0.06096304 0.09984772 0.09068335 0.09824618 [25] 0.09479633 0.05736648 0.14051878 0.08998919 0.11025208 0.06972913 [31] 0.15547657 0.05446337 0.17036438 0.05885326 0.09122318 0.06892613 [37] 0.15197364 0.09144881 0.05135055 0.09067535 0.05992180 0.06564848 [43] 0.07833158 0.06210313 0.07834012 0.08844425 0.12542433 0.12903450 [49] 0.09622956 0.10265134 0.12303826 0.25018175 0.12882174 0.11105994 [55] 0.04252059 0.15718788 0.10427738 0.14001611 0.08601192 0.04355966 [61] 0.04404462 0.11347792 0.08235885 0.21988016 0.19063393 0.07606801 [67] 0.08517325 0.18430950 0.09082427 0.03942690 0.05458829 0.07304322 [73] 0.21085018 0.13814910 0.07015611 0.02650754 0.11553201 0.08409508 [79] 0.04298015 0.09460796 0.06704933 0.17072363 0.06564848 0.05844530 [85] 0.15740470 0.12884679 0.13975098 0.05082041 0.09730717 0.10583388 [91] 0.07838485 0.08193712 0.06435627 0.03928342 0.11956921 0.19764768 [97] 0.11007938 0.15539308 0.14402286 0.10062406 0.11777198 0.03648652 [103] 0.10563969 0.02546788 0.03382288 0.21047965 0.21285591 0.22626875 [109] 0.05942377 0.13307492 0.12111886 0.04804673 0.20934168 0.09780491 [115] 0.09565631 0.06610443 0.11835788 0.10179626 0.04208397 0.13316489 [121] 0.08880813 0.11819947 0.12590609 0.06382867 0.04873562 0.10096954 [127] 0.16895727 0.07071595 0.18176576 0.20368670 0.09668642 0.07655312 [133] 0.07285402 0.07591408 0.12014994 0.09658546 0.03408213 0.06557961 [139] 0.03435344 0.13741559 0.10427738 0.08179931 0.15238319 0.03225919 [145] 0.08846023 0.09654372 0.06517369 0.28141342 0.04392140 0.15149827 [151] 0.05417799 0.05188419 0.09752234 0.07855038 0.13698315 0.14809067 [157] 0.05359203 0.08025105 0.10941897 0.04681826 0.13319633 0.17033481 [163] 0.19015207 0.12633207 0.30376647 0.14051878 0.07729420 0.21937069 [169] 0.13915965 0.10309765 0.11141529 0.03159418 0.06460334 0.09468050 [175] 0.16077544 0.03304408 0.10694633 0.07742893 0.04473972 0.27469558 [181] 0.09041641 0.15658452 0.10116304 0.18469702 0.12803695 0.17072363 [187] 0.13595962 0.10146773 0.03904361 0.03007758 0.15658452 0.07039247 [193] 0.09417842 0.05646481 0.05803661 0.02422909 0.04738359 0.09155494 [199] 0.04840780 0.05274956 0.11819947 0.05210082 0.05404334 0.02413592 [205] 0.02878355 0.03142448 0.05062984 0.04840780 0.12497580 0.16046323 [211] 0.15009166 0.18326622 0.25738807 0.15772385 0.11591106 0.10525326 [217] 0.18527256 0.18794109 0.07304322 0.13907962 0.02480003 0.05553655 [223] 0.11662242 0.15864716 0.06143062 0.08307127 0.06898616 0.15864716 [229] 0.05704521 0.13545658 0.13698315 0.04341096 0.06704933 0.12045008 [235] 0.23225539 0.20559001 0.02413592 0.07885025 0.10525326 0.10755239 [241] 0.12372829 0.02029707 0.04575778 0.14922495 0.07898390 0.10563969 [247] 0.04681826 0.21571325 0.10580636 0.08286297 0.10309765 0.07860207 [253] 0.10988954 0.12730285 0.11327565 0.05750026 0.15965917 0.15655743 [259] 0.10654207 0.07749487 0.12907222 0.09986280 0.06540815 0.05686213 [265] 0.07573609 0.04980711 0.19687400 0.13466928 0.21654484 0.05558967 [271] 0.08896917 0.05844530 0.12049330 0.21836407 0.06235074 0.12497580 [277] 0.15123588 0.05499386 0.21449427 0.15277050 0.12141357 0.06712434 [283] 0.09545904 0.15509186 0.04571302 0.08168226 0.08913627 0.07258167 [289] 0.07422991 0.21576101 0.09629293 0.14533889 0.17982497 0.08372212 [295] 0.07998278 0.08197335 0.15718652 0.06586857 0.07040561 0.11141529 [301] 0.07880781 0.05417799 0.14865312 0.06731261 0.05164649 0.06696182 [307] 0.02863206 0.09668642 0.14191981 0.13741559 0.31586013 0.13523592 [313] 0.10507745 0.18469702 0.06658210 0.10869934 0.05160325 0.07475412 [319] 0.07044300 0.30783321 0.15285841 0.12684744 0.06243568 0.16734576 [325] 0.03239226 0.02346766 0.07079790 0.04601782 0.02035337 0.10179626 [331] 0.11776292 0.10146773 0.07479562 0.13994828 0.06299784 0.06194819 [337] 0.09576355 0.11667209 0.13126306 0.09127886 0.10988954 0.07429488 [343] 0.08189480 0.16036028 0.07385429 0.07344905 0.10571489 0.06624503 [349] 0.10624899 0.14096164 0.07172947 0.23855091 0.03422069 0.07588839 [355] 0.05915387 0.07933317 0.21654484 0.23855091 0.08307127 0.09607319 [361] 0.13741671 0.30376647 0.06951330 0.06586857 0.07759958 0.08638687 [367] 0.10599399 0.03775993 0.09733867 0.06455493 0.13487043 0.06107637 [373] 0.15268202 0.12824608 0.09518469 0.06282433 0.02035337 0.13741559 [379] 0.08943488 0.15477094 0.09876669 0.12128892 0.10066604 0.07838485 [385] 0.08846023 0.09698602 0.11188880 0.21937069 0.16434270 0.05139532 [391] 0.25084834 0.07422991 0.05706483 0.05568421 0.07474594 0.15277050 [397] 0.20559001 0.05953436 0.12633207 0.05556300 0.08844425 0.05855555 [403] 0.18176576 0.06556266 0.05623536 0.07705890 0.08528123 0.29497074 [409] 0.13168938 0.05579764 0.12455365 0.06149931 0.19063393 0.12158041 [415] 0.31494704 0.09906345 0.06235074 0.17689692 0.04601782 0.12549051 [421] 0.15123588 0.05953436 0.06677790 0.05924598 0.18385622 0.12622961 [427] 0.10341355 0.11591106 0.08307127 0.05683503 0.15853871 0.16844252 [433] 0.16017556 0.10728050 0.06020744 0.05499386 0.02413592 0.04656228 [439] 0.05963063 0.09357696 0.14298225 0.11347792 0.04575778 0.02406857 [445] 0.03925991 0.13316489 0.12329796 0.07923424 0.16617058 0.05055416 [451] 0.09876669 0.05942377 0.12151063 0.15268202 0.04753762 0.04597225 [457] 0.09468634 0.13915965 0.23769474 0.09361184 0.04009169 0.03353687 [463] 0.04707694 0.06020744 0.06973542 0.06540815 0.09441396 0.14121340 [469] 0.10923111 0.15539308 0.05458829 0.07932713 0.14583167 0.15011108 [475] 0.02177921 0.05620452 0.10599399 0.11458671 0.02868686 0.06731261 [481] 0.08559506 0.07765065 0.13545658 0.11030213 0.09161255 0.11752067 [487] 0.11964934 0.08638706 0.12105367 0.10226368 0.06520892 0.06539690 [493] 0.13316489 0.04041857 0.10591105 0.07737549 0.11844535 0.18430950 [499] 0.05937309 0.08358976 0.05584211 0.02413592 0.02948229 0.07178350 [505] 0.06739426 0.05942377 0.09360248 0.06266770 0.03734101 0.11088942 [511] 0.49089902 0.04450702 0.15009166 0.11781114 0.08810124 0.13854292 [517] 0.11440868 0.07494213 0.09082427 0.08409508 0.18794109 0.04509637 [523] 0.08714048 0.12290235 0.07178350 0.21576101 0.11347792 0.02270636 [529] 0.12303826 0.14105943 0.05294485 0.15015624 0.21576101 0.12732682 [535] 0.10347859 0.11306829 0.18696323 0.13907962 0.05646481 0.07281746 [541] 0.07278264 0.06871322 0.02029707 0.19455322 0.05024100 0.12876031 [547] 0.20559001 0.09176401 0.08235885 0.13709401 0.06891976 0.04022558 [553] 0.09113987 0.09082427 0.05942377 0.07053931 0.02346766 0.05446337 [559] 0.09376837 0.08229653 0.16525135 0.05024100 0.11877988 0.04463462 [565] 0.07741763 0.04422210 0.09631311 0.07121873 0.07759958 0.14530674 [571] 0.44803236 0.08444299 0.07525638 0.11403867 0.09161001 0.04404462 [577] 0.13595962 0.02791917 0.18565138 0.08946280 0.11273894 0.13850274 [583] 0.05885326 0.15853871 0.18890407 0.10624899 0.05062984 0.06726032 [589] 0.16145167 0.03133555 0.06143062 0.14377635 0.09795128 0.01071189 [595] 0.06094699 0.11088942 0.02480003 0.05885326 0.11336569 0.09730717 [601] 0.15277050 0.13487043 0.11025208 0.06578481 0.16077544 0.04699788 [607] 0.17828169 0.05294485 0.15740470 0.07923424 0.12639083 0.04404462 [613] 0.07932713 0.05288088 0.09607319 0.13929808 0.07712070 0.04774235 [619] 0.07361046 0.08412289 0.06588054 0.07525638 0.09306775 0.15027961 [625] 0.09672776 0.05538817 0.07865526 0.08466485 0.09241878 0.07729701 [631] 0.12593668 0.17140379 0.15764580 0.09298986 0.10909276 0.05458829 [637] 0.07432833 0.06704933 0.12733039 0.04575778 0.09629293 0.03074039 [643] 0.09097702 0.20722867 0.04725275 0.21047955 0.15509186 0.11206435 [649] 0.21116904 0.11460391 0.07758077 0.12290235 0.18255298 0.17082431 [655] 0.05829412 0.07855038 0.15547657 0.26252353 0.04235579 0.07927900 [661] 0.10045539 0.21449427 0.14416553 0.14891655 0.12482993 0.17828169 [667] 0.12594687 0.03064742 0.08991247 0.10692266 0.13833338 0.04122956 [673] 0.11249845 0.04417553 0.10571489 0.18999922 0.09196817 0.03816957 [679] 0.10104815 0.09441396 0.23363694 0.07944806 0.13238260 0.04361223 [685] 0.09183520 0.04341096 0.08280372 0.14105943 0.08507673 0.09654372 [691] 0.19610627 0.10116304 0.10863997 0.07832669 0.09189772 0.06972913 [697] 0.10341355 0.41027359 0.05924598 0.02346766 0.04862709 0.06986529 [703] 0.08608685 0.05325900 0.19610627 0.08763094 0.08329037 0.06361365 [709] 0.04767111 0.07071595 0.10991018 0.08193712 0.23363694 0.04445303 [715] 0.11245746 0.09183520 0.04873562 0.08793354 0.22825268 0.06073322 [721] 0.09176401 0.12008265 0.06020744 0.11844535 0.10563969 0.10234311 [727] 0.06520892 0.07281746 0.15123588 0.05556300 0.13777906 0.03353687 [733] 0.02458504 0.07479562 0.07587043 0.13307492 0.13915965 0.07927900 [739] 0.05802982 0.07899391 0.20255292 0.06266770 0.05984799 0.07860207 [745] 0.05323757 0.09144881 0.08896917 0.13479155 0.08946280 0.16786715 [751] 0.04123342 0.08564806 0.23511008 0.04341096 0.05750026 0.14795599 [757] 0.06314469 0.15831410 0.07559751 0.12882916 0.11153990 0.05987319 [763] 0.13994828 0.16664323 0.03940269 0.23769474 0.16837187 0.08975351 [769] 0.14795599 0.13994828 0.07981866 0.10341355 0.07840968 0.12092804 [775] 0.04090392 0.05789221 0.12594687 0.14881680 0.07572783 0.09082427 [781] 0.18581487 0.12684744 0.07834012 0.15067216 0.03142448 0.04418763 [787] 0.10178808 0.03865712 0.05568421 0.10591105 0.06871322 0.16362535 [793] 0.05790428 0.12785082 0.10755239 0.12730285 0.07053931 0.10909276 [799] 0.14234255 0.05683503 0.04980711 0.10347859 0.08768850 0.08400254 [805] 0.03064742 0.08907615 0.18469702 0.18326622 0.07759958 0.09065016 [811] 0.12158041 0.07765348 0.10228816 0.04122956 0.09357696 0.11774699 [817] 0.09884417 0.07646570 0.05789020 0.08793354 0.09576355 0.06731261 [823] 0.12593668 0.07876001 0.06266770 0.11802980 0.04905024 0.08412289 [829] 0.11835788 0.14001611 0.10347859 0.07224078 0.12882916 0.05984799 [835] 0.08735718 0.13466928 0.03516026 0.07572783 0.09127886 0.07765348 [841] 0.16036028 0.04840780 0.02585494 0.12141357 0.06557961 0.02868686 [847] 0.09545904 0.17291042 0.14374543 0.08714048 0.15210082 0.06204146 [853] 0.03214978 0.05706483 0.07712070 0.20854512 0.16525135 0.16837187 [859] 0.16739062 0.10923111 0.05704521 0.03804701 0.06712434 0.11141529 [865] 0.10348343 0.05915387 0.11554524 0.05624803 0.19624718 0.08002494 [871] 0.17586510 0.17140379 0.07714308 0.04578117 0.17695605 0.22004235 [877] 0.10507745 0.11776292 0.02237537 0.08380918 0.11591106 0.05315705 [883] 0.07076261 0.06098961 0.04450702 0.15547657 0.05803798 0.07932713 [889] 0.10966650 0.07834012 0.03239226 0.12949018 0.08559506 0.04767111 [895] 0.39346964 0.14001611 0.30092792 0.16001529 0.03925991 0.09237448 [901] 0.15740470 0.02480003 0.21836407 0.07482267 0.03440945 0.15658452 [907] 0.12984470 0.06103167 0.24337076 0.08317427 0.08638687 0.11306829 [913] 0.11019091 0.06972913 0.07015611 0.39346964 0.08377308 0.17627260 [919] 0.10923111 0.04252059 0.16739062 0.20934168 0.08433168 0.06812109 [925] 0.10236681 0.18106708 0.07860207 0.21812565 0.09629293 0.13850274 [931] 0.18326622 0.04469137 0.11974209 0.08559506 0.09669284 0.08179931 [937] 0.08916867 0.07044300 0.21370951 0.03440945 0.07262374 0.03914118 [943] 0.05869129 0.04473972 0.07876001 0.08046860 0.17672553 0.06539690 [949] 0.17809447 0.08577074 0.18106708 0.10787718 0.09460796 0.03074039 [955] 0.18469702 0.09045546 0.12547784 0.05553655 0.07338860 0.23864372 [961] 0.09753612 0.09906345 0.12594687 0.18565138 0.02860923 0.04211603 [967] 0.15539308 0.07463707 0.15123588 0.26112435 0.16046323 0.17581946 [973] 0.09629293 0.12502798 0.11327565 0.09565631 0.14374543 0.02967030 [979] 0.04469137 0.15190665 0.19860230 0.05706483 0.10591105 0.13523885 [985] 0.06545116 0.07874442 0.16145167 0.12030718 0.09197837 0.06755633 [991] 0.14891655 0.05046991 0.05789221 0.07816301 0.13181789 0.04633819 [997] 0.11306829 0.08380918 0.05908805 0.07654951 0.09065016 0.13319633 [1003] 0.04090392 0.10427738 0.05803661 0.10991018 0.07654951 0.13904688 [1009] 0.15636949 0.04738359 0.07511075 0.26156918 0.13847522 0.21988016 [1015] 0.12926921 0.04812723 0.04978817 0.04548222 0.08564806 0.19559255 [1021] 0.07865526 0.04009169 0.06118780 0.10988954 0.07591408 0.08286297 [1027] 0.27406046 0.07813619 0.13709120 0.10591105 0.05450325 0.07429488 [1033] 0.11069369 0.03194120 0.06390939 0.19624718 0.11105994 0.11591106 [1039] 0.09189772 0.15660634 0.12549051 0.18527256 0.06303880 0.17072363 [1045] 0.08826893 0.04473972 0.06012776 0.07875266 0.04016170 0.14096164 [1051] 0.08687466 0.10966650 0.12594687 0.19354504 0.09357696 0.16541363 [1057] 0.06883807 0.08977614 0.10755239 0.08934028 0.18176576 0.07363497 [1063] 0.05189083 0.06849126 0.11458671 0.08714048 0.07015611 0.12210389 [1069] 0.16837187 0.06557961 0.08958308 0.17679070 0.03712945 0.16555231 [1075] 0.05417799 0.33177673 0.08763094 0.08897925 0.17072363 0.12785082 [1081] 0.33836080 0.15636949 0.11877988 0.08799498 0.19606961 0.12628534 [1087] 0.03382288 0.02546788 0.08168226 0.11844535 0.10116304 0.06675867 [1093] 0.04597225 0.19606961 0.05380227 0.11640753 0.10504201 0.16555231 [1099] 0.19515666 0.04443051 0.05810964 0.04041857 0.04162643 0.11188880 [1105] 0.02035337 0.07898390 0.23363694 0.15718788 0.08638706 0.02480003 [1111] 0.05568421 0.04597225 0.14121340 0.09468050 0.11053543 0.10953160 [1117] 0.12644428 0.06712434 0.05123222 0.15052632 0.07880781 0.12884679 [1123] 0.08799498 0.11964934 0.06900802 0.07818921 0.04211603 0.05568421 [1129] 0.17982497 0.03134819 0.13082140 0.06421695 0.10988954 0.07262374 [1135] 0.05821758 0.11539683 0.13238260 0.09668642 0.04334211 0.02298995 [1141] 0.12824608 0.10402626 0.07053931 0.13912130 0.10692266 0.11245746 [1147] 0.06096304 0.09623007 0.15294946 0.07855038 0.24149959 0.09237448 [1153] 0.02876609 0.03435344 0.09518469 0.25018175 0.13319633 0.25110678 [1159] 0.09698602 0.05438601 0.13110690 0.05797913 0.15515778 0.15285841 [1165] 0.26156918 0.08179931 0.11878402 0.10184765 0.13907962 0.15965917 [1171] 0.12903450 0.04597225 0.05696534 0.05619540 0.09357037 0.05802982 [1177] 0.09237977 0.17036438 0.11273894 0.07527117 0.20255292 0.07172160 [1183] 0.05329707 0.04041857 0.22004235 0.07606801 0.43936279 0.05696534 [1189] 0.05123222 0.09515797 0.03734101 0.09196817 0.08444299 0.06096304 [1195] 0.03512500 0.14729850 0.07321257 0.09237448 0.03635040 0.28748336 [1201] 0.21285591 0.08973695 0.05381279 0.04738359 0.08517325 0.10309765 [1207] 0.09635941 0.06332807 0.08937260 0.03816957 0.04211603 0.05953436 [1213] 0.13238260 0.08372212 0.05496640 0.10347859 0.09515797 0.11878402 [1219] 0.16712555 0.06103167 0.06875493 0.03734101 0.10504201 0.08667288 [1225] 0.05188419 0.16145167 0.06363978 0.04681826 0.22626875 0.19221686 [1231] 0.12884679 0.07285402 0.02650754 0.06578481 0.07511075 0.06539690 [1237] 0.06020744 0.10284033 0.06557961 0.22535562 0.11198121 0.05417799 [1243] 0.06460334 0.10654207 0.06556266 0.08229653 0.18234587 0.17280704 [1249] 0.10450151 0.06595300 0.05782075 0.06540815 0.09447458 0.06152664 [1255] 0.04298015 0.04753762 0.05123222 0.06972913 0.15772385 0.17072363 [1261] 0.11458671 0.12372829 0.11336569 0.17072363 0.05903124 0.07944806 [1267] 0.14891655 0.10876130 0.06232278 0.04473972 0.04123342 0.10226368 [1273] 0.05315705 0.06210313 0.12105367 0.10599399 0.09298986 0.08934028 [1279] 0.11877988 0.11048328 0.09255798 0.03942690 0.08673013 0.05706483 [1285] 0.08907615 0.04367266 0.08824778 0.10941897 0.06314469 0.05139532 [1291] 0.11273894 0.06588054 0.08197335 0.10863997 0.18223407 0.09752234 [1297] 0.04031695 0.09986280 0.03589551 0.11458671 0.15294946 0.05274956 [1303] 0.20255292 0.12455365 0.09479633 0.10122389 0.04299998 0.11830660 [1309] 0.14530674 0.11206132 0.02480003 0.11554524 0.04681826 0.16046323 [1315] 0.08673013 0.03589551 0.12644428 0.09306775 0.31586013 0.08601192 [1321] 0.17355770 0.11877988 0.11860177 0.05984799 0.08788305 0.27406046 [1327] 0.15027961 0.02035337 0.18430950 0.09122318 0.07475412 0.13583966 [1333] 0.12141357 0.05762591 0.09607319 0.05803798 0.05318136 0.13624265 [1339] 0.05338334 0.21370951 0.11336569 0.08255590 0.08745484 0.11153990 [1345] 0.05323757 0.07538912 0.05987319 0.10654207 0.14045792 0.11542011 [1351] 0.09774319 0.07875266 0.05438447 0.15294946 0.05493713 0.19259562 [1357] 0.15009166 0.23855091 0.09161001 0.07363497 0.15294946 0.03382288 [1363] 0.07198320 0.12146508 0.06292235 0.06675867 0.03925991 0.09082427 [1369] 0.11961122 0.12903450 0.02029707 0.13319633 0.04964369 0.12628534 [1375] 0.09208343 0.24149959 0.10146773 0.11392187 0.06360322 0.06332807 [1381] 0.05893901 0.02851325 0.04862709 0.04738359 0.10373571 0.06801381 [1387] 0.06996488 0.03382288 0.02270636 0.04626376 0.14481151 0.06210313 [1393] 0.10263175 0.04460879

$rocPoints falsePositiveRate trueNegativeScore matthewsCorrelationCoefficient 1 0.0000000000 1392 0.00000000 2 0.0000000000 1392 0.09151969 3 0.0000000000 1392 0.25999821 4 0.0000000000 1392 0.30545398 5 0.0000000000 1392 0.31241798 6 0.0000000000 1392 0.31923865 7 0.0000000000 1392 0.32592500 8 0.0000000000 1392 0.33248516 9 0.0007183908 1391 0.32449791 10 0.0028735632 1388 0.30969727 11 0.0028735632 1388 0.31646591 12 0.0057471264 1384 0.30585631 13 0.0079022989 1381 0.31642854 14 0.0093390805 1379 0.31332215 15 0.0165229885 1369 0.28563772 16 0.0186781609 1366 0.28758686 17 0.0280172414 1353 0.26677433 18 0.0373563218 1340 0.25224019 19 0.0452586207 1329 0.24553375 20 0.0466954023 1327 0.24162876 21 0.0567528736 1313 0.22682862 22 0.0660919540 1300 0.22107718 23 0.0754310345 1287 0.22141642 24 0.0854885057 1273 0.21194305 25 0.0962643678 1258 0.19801804 26 0.1048850575 1246 0.20198435 27 0.1142241379 1233 0.20074277 28 0.1235632184 1220 0.19990462 29 0.1293103448 1212 0.20019466 30 0.1336206897 1206 0.19843961 31 0.1429597701 1193 0.19827150 32 0.1508620690 1182 0.20798616 33 0.1609195402 1168 0.20336525 34 0.1688218391 1157 0.21302188 35 0.1788793103 1143 0.20888524 36 0.1889367816 1129 0.20875474 37 0.1989942529 1115 0.20519019 38 0.2076149425 1103 0.21056210 39 0.2155172414 1092 0.22021706 40 0.2262931034 1077 0.21287937 41 0.2370689655 1062 0.20586003 42 0.2478448276 1047 0.20258525 43 0.2571839080 1034 0.20425913 44 0.2665229885 1021 0.20601772 45 0.2765804598 1007 0.20385094 46 0.2859195402 994 0.20580461 47 0.2974137931 978 0.19600477 48 0.3067528736 965 0.19819336 49 0.3168103448 951 0.19986295 50 0.3275862069 936 0.19446727 51 0.3347701149 926 0.18876785 52 0.3390804598 920 0.18538371 53 0.3491379310 906 0.18412504 54 0.3591954023 892 0.18296062 55 0.3685344828 879 0.18567420 56 0.3807471264 862 0.17659597 57 0.3908045977 848 0.17571133 58 0.3987068966 837 0.18615352 59 0.4087643678 823 0.18539667 60 0.4181034483 810 0.18845156 61 0.4274425287 797 0.19157176 62 0.4375000000 783 0.19426419 63 0.4482758621 768 0.19009254 64 0.4583333333 754 0.18971912 65 0.4691091954 739 0.18570468 66 0.4777298851 727 0.19291864 67 0.4877873563 713 0.19277558 68 0.4985632184 698 0.19224250 69 0.5100574713 682 0.18479376 70 0.5215517241 666 0.17739286 71 0.5316091954 652 0.17753010 72 0.5423850575 637 0.17398167 73 0.5531609195 622 0.17048433 74 0.5653735632 605 0.16279635 75 0.5754310345 591 0.16318110 76 0.5854885057 577 0.16364122 77 0.5962643678 562 0.16033808 78 0.6056034483 549 0.16479885 79 0.6170977011 533 0.15772926 80 0.6293103448 516 0.15020105 81 0.6408045977 500 0.14309019 82 0.6508620690 486 0.14389236 83 0.6616379310 471 0.14077617 84 0.6731321839 455 0.13363858 85 0.6839080460 440 0.13052497 86 0.6939655172 426 0.13155047 87 0.7047413793 411 0.13223126 88 0.7155172414 396 0.12924819 89 0.7162356322 395 0.12879802 90 0.7270114943 380 0.12200059 91 0.7363505747 367 0.12765544 92 0.7464080460 353 0.12916775 93 0.7557471264 340 0.13529738 94 0.7650862069 327 0.14585129 95 0.7765804598 311 0.13885082 96 0.7880747126 295 0.13172254 97 0.7988505747 280 0.12926265 98 0.8103448276 264 0.12191850 99 0.8204022989 250 0.12447255 100 0.8326149425 233 0.11649865 101 0.8433908046 218 0.11408507 102 0.8527298851 205 0.12273163 103 0.8627873563 191 0.12650836 104 0.8735632184 176 0.12494356 105 0.8850574713 160 0.11742340 106 0.8972701149 143 0.10908200 107 0.9087643678 127 0.10082030 108 0.9195402299 112 0.09930645 109 0.9310344828 96 0.09041397 110 0.9425287356 80 0.08078676 111 0.9533045977 65 0.07954530 112 0.9655172414 48 0.06798088 113 0.9770114943 32 0.05522224 114 0.9885057471 16 0.03885031 115 0.9971264368 4 0.01935199 116 1.0000000000 0 0.00000000 truePositiveScore trueNegativeRate falseNegativeScore threshold 1 0 1.000000000 208 1.00000000 2 2 1.000000000 206 0.97164696 3 16 1.000000000 192 0.93167734 4 22 1.000000000 186 0.88963861 5 23 1.000000000 185 0.81805729 6 24 1.000000000 184 0.80178826 7 25 1.000000000 183 0.58505843 8 26 1.000000000 182 0.52941859 9 26 0.999281609 182 0.49089902 10 27 0.997126437 181 0.43936279 11 28 0.997126437 180 0.42731926 12 30 0.994252874 178 0.35724214 13 34 0.992097701 174 0.31651636 14 35 0.990660920 173 0.30856903 15 37 0.983477011 171 0.27406046 16 39 0.981321839 169 0.26156918 17 42 0.971982759 166 0.24545118 18 45 0.962643678 163 0.23125668 19 48 0.954741379 160 0.22004235 20 48 0.953304598 160 0.21950930 21 50 0.943247126 158 0.21047955 22 53 0.933908046 155 0.20227999 23 57 0.924568966 151 0.19504046 24 59 0.914511494 149 0.18771500 25 60 0.903735632 148 0.18234587 26 64 0.895114943 144 0.17700579 27 67 0.885775862 141 0.17483333 28 70 0.876436782 138 0.17140379 29 72 0.870689655 136 0.16903128 30 73 0.866379310 135 0.16844252 31 76 0.857040230 132 0.16525135 32 81 0.849137931 127 0.16113377 33 83 0.839080460 125 0.15801978 34 88 0.831178161 120 0.15603598 35 90 0.821120690 118 0.15238319 36 93 0.811063218 115 0.15011108 37 95 0.801005747 113 0.14763167 38 99 0.792385057 109 0.14402286 39 104 0.784482759 104 0.14124670 40 105 0.773706897 103 0.13912130 41 106 0.762931034 102 0.13742623 42 108 0.752155172 100 0.13505667 43 111 0.742816092 97 0.13199302 44 114 0.733477011 94 0.12984470 45 116 0.723419540 92 0.12870438 46 119 0.714080460 89 0.12677560 47 119 0.702586207 89 0.12502798 48 122 0.693247126 86 0.12271065 49 125 0.683189655 83 0.12065292 50 126 0.672413793 82 0.11930273 51 126 0.665229885 82 0.11835788 52 126 0.660919540 82 0.11776391 53 128 0.650862069 80 0.11554524 54 130 0.640804598 78 0.11392187 55 133 0.631465517 75 0.11206132 56 133 0.619252874 75 0.11015398 57 135 0.609195402 73 0.10876130 58 140 0.601293103 68 0.10692266 59 142 0.591235632 66 0.10571489 60 145 0.581896552 63 0.10369122 61 148 0.572557471 60 0.10228816 62 151 0.562500000 57 0.10079098 63 152 0.551724138 56 0.09890623 64 154 0.541666667 54 0.09752234 65 155 0.530890805 53 0.09609709 66 159 0.522270115 49 0.09468634 67 161 0.512212644 47 0.09357037 68 163 0.501436782 45 0.09197837 69 163 0.489942529 45 0.09113987 70 163 0.478448276 45 0.08973695 71 165 0.468390805 43 0.08892835 72 166 0.457614943 42 0.08793354 73 167 0.446839080 41 0.08673013 74 167 0.434626437 41 0.08506275 75 169 0.424568966 39 0.08377308 76 171 0.414511494 37 0.08256009 77 172 0.403735632 36 0.08088664 78 175 0.394396552 33 0.07939348 79 175 0.382902299 33 0.07872838 80 175 0.370689655 33 0.07765348 81 175 0.359195402 33 0.07705890 82 177 0.349137931 31 0.07584304 83 178 0.338362069 30 0.07469378 84 178 0.326867816 30 0.07329380 85 179 0.316091954 29 0.07172947 86 181 0.306034483 27 0.07040561 87 183 0.295258621 25 0.06896378 88 184 0.284482759 24 0.06739426 89 184 0.283764368 24 0.06731261 90 184 0.272988506 24 0.06644182 91 187 0.263649425 21 0.06520892 92 189 0.253591954 19 0.06421695 93 192 0.244252874 16 0.06292235 94 196 0.234913793 12 0.06202997 95 196 0.223419540 12 0.06073322 96 196 0.211925287 12 0.05915387 97 197 0.201149425 11 0.05810964 98 197 0.189655172 11 0.05696534 99 199 0.179597701 9 0.05568421 100 199 0.167385057 9 0.05445611 101 200 0.156609195 8 0.05338334 102 203 0.147270115 5 0.05189083 103 205 0.137212644 3 0.05062680 104 206 0.126436782 2 0.04841284 105 206 0.114942529 2 0.04750371 106 206 0.102729885 2 0.04571302 107 206 0.091235632 2 0.04418763 108 207 0.080459770 1 0.04298015 109 207 0.068965517 1 0.04041857 110 207 0.057471264 1 0.03816957 111 208 0.046695402 0 0.03550797 112 208 0.034482759 0 0.03214978 113 208 0.022988506 0 0.02863206 114 208 0.011494253 0 0.02422909 115 208 0.002873563 0 0.02029707 116 208 0.000000000 0 0.01071189 negativePredictiveValue positivePredictiveValue falsePositiveScore 1 0.8700000 0.0000000 0 2 0.8710889 1.0000000 0 3 0.8787879 1.0000000 0 4 0.8821293 1.0000000 0 5 0.8826886 1.0000000 0 6 0.8832487 1.0000000 0 7 0.8838095 1.0000000 0 8 0.8843710 1.0000000 0 9 0.8842975 0.9629630 1 10 0.8846399 0.8709677 4 11 0.8852041 0.8750000 4 12 0.8860435 0.7894737 8 13 0.8881029 0.7555556 11 14 0.8885309 0.7291667 13 15 0.8889610 0.6166667 23 16 0.8899023 0.6000000 26 17 0.8907176 0.5185185 39 18 0.8915502 0.4639175 52 19 0.8925453 0.4324324 63 20 0.8924008 0.4247788 65 21 0.8925901 0.3875969 79 22 0.8934708 0.3655172 92 23 0.8949930 0.3518519 105 24 0.8952180 0.3314607 119 25 0.8947368 0.3092784 134 26 0.8964029 0.3047619 146 27 0.8973799 0.2964602 159 28 0.8983800 0.2892562 172 29 0.8991098 0.2857143 180 30 0.8993289 0.2818533 186 31 0.9003774 0.2763636 199 32 0.9029794 0.2783505 210 33 0.9033256 0.2703583 224 34 0.9060298 0.2724458 235 35 0.9064235 0.2654867 249 36 0.9075563 0.2612360 263 37 0.9079805 0.2553763 277 38 0.9100660 0.2551546 289 39 0.9130435 0.2574257 300 40 0.9127119 0.2500000 315 41 0.9123711 0.2431193 330 42 0.9128160 0.2384106 345 43 0.9142352 0.2366738 358 44 0.9156951 0.2350515 371 45 0.9162875 0.2315369 385 46 0.9178209 0.2301741 398 47 0.9165886 0.2232645 414 48 0.9181732 0.2222222 427 49 0.9197292 0.2208481 441 50 0.9194499 0.2164948 456 51 0.9186508 0.2128378 466 52 0.9181637 0.2107023 472 53 0.9188641 0.2084691 486 54 0.9195876 0.2063492 500 55 0.9213836 0.2058824 513 56 0.9199573 0.2006033 530 57 0.9207383 0.1988218 544 58 0.9248619 0.2014388 555 59 0.9257593 0.1997187 569 60 0.9278351 0.1994498 582 61 0.9299883 0.1991925 595 62 0.9321429 0.1986842 609 63 0.9320388 0.1958763 624 64 0.9331683 0.1944444 638 65 0.9330808 0.1918317 653 66 0.9368557 0.1929612 665 67 0.9381579 0.1916667 679 68 0.9394347 0.1901984 694 69 0.9381018 0.1867125 710 70 0.9367089 0.1833521 726 71 0.9381295 0.1823204 740 72 0.9381443 0.1802389 755 73 0.9381599 0.1782284 770 74 0.9365325 0.1750524 787 75 0.9380952 0.1742268 801 76 0.9397394 0.1734280 815 77 0.9397993 0.1716567 830 78 0.9432990 0.1719057 843 79 0.9416961 0.1692456 859 80 0.9398907 0.1665081 876 81 0.9380863 0.1640112 892 82 0.9400387 0.1634349 906 83 0.9401198 0.1619654 921 84 0.9381443 0.1596413 937 85 0.9381663 0.1582670 952 86 0.9403974 0.1578030 966 87 0.9426606 0.1572165 981 88 0.9428571 0.1559322 996 89 0.9427208 0.1558002 997 90 0.9405941 0.1538462 1012 91 0.9458763 0.1542904 1025 92 0.9489247 0.1539088 1039 93 0.9550562 0.1543408 1052 94 0.9646018 0.1554322 1065 95 0.9628483 0.1534847 1081 96 0.9609121 0.1515855 1097 97 0.9621993 0.1504966 1112 98 0.9600000 0.1486792 1128 99 0.9652510 0.1483967 1142 100 0.9628099 0.1465390 1159 101 0.9646018 0.1455604 1174 102 0.9761905 0.1460432 1187 103 0.9845361 0.1458037 1201 104 0.9887640 0.1448664 1216 105 0.9876543 0.1432545 1232 106 0.9862069 0.1415808 1249 107 0.9844961 0.1400408 1265 108 0.9911504 0.1392065 1280 109 0.9896907 0.1377246 1296 110 0.9876543 0.1362739 1312 111 1.0000000 0.1355049 1327 112 1.0000000 0.1340206 1344 113 1.0000000 0.1326531 1360 114 1.0000000 0.1313131 1376 115 1.0000000 0.1303258 1388 116 0.0000000 0.1300000 1392 truePositiveRate f1Score accuracy 1 0.000000000 0.00000000 0.870000 2 0.009615385 0.01904762 0.871250 3 0.076923077 0.14285714 0.880000 4 0.105769231 0.19130435 0.883750 5 0.110576923 0.19913420 0.884375 6 0.115384615 0.20689655 0.885000 7 0.120192308 0.21459227 0.885625 8 0.125000000 0.22222222 0.886250 9 0.125000000 0.22127660 0.885625 10 0.129807692 0.22594142 0.884375 11 0.134615385 0.23333333 0.885000 12 0.144230769 0.24390244 0.883750 13 0.163461538 0.26877470 0.884375 14 0.168269231 0.27343750 0.883750 15 0.177884615 0.27611940 0.878750 16 0.187500000 0.28571429 0.878125 17 0.201923077 0.29065744 0.871875 18 0.216346154 0.29508197 0.865625 19 0.230769231 0.30094044 0.860625 20 0.230769231 0.29906542 0.859375 21 0.240384615 0.29673591 0.851875 22 0.254807692 0.30028329 0.845625 23 0.274038462 0.30810811 0.840000 24 0.283653846 0.30569948 0.832500 25 0.288461538 0.29850746 0.823750 26 0.307692308 0.30622010 0.818750 27 0.322115385 0.30875576 0.812500 28 0.336538462 0.31111111 0.806250 29 0.346153846 0.31304348 0.802500 30 0.350961538 0.31263383 0.799375 31 0.365384615 0.31469979 0.793125 32 0.389423077 0.32464930 0.789375 33 0.399038462 0.32233010 0.781875 34 0.423076923 0.33145009 0.778125 35 0.432692308 0.32906764 0.770625 36 0.447115385 0.32978723 0.763750 37 0.456730769 0.32758621 0.756250 38 0.475961538 0.33221477 0.751250 39 0.500000000 0.33986928 0.747500 40 0.504807692 0.33439490 0.738750 41 0.509615385 0.32919255 0.730000 42 0.519230769 0.32677761 0.721875 43 0.533653846 0.32791728 0.715625 44 0.548076923 0.32900433 0.709375 45 0.557692308 0.32722144 0.701875 46 0.572115385 0.32827586 0.695625 47 0.572115385 0.32118758 0.685625 48 0.586538462 0.32232497 0.679375 49 0.600961538 0.32299742 0.672500 50 0.605769231 0.31898734 0.663750 51 0.605769231 0.31500000 0.657500 52 0.605769231 0.31265509 0.653750 53 0.615384615 0.31143552 0.646250 54 0.625000000 0.31026253 0.638750 55 0.639423077 0.31147541 0.632500 56 0.639423077 0.30539610 0.621875 57 0.649038462 0.30439684 0.614375 58 0.673076923 0.31007752 0.610625 59 0.682692308 0.30903156 0.603125 60 0.697115385 0.31016043 0.596875 61 0.711538462 0.31125131 0.590625 62 0.725961538 0.31198347 0.583750 63 0.730769231 0.30894309 0.575000 64 0.740384615 0.30800000 0.567500 65 0.745192308 0.30511811 0.558750 66 0.764423077 0.30813953 0.553750 67 0.774038462 0.30725191 0.546250 68 0.783653846 0.30610329 0.538125 69 0.783653846 0.30157262 0.528125 70 0.783653846 0.29717411 0.518125 71 0.793269231 0.29649596 0.510625 72 0.798076923 0.29406554 0.501875 73 0.802884615 0.29170306 0.493125 74 0.802884615 0.28743546 0.482500 75 0.812500000 0.28692699 0.475000 76 0.822115385 0.28643216 0.467500 77 0.826923077 0.28429752 0.458750 78 0.841346154 0.28548124 0.452500 79 0.841346154 0.28180354 0.442500 80 0.841346154 0.27799841 0.431875 81 0.841346154 0.27450980 0.421875 82 0.850961538 0.27420604 0.414375 83 0.855769231 0.27237950 0.405625 84 0.855769231 0.26908541 0.395625 85 0.860576923 0.26736370 0.386875 86 0.870192308 0.26715867 0.379375 87 0.879807692 0.26676385 0.371250 88 0.884615385 0.26512968 0.362500 89 0.884615385 0.26493880 0.361875 90 0.884615385 0.26210826 0.352500 91 0.899038462 0.26338028 0.346250 92 0.908653846 0.26323120 0.338750 93 0.923076923 0.26446281 0.332500 94 0.942307692 0.26684820 0.326875 95 0.942307692 0.26397306 0.316875 96 0.942307692 0.26115923 0.306875 97 0.947115385 0.25972314 0.298125 98 0.947115385 0.25701239 0.288125 99 0.956730769 0.25693996 0.280625 100 0.956730769 0.25415070 0.270000 101 0.961538462 0.25284450 0.261250 102 0.975961538 0.25406758 0.255000 103 0.985576923 0.25402726 0.247500 104 0.990384615 0.25276074 0.238750 105 0.990384615 0.25030377 0.228750 106 0.990384615 0.24774504 0.218125 107 0.990384615 0.24538416 0.208125 108 0.995192308 0.24424779 0.199375 109 0.995192308 0.24196376 0.189375 110 0.995192308 0.23972206 0.179375 111 1.000000000 0.23866896 0.170625 112 1.000000000 0.23636364 0.160000 113 1.000000000 0.23423423 0.150000 114 1.000000000 0.23214286 0.140000 115 1.000000000 0.23059867 0.132500 116 1.000000000 0.23008850 0.130000

$positiveClassPredictions [1] 0.16122640 0.58505843 0.06515376 0.45750322 0.95179779 0.94652867 [7] 0.20361004 0.15782583 0.12943243 0.09828108 0.09787355 0.20403487 [13] 0.31651636 0.07469378 0.16770250 0.15926946 0.81805729 0.05105972 [19] 0.06264669 0.31651636 0.35724214 0.52941859 0.09219515 0.26074285 [25] 0.06379918 0.31651636 0.06528269 0.16434127 0.15242557 0.16122640 [31] 0.11365692 0.21309482 0.16903128 0.12096168 0.16457471 0.15137231 [37] 0.12722454 0.11962273 0.09723168 0.94321719 0.22623163 0.05828056 [43] 0.17501156 0.14313264 0.06264669 0.94652867 0.10983634 0.12096168 [49] 0.20227999 0.10081245 0.14737117 0.95515922 0.13658701 0.30856903 [55] 0.13658701 0.26074285 0.22416046 0.05105972 0.26762148 0.34941522 [61] 0.14124670 0.34583277 0.04993154 0.07211376 0.05678107 0.05678107 [67] 0.24981795 0.10079098 0.06598228 0.12065292 0.10678992 0.15242557 [73] 0.91669905 0.91317422 0.10460039 0.88963861 0.97164696 0.13658701 [79] 0.16852909 0.91317422 0.14211201 0.12266762 0.11962273 0.13658701 [85] 0.14124670 0.14211201 0.33612284 0.08948366 0.16122640 0.16457471 [91] 0.10733591 0.06528269 0.06515376 0.11365692 0.14737117 0.11528212 [97] 0.13032088 0.13965648 0.24364131 0.06598228 0.94514432 0.10678992 [103] 0.08948366 0.94321719 0.95179779 0.06528269 0.10008724 0.15615265 [109] 0.10308902 0.12065292 0.91317422 0.10501180 0.17220516 0.13084765 [115] 0.10460039 0.09522534 0.10501180 0.09491720 0.10501180 0.13394290 [121] 0.17344203 0.15010813 0.08259411 0.19953162 0.31651636 0.97129902 [127] 0.15784922 0.07953740 0.12953964 0.30856903 0.34941522 0.09787355 [133] 0.08308191 0.19847179 0.12722454 0.52941859 0.13394290 0.22117127 [139] 0.95521040 0.21309482 0.10008724 0.45750322 0.22623163 0.09307678 [145] 0.13084765 0.09828108 0.10079098 0.09491720 0.39965357 0.11376283 [151] 0.15137231 0.18056747 0.04993154 0.20403487 0.80178826 0.12425629 [157] 0.10110228 0.15782583 0.10815967 0.06379918 0.27896176 0.09476555 [163] 0.13032088 0.05260993 0.26074285 0.09723168 0.26201676 0.80178826 [169] 0.14372075 0.19953162 0.09307678 0.06206788 0.18056747 0.06623039 [175] 0.10815967 0.12425629 0.06232102 0.52941859 0.20227999 0.13494811 [181] 0.94883438 0.12311604 0.06785913 0.09563065 0.15615265 0.17501156 [187] 0.12425629 0.15784922 0.06232102 0.10343946 0.14372075 0.15925625 [193] 0.08479112 0.93167734 0.20403487 0.06379918 0.10008724 0.05105972 [199] 0.11376283 0.58505843 0.09307678 0.95179779 0.12711593 0.19953162 [205] 0.93167734 0.10008724

You can then plot the results:

dr_roc_green = "#03c75f"
ValidationRocCurve <- GetRocCurve(bestModel)
ValidationRocPoints <- ValidationRocCurve[["rocPoints"]]
saveRDS(ValidationRocPoints, "ValidationRocPoints.rds")
par(bg = dr_dark_blue, xaxs = "i", yaxs = "i")
plot(ValidationRocPoints$falsePositiveRate, ValidationRocPoints$truePositiveRate, main = "ROC Curve",
     xlab="False Positive Rate (Fallout)", ylab="True Positive Rate (Sensitivity)", col = dr_roc_green,
     ylim=c(0,1), xlim=c(0,1), pch=20, type='b')

All the available ROC curve data can be retreived using GetAllRocCurves. Here again is an example to retreive data for all the available partitions, followed by plotting the cross vallidation partition:

AllRocCurve <- GetAllRocCurves(bestModel)
CrossValidationRocPoints <- AllRocCurve[['crossValidation']][['rocPoints']]
saveRDS(CrossValidationRocPoints, 'CrossValidationRocPoints.rds')
par(bg = dr_dark_blue, xaxs = "i", yaxs = "i")
plot(CrossValidationRocPoints$falsePositiveRate, CrossValidationRocPoints$truePositiveRate, main="ROC Curve",
     xlab = "False Positive Rate (Fallout)", ylab = "True Positive Rate (Sensitivity)", col = dr_roc_green,
     ylim = c(0, 1), xlim = c(0, 1), pch = 20, type = "b")

You can also plot the ROC curve using ggplot2:

ggplot(ValidationRocPoints, aes(x = falsePositiveRate, y = truePositiveRate)) + geom_line()

Threshold operations

You can get the recommended threshold value with maximal F1 score. That is the same threshold that is preselected in DataRobot when you open the “ROC curve” tab.

threshold <- ValidationRocPoints$threshold[which.max(ValidationRocPoints$f1Score)]

You can also estimate metrics for different threshold values. This will produce the same results as updating the threshold on the DataRobot “ROC curve” tab.

ValidationRocPoints[ValidationRocPoints$threshold == tail(Filter(function(x) x > threshold, ValidationRocPoints$threshold) , 1),]

##    falsePositiveRate trueNegativeScore matthewsCorrelationCoefficient
## 34         0.1609195              1168                      0.2033652
##    truePositiveScore trueNegativeRate falseNegativeScore threshold
## 34                83        0.8390805                125 0.1578901
##    negativePredictiveValue positivePredictiveValue falsePositiveScore
## 34               0.9033256               0.2703583                224
##    truePositiveRate   f1Score accuracy
## 34        0.3990385 0.3223301 0.781875

Word Cloud

The word cloud is a type of insight available for some text-processing models for datasets containing text columns. You can get information about how the appearance of each ngram (word or sequence of words) in the text field affects the predicted target value.

This example will show you how to obtain word cloud data and visualize it, similar to how DataRobot visualizes the word cloud in the “Model Insights” tab interface.

The visualization example here uses the colormap and modelwordcloud packages.

# Install libraries
install.packages(c("colormap", "modelwordcloud"))
library(colormap)
library(modelwordcloud)

Now let’s find our word cloud:

# Find word-based models by looking for "word" processes
wordModels <- allModels[grep("Word", lapply(allModels, `[[`, "processes"))]
wordModel <- wordModels[[1]]
# Get word cloud
wordCloud <- GetWordCloud(project, wordModel$modelId)
saveRDS(wordCloud, "wordCloudModelInsights.rds")

Now we plot it!

# Remove stop words
wordCloud <- wordCloud[!wordCloud$isStopword, ]

# Make word cloud
colors <- c(colormap(c("#255FEC", "#2DBEF9")), colormap(c("#FFAC9D", "#D80909"), reverse = TRUE))
wordcloud(words = wordCloud$ngram,
          freq = wordCloud$frequency,
          coefficients = wordCloud$coefficient,
          colors = colors,
          scale = c(3, 0.3))