3

For the imbalanced datasets:

  1. Can we say the Precision-Recall curve is more informative, thus accurate, than ROC curve?
  2. Can we rely on F1-score to evaluate the skillfulness of the resulted model in this case?
Dave
  • 238
  • 2
  • 7
  • Yes. Precision-Recall curve is absolutely reliable for imbalanced datasets when compared with ROC curve. F-1 score is so far one of the most reliable metric to measure the performance of a model. You are thinking in the right direction. – Adhira Deogade May 07 '20 at 18:07
  • 1
    @AdhiraDeogade Thank you very much for your note. very valuable – Dave May 07 '20 at 18:17
  • Sure. Keep asking. – Adhira Deogade May 07 '20 at 18:18
  • @AdhiraDeogade Reliable with respect to what? It seems there is another measure apart from Precision-Recall/ROC/F1 that you are using to judge these. – JTH May 07 '20 at 23:35
  • @JTH When it comes to imbalanced dataset in particular, precision and recall help to understand the performance with respect to the desired class.ROC curve is useful only when the data is balanced. If you have a class of interest, PR curve seems reliable measure. What are your thoughts? – Adhira Deogade May 08 '20 at 00:13
  • 2
    I just think it is reckless to say "measure A is better than measure B" when no evidence or benchmark is given. I think the value of one measure over another additionally depends on what the practitioner intends to do with the model, unless we are just talking about the intrinsic value of one model over another, then I believe proper scoring rules should be used. – JTH May 08 '20 at 02:29

1 Answers1

1

Precision-recall curves are argued to be more useful than ROC curves in "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets" by Saito and Rehmsmeier. They argue that ROC might lead to the wrong visual interpretation of specificity.

F1-score equally balances precision and recall. In some domains it might be more useful to more heavily weight precision (F < 1) or recall (F > 1).

Brian Spiering
  • 20,142
  • 2
  • 25
  • 102