6

I am using CNN in order to predict codes after analyzing text. As an example, I will write "I am crazy" .. the model will predict some code " X321". All this based on CNN.

I want to evaluate my model. I used Fscore (recall and precision). Can you advice me more metrics?

Green Falcon
  • 13,868
  • 9
  • 55
  • 98
Lizou
  • 215
  • 3
  • 7
  • 1
    I have summarized all the important metrics used in machine learning in this article https://medium.com/machine-learning-through-visuals/performance-metrics-machine-learning-through-visuals-84ebd7cdcc20 – user3303020 Aug 15 '18 at 20:19

2 Answers2

4

For model evaluation there are different metrics based on your model:

  • Confusion matrix

    Classification accuracy:
    (TP + TN) / (TP + TN + FP + FN)
    Error rate:
    (FP + FN) / (TP + TN + FP + FN)

  • Paired criteria

    Precision: (or Positive predictive value)
    proportion of predicted positives which are actual positive
    TP / (TP + FP)
    Recall: proportion of actual positives which are predicted positive
    TP / (TP + FN)

    Sensitivity: proportion of actual positives which are predicted positive
    TP / (TP + FN)
    Specificity: proportion of actual negative which are predicted negative
    TN / (TN + FP)
    True positive rate: proportion of actual positives which are predicted positive
    TP / (TP + FN)
    True negative rate: proportion of actual negative which are predicted negative
    TN / (TN + FP)
    Positive likelihood: likelihood that a predicted positive is an actual positive
    sensitivity / (1 - specificity)
    Negative likelihood: likelihood that a predicted negative is an actual negative
    (1 - sensitivity) / specificity

  • Combined criteria

    BCR: Balanced Classification Rate
    ½ (TP / (TP + FN) + TN / (TN + FP))
    BER: Balanced Error Rate, or HTER
    Half Total Error Rate: 1 - BCR
    F-measure harmonic mean between precision and recall
    2 (precision . recall) / (precision + recall)
    Fβ-measure weighted harmonic mean between precision and recall
    (1+β)2 TP / ((1+β)2 TP + β2 FN + FP)

    The harmonic mean between specificity and sensitivity is also often used and sometimes referred to as F-measure.

  • Youden's index: arithmetic mean

    between sensitivity and specificity
    sensitivity - (1 - specificity)
    Matthews correlation: correlation between the actual and predicted
    (TP . TN – FP . FN) / ((TP+FP) (TP+FN) (TP + FP) (TN+FN)) ^ (1/2)
    comprised between -1 and 1 Discriminant power normalized likelihood index sqrt(3) / π . (log (sensitivity / (1 – specificity)) + log (specificity / (1 - sensitivity))) <1 = poor, >3 = good, fair otherwise

You can find much more here. Also there are some explanations here and you can find useful code snippet from here which are implemented.

Green Falcon
  • 13,868
  • 9
  • 55
  • 98
  • Recall, TPR and sensitivity are the same. It’s confusing to have them all as if they are different metrics. – kbrose Jan 23 '18 at 15:41
  • precision and recall are different. Recall, TPR, and sensitivity are not. Your quote talks about precision and recall but not about Recall, TPR, and sensitivity. – kbrose Jan 23 '18 at 19:11
  • "Sensitivity: proportion of actual positives which are predicted positive TP / (TP + FN)" "True positive rate: proportion of actual positives which are predicted positive TP / (TP + FN)" These are the exact same sentences except for the label. I'm not seeing how that helps interpretability. – kbrose Jan 23 '18 at 19:29
0

SHORT ANSWER: Bayesian cost/benefit calculations directly tie "usefulness" to the evaluation of a model with metrics. Therefore, they are the only metrics (and there are an infinite number of them) which are actually useful.

For classification, use a Bayesian prior estimate for each class prevalence (relative class balance/imbalance) to convert a stochastic confusion matrix into an unconditional probability estimate of classification results, including both mistakes and correct classifications. Multiply each term of this matrix by a corresponding term in a cost/benefit matrix and sum them all up; then maximize benefit/minimize cost when comparing the classification algorithms to each other.

For regression, select a point estimator (vector) which minimizes the Bayesian loss function through variational calculus. For example, use the posterior mean if your loss is quadratic, the median if it is an inverted triangle, and the mode if it is a Dirac delta loss. Please note that I have never derived any such point estimator using the calculus of variations; I am going completely off my memory of E.T. Jaynes text Probability Theory: The Logic of Science.

brethvoice
  • 200
  • 8