What is the correct way to compute Mean F1 score?

Question

I have a set of 10 experiments that compute precision, recall and f1-score for each experiment. Now, average precision & average recall is easy to compute. I have some confusion regarding average f1-score.

There are 2 ways on how i can compute mean f1-score:

Take f1 scores for each of the 10 experiments and compute their average.
Take average precision & average recall and then compute f1-score using the formula f1 = 2*p*r/(p+r)

I could not find any strong reference to support any of the arguments. The closest document i could find is this: https://www.kaggle.com/wiki/MeanFScore

Can anyone explain with some reference (if possible) which of the methods is correct and why?

EDIT: One of the members suggested this source. Though, i still suspect the reliability of the source. I have seen people not using the method explained above in their research publications. (Even i would be using it in one of my publications) I would expect some more opinions from the community to verify this idea.

You might find [this](http://rushdishams.blogspot.in/2011/08/micro-and-macro-average-of-precision.html) useful. — Kiritee Gak, Jan 09 '17 at 06:42
How are you calculating the average precision/recall? Micro or macro? — Hima Varsha, Jan 09 '17 at 11:43
@Kiritee From that reference it looks like in both the methods(micro/macro) the method used is HM of average-precision & average-recall. One of my professors also suggested this. So i guess this should be correct. — Pinkesh Badjatiya, Jan 09 '17 at 18:13
@oW_ That is a good idea only when i am absolutely sure about the answer. I would still expect a few more suggestions. — Pinkesh Badjatiya, Jan 10 '17 at 21:08
Like ROC-AUC, this metric is ill-defined for multiclass problems. There is no consesus. Sklearn implements several [averaging strategies.](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) — Ricardo Magalhães Cruz, Jan 13 '17 at 00:47
See the book [Modern Information Retrieval](http://www.mir2ed.org/) in chapter 4. — Howard, Jan 13 '17 at 12:11
@Howard It would be more helpful if you can point to some specific page or put some excerpts from the book as most people do not have access to the text. — Kiritee Gak, Jan 13 '17 at 15:19
what's meaning of mean F-score https://www.kaggle.com/wiki/MeanFScore could you explain it, please! — tktktk0711, Sep 08 '17 at 02:02

score 9 · Answer 1 · answered Jan 22 '17 at 13:11

This paper Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement by Forman and Scholz discuss the different methods for computing the average F-score in cross validation. The paper shows that under very high class imbalance some of the computation methods (average of individual folds F-scores or F-score based on average of individual folds precision and recall) can lead to biased results. The paper recommends computing the F-score by adding the TP, FP, FN from each fold, computing the precision and recall and finally the F-score.

Pinkesh Badjatiya · Accepted Answer · 2018-01-01T12:32:16.917

2

As mentioned by other users, the solution is not very clear. The general approach is to follow what is mentioned here.

Also, as suggested by one of the senior R&D employee and my mentor, the method in practice is to calculate average f1-score as the HM of average precision and average recall.

This surely depends on your usecase as well as how you are calculating the metric (micro/macro).

edited Jan 01 '18 at 12:32

answered Jan 21 '17 at 23:47

Pinkesh Badjatiya

239
1
2
6

score 0 · Answer 3 · answered Jan 13 '17 at 16:19

As you observed, one can argue for any of your definitions. It is most important that you document what you mean with "mean F1-score". You should also consider which of the 2 options provides a more meaningful evaluation. This depends on your specific application or task.

In my opinion, "mean F1-score" clearly means that you calculate the mean of individual F1-scores. In some situations, option 2 can be described as the overall F1-score. This depends on what you are aggregating. "F1-score of the mean precision and recall" might be a good general description of option 2.

What is the correct way to compute Mean F1 score?

3 Answers3