1

I think I know the answer to this question but I am looking for a sanity check here: Is it appropriate to use z-test scores in order to evaluate the performance of my model?

I have a binary model that I have developed with a NN in Keras. I know the size of my (equally balanced) training set and it has a proportion of 0.5 (duh!). I know that with my business use case, false-positives are financially expensive so I'm focusing on Precision as my metric. So, in validation, can't I take that Precision metric as a proportion of my validation set (which I also know the size of) and then get the z-test score calculation? That should give me a threshold for validation Precision at which my model is doing more than just flipping a coin.

Can someone confirm my line of thinking or am I way off base here?

I_Play_With_Data
  • 2,079
  • 2
  • 16
  • 39
  • 1
    Do you mean a z-test? // You might be interested in proper scoring rules and the probability outputs of your neural network. Frank Harrell's blog discusses this topic: [1](https://www.fharrell.com/post/class-damage/) [2](https://www.fharrell.com/post/classification/). – Dave Jun 21 '21 at 15:23
  • @Dave Fair point. I have edited my question – I_Play_With_Data Jun 21 '21 at 15:25
  • 1
    Why not just take every observation as a negative? Then you have zero false positives? (I get why this will get you fired, but consider why this is not a viable business solution.) – Dave Jun 21 '21 at 15:35

1 Answers1

0

From your description, you can not use Z-test because the z-test requires knowing the population variance.

Brian Spiering
  • 20,142
  • 2
  • 25
  • 102