Data Science Stack Exchange
Data Science Stack Exchange

Questions tagged [confidence]

25 questions
2
votes
2 answers

Confidence rating for regression tasks

In classification tasks, we can interpret the output vector as how "confident" the model is that the input has a certain label. For example, y = [0.01 0.20 0.99 0.10] would mean the model is 99% certain the input has label with index 2 and 1%…
Mossmyr
  • 123
  • 4
2
votes
1 answer

Relation between Cross Validation and Confidence Intervals

I've read from a source which I forgot where that 'In cross validation, the model with best scores at 95% confidence interval is picked'. But according to my stat knowledge, in order for CI (confidence interval) to works, you need normality…
Wong
  • 103
  • 4
2
votes
2 answers

Predict_proba for Binary classifier in Tensorflow

I'm working in binary classifier problem, where I have used Tensorflow low level API's. Last layer wrapped with Sigmoidal Function and just returning a single value. For my prediction I just set a standard threshold value of 0.5 and hence if it's…
vipin bansal
  • 1,252
  • 9
  • 17
1
vote
1 answer

Plotting confidence intervals

For the following dataframe, I am trying to plot the means of a sample of 5 random rows . And also plot their respective confidence intervals using errorbars. I am unable to figure how to plot the confidence intervals using errorbars. col0 col1 …
freshman_2021
  • 143
  • 5
1
vote
1 answer

Confidence score for all observations is between 0.50 - 0.55

Hello Data Science Stack Exchange Community, This question will appear to be open-ended, however any answers or thought will be much appreciated. I am trying to go-through a pre-trained Random Model Classifier with minimum documentation like what…
Aman Raparia
  • 257
  • 2
  • 8
1
vote
0 answers

Confidence in the rewards for a RL task

For a RL task that I am trying to solve, for which I train once per day, I have the rewards stored for each of those days, so that I can see the progress on daily basis. In the beginning of the learning process, the reward for a given state…
ilirosmanaj
  • 11
  • 1
1
vote
1 answer

Interpreting confidence interval results for datasets

I have created a dataset automatically and wanted to clarify my interpretation of the amount of noise using the confidence interval. I selected a random sample and manually annotated the sample and found that 98% of the labels were correct. Based on…
dmnte
  • 23
  • 3
1
vote
1 answer

Understand how to simulate a statistics

This solution describes how to simulate statistics to find a confidence interval. A journalist called 1000 people in town to ask who will they be voting for out of candidates A and B. The observed value came out to be 511 votes for A and 489 votes…
maindola
  • 11
  • 2
1
vote
0 answers

Correctness of derivation for binary F1 variance for F1 confidence intervals

I'm developing a python library for confidence intervals for common accuracy metrics, with both analytic and bootstrap computations. Following this paper, I implemented the Macro and Micro F1 scores analytic confidence intervals. However the…
Jacob G
  • 121
  • 2
1
vote
1 answer

Overall acurracy +/- E (with 90% C.I.)

I am assessing the accuracy of my classification model. I performed a 4-folds cross-validation and I obtained the following Overall Accuracy: OA = (0.910, 0.920, 0.880, 0.910). So, the average OA is 0.905. My dataset contains 120 samples, therefore…
sermomon
  • 63
  • 7
1
vote
1 answer

How to test the confidence for a rule based system?

I have a multi-class dataset and I generated based on it rules. That is, if certain features are seen then it must be a certain class. I chose only rules with precision 1 (with respect to the whole dataset) It is worth mentioning the dataset is…
greenButMellow
  • 131
  • 2
1
vote
1 answer

How to use confidence labels?

I have 2 sets of training data in csv files. The training data have class labels, 1 for memorable, and 0 for not memorable. In addition, there is also a confidence label for each sample. The class labels were assigned based on decisions from 3…
user135529
1
vote
0 answers

Evaluate Dendrogram Statistical Significance

I have N=21 objects and each one has about 80 possible not NaN descriptors. I carried out a hierarchical clustering on the objects and I obtained this dendrogram. I want some kind of 'confidence' index for the dendrogram or for each node. I saw…
Mirko
  • 11
  • 3
1
vote
2 answers

How are the confidence intervals of a model interpreted?

I am doing some work with R and after obtaining the confusion matrix I have obtained the following metrics corresponding to a logistic regression: Accuracy : 0.7763 95% CI : (0.6662, 0.864) No Information Rate : 0.5395 …
PicaR
  • 304
  • 2
  • 13
1
vote
0 answers

Association rules - Find 100% confidence rules

Suppose there are 100 items, numbered 1 to 100, and also 100 baskets, also numbered 1 to 100. Item i is in basket b if and only if i divides b with no remainder. Thus, item 1 is in all the baskets, item 2 is in all fifty of the even-numbered…
AskSmart
  • 11
  • 1
1
2