Naive Bayes: Divide by Zero error

Question

OK this is my first time in ML and for starter I am implementing Naive Bayes. I have Cricket(sports) data in which I have to check whether the team will win or lost based on Toss Won|Lost and Bat First|Second. Below is my code:

from sklearn.naive_bayes import GaussianNB
import numpy as np

"""
    Labels : Lost, Draw, Won [-1,0,1]
    Features
    ==========
        Toss(Lost,Won) = [-1,1]
        Bat(First, Second) = [-1,1]
"""
#Based on Existing Data Features are:
features = np.array([[-1, 1],[-1, 1]])
labels = np.array([0,1])
# Create a Gaussian Classifier
model = GaussianNB()

# Train the model using the training sets
model.fit(features, labels)

# Predict Output
predicted = model.predict([[1,0]])
print(predicted)

On running this I get error:

/anaconda3/anaconda/lib/python3.5/site-packages/sklearn/naive_bayes.py:393: RuntimeWarning: divide by zero encountered in log
[0]
  n_ij = - 0.5 * np.sum(np.log(2. * np.pi * self.sigma_[i, :]))
/anaconda3/anaconda/lib/python3.5/site-packages/sklearn/naive_bayes.py:395: RuntimeWarning: divide by zero encountered in true_divide
  (self.sigma_[i, :]), 1)
/anaconda3/anaconda/lib/python3.5/site-packages/sklearn/naive_bayes.py:395: RuntimeWarning: invalid value encountered in subtract
  (self.sigma_[i, :]), 1)

Update

Code given here

Hey, I used it in python 2.7 and it seems to be working fine. Maybe some changes in the versions are giving rise to the warnings. Isn't the predictions working still? — Hima Varsha, Sep 20 '16 at 08:17
@HimaVarsha I actually had to add more features to make it unique and then it worked.. Not sure why is like that — Volatil3, Sep 20 '16 at 09:28

Reii Nakano · Accepted Answer · 2017-07-04T03:49:32.720

5

Although I haven't verified it, a first glance at the features and training set you used shows an obvious problem. You have just two data samples with the exact same features while you give it different labels.

Although other types of models might not break and just give both samples an equal probability of being 0 or 1, something in the Naive Bayes classifier internal calculations probably requires there to be at least some difference in differently labeled samples (not an unreasonable assumption).

I can't confirm this for myself as I'd have to go deeper into the source code to actually check the scikit implementation so perhaps someone else more familiar with it can check.

EDIT: I just tested this out by changing one line in your code to

features = np.array([[-1, 1],[-1, 0]])

It worked.

edited Jul 04 '17 at 03:49

answered Sep 21 '16 at 08:06

Reii Nakano

291
1
5

The code and data given here (https://github.com/kadnan/PakistanEnglanTestMatches) - Also I think small test data does not make difference regardless of whichever Algo is used? – Volatil3 Sep 22 '16 at 04:25
1

The point is not small test data. The point is you have two classes with virtually no difference between the two classes. – Reii Nakano Jul 04 '17 at 03:47

score 1 · Answer 2 · answered Jan 30 '18 at 21:27

First of all, it seems that your data is discrete, and therefore I would advise using Multinomial Naive Bayes (scikit-learn also provides an implementation).

I ran your code and the reason your code breaks is not because of the implementation, but because of the data.

Fitting works, but if you run model.sigma_, you will see that the variance of your features equals 0.

This will cause a runtime error since Gaussian Naive Bayes models class probabilities as follows:

Since sigma equals zero here, your code will crash. If you simply change your code to:

from sklearn.naive_bayes import GaussianNB
import numpy as np

"""
    Labels : Lost, Draw, Won [-1,0,1]
    Features
    ==========
        Toss(Lost,Won) = [-1,1]
        Bat(First, Second) = [-1,1]
"""
#Based on Existing Data Features are:
features = np.array([[-1, -1],[-1, 1]])
labels = np.array([0,1])
# Create a Gaussian Classifier
model = GaussianNB()

# Train the model using the training sets
model.fit(features, labels)

# Predict Output
predicted = model.predict([[1,0]])
print(predicted)

Where all you do is change your features to features = np.array([[-1, -1],[-1, 1]]), your code will now run.

Naive Bayes: Divide by Zero error

2 Answers2