Questions tagged [knowledge-distillation]
5 questions
1
vote
1 answer
Distilling a Random Forest to a single DecisionTree, does it make sense?
I stumbled into this blog which shows how a decision tree trained to overfit the predictions of a properly trained random forest model is able to generalize in pretty much the same way as the original random forest.
I'm interested in this as I'm…
amiando
- 35
- 4
1
vote
1 answer
Why there is so few research on neural code of artificial neural networks and are there alternatives to the neural code approach?
I feel that the neural code/neural coding (how neurons or biases enode the symbolic concepts of the chains of concepts, e.g. each feature is chain of symbolic functions and their parameters) is the key to understanding neural networks and hence -…
TomR
- 141
- 4
1
vote
0 answers
Can I distill knowledge across different neural network frameworks?
I'm interested in using knowledge distillation to compress a large deep learning model to a smaller size so it will run on an embedded device. I've found a number of open source examples for the knowledge distillation…
Sledge
- 254
- 1
- 9
0
votes
0 answers
What are the state-of-the-art knowledge distillation methods?
I need to implement some state-of-the-art knowledge distillation (KD) methods to distill dark knowledge of the teacher network to the student network with Pytorch.
I would really appreciated to any advice to find the state-of-the-art KD methods.
SaMo
- 1
- 1
0
votes
0 answers
Can knowledge distillation be applied to non-neural network?
I am new to knowledge distillation. I have read the paper, and I understand that it works by minimising the KL divergence of the probability distribution output of the teacher and student network, (the output before applying the sigmoid…
