After the training phase, is it better to run neural networks on a GPU or CPU?

Question

My understanding is that GPUs are more efficient for running neural nets, but someone recently suggested GPUs are only needed for the training phase and that once trained, it's actually more efficient to run them on CPUs.

Is this true?

Answer 1

That would depend entirely on what software platform you're running on, not on any particular characteristic of neural nets or their constituent objects. For example, Theano leverages GPUs because they're optimized for matrix math, which neural net weights and activations are often expressed as. On the other hand, they can also be expressed as sets, which makes them ideal for set-based languages like SQL; I prefer this approach, since sets are more flexible (you can, for example, operate easily on ragged dimensional sets, unlike with matrices) and SQL is a somewhat portable standard that makes it easier to think about neural net layers conceptually. As my screen name suggests, I use SQL Server, which is one of many database server platforms that can do some serious number-crunching on very large neural nets, all without the use of any GPU power. Long ago I also programmed neural nets in VB.Net, VB 6 and C# that efficiently made use of arrays, without leveraging GPUs at all.

I think the choice may come down to a combination of your existing skill set, what architectures you have access to, your portability requirements, code maintenance issues and all that, not to any explicit linkage between specific hardware components and the inner workings of neural nets. There may be slight advantages to modeling ragged matrices in SQL or to the matrix-crunching power of GPUs, but I don't think there's much practical difference in most scenarios, nor can I think of a way of testing such a comparison without getting into an apples vs. oranges conundrum. That being said, within a specific platform, it may be true that "GPUs are only needed for the training phase" and "once trained, it's actually more efficient to run them on CPUs." That I can't comment on, since I still with SQL and the .Net stack and avoid Theano etc. altogether (though I respect their capabilities). I hope that helps.