Reply To: Stochastic rounding for machine learning

I’m just learning about neural networks myself (stanford coursera class). I immediately thought about using lower precision fixed point instead of the standard 32 bit floating point that it seems like everybody uses. I ran across that paper and looked around to see if there was any support for it in hardware and found none.

I think the lack of information is due to the lack of available hardware :). The paper I linked to had to use FPGA to test out the idea.

One more paper I found describes an 8 bit FP representation with 16 bit accumulation. It doesn’t seem to discuss training much, just feed forward (much less dependence on precision):

http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/37631.pdf

It is the back propagation phase which computes gradients which requires the precision.

This paper is referenced from the original one:

http://arxiv.org/pdf/1412.7024v4.pdf

It is mostly about comparing different representations and a sort of dynamic fixed point where the exponent changes as the gradients decrease over the training iterations.