Academic CS in Production Code

8-bit Numerical Formats for Deep Neural Networks

Badreddine Noune, Philip Jones, Daniel Justus, Dominic Masters, Carlo Luschi
2022
16 references

Abstract

Given the current trend of increasing size and complexity of machine learning architectures, it has become of critical importance to identify new approaches to improve the computational efficiency of model training. In this context, we address the advantages of floating-point over fixed-point representation, and present an in-depth study on the use of 8-bit floating-point number formats for activations, weights, and gradients for both training and inference. We explore the effect of different bit-widths for exponents and significands and different exponent biases. The experimental results demonstrate that a suitable choice of these low-precision formats enables faster training and reduced power consumption without any degradation in accuracy for a range of deep learning models for image classification and language processing.

View Paper PDF DOI
⚙️ Classical Compilers 🤖 Machine Learning 🧠 ML Compilers 🖥️ Operating Systems
6 repositories
12 references

Code References

▶ freebsd/freebsd-src
1 file
▶ contrib/llvm-project/llvm/include/llvm/ADT/APFloat.h
2
// and bit layout S1E5M2 described in https://arxiv.org/abs/2206.02915,
// and bit layout S1E4M3 described in https://arxiv.org/abs/2206.02915,
▶ iree-org/iree
1 file
▶ runtime/src/iree/base/internal/math.h
2
// and bit layout S1E5M2 described in https://arxiv.org/abs/2206.02915,
// and bit layout S1E4M3 described in https://arxiv.org/abs/2206.02915,
▶ llvm/llvm-project
1 file
▶ llvm/include/llvm/ADT/APFloat.h
2
// and bit layout S1E5M2 described in https://arxiv.org/abs/2206.02915,
// and bit layout S1E4M3 described in https://arxiv.org/abs/2206.02915,
▶ microsoft/onnxruntime
1 file
▶ csharp/tools/Microsoft.ML.OnnxRuntime.PerfTool/OnnxMl.cs
1
/// 8-bit Numerical Formats For Deep Neural Networks, https://arxiv.org/pdf/2206.02915.pdf.
▶ onnx/onnx
1 file
▶ docs/docsgen/source/technical/float8.md
1
Second paper [8-bit Numerical Formats For Deep Neural Networks](https://arxiv.org/pdf/2206.02915.pdf) introduces
▶ pytorch/pytorch
3 files
▶ docs/source/tensor_attributes.rst
2
``torch.float8_e4m3fnuz`` [shell]_, [1]_ 8-bit floating point, S-E-M 1-4-3, from https://arxiv.org/pdf/2206.02915
``torch.float8_e5m2fnuz`` [shell]_, [1]_ 8-bit floating point, S-E-M 1-5-2, from https://arxiv.org/pdf/2206.02915
▶ torch/headeronly/util/Float8_e4m3fnuz.h
1
/// Implementation based on the paper https://arxiv.org/pdf/2206.02915.pdf and
▶ torch/headeronly/util/Float8_e5m2fnuz.h
1
/// Implementation based on the paper https://arxiv.org/pdf/2206.02915.pdf and
Link copied to clipboard!