Why is there no 2-byte float and does an implementation already exist?

by

TL;DR: 16-bit floats do exist and there are various software as well as hardware implementations

There are currently 2 common standard 16-bit float formats: IEEE-754 binary16 and Google’s bfloat16. Since they’re standardized, obviously anyone who knows the spec can write an implementation. Some examples:

Or if you don’t want to use them, you can also design a different 16-bit float format and implement it

2-byte floats are generally not used, because even float’s precision is not enough for normal operations and double should always be used by default unless you’re limited by bandwidth or cache size. Floating-point literals are also double when using without a suffix in C and C-like languages. See

However less-than-32-bit floats do exist. They’re mainly used for storage purposes, like in graphics when 96 bits per pixel (32 bits per channel * 3 channels) are far too wasted, and will be converted to a normal 32-bit float for calculations (except on some special hardware). Various 10, 11, 14-bit float types exist in OpenGL. Many HDR formats use a 16-bit float for each channel, and Direct3D 9.0 as well as some GPUs like the Radeon R300 and R420 have a 24-bit float format. A 24-bit float is also supported by compilers in some 8-bit microcontrollers like PIC where 32-bit float support is too costly. 8-bit or narrower float types are less useful but due to their simplicity, they’re often taught in computer science curriculum. Besides, a small float is also used in ARM’s instruction encoding for small floating-point immediates.

The IEEE 754-2008 revision officially added a 16-bit float format, A.K.A binary16 or half-precision, with a 5-bit exponent and an 11-bit mantissa

Some compilers had support for IEEE-754 binary16, but mainly for conversion or vectorized operations and not for computation (because they’re not precise enough). For example ARM’s toolchain has __fp16 which can be chosen between 2 variants: IEEE and alternative depending on whether you want more range or NaN/inf representations. GCC and Clang also support __fp16 along with the standardized name _Float16. See How to enable __fp16 type on gcc for x86_64

Recently due to the rise of AI, another format called bfloat16 (brain floating-point format) which is a simple truncation of the top 16 bits of IEEE-754 binary32 became common

The motivation behind the reduced mantissa is derived from Google’s experiments that showed that it is fine to reduce the mantissa so long it’s still possible to represent tiny values closer to zero as part of the summation of small differences during training. Smaller mantissa brings a number of other advantages such as reducing the multiplier power and physical silicon area.

  • float32: 242=576 (100%)
  • float16: 112=121 (21%)
  • bfloat16: 82=64 (11%)

Many compilers like GCC and ICC now also gained the ability to support bfloat16

More information about bfloat16:

In cases where bfloat16 is not enough there’s also the rise of a new 19-bit type called TensorFloat

More Related Contents:

  1. How do I print a double value with full precision using cout?
  2. round() for float in C++
  3. Floating point division vs floating point multiplication
  4. Floating point comparison [duplicate]
  5. Why can’t I use float value as a template parameter?
  6. Floating point vs integer calculations on modern hardware
  7. Sin and Cos give unexpected results for well-known angles
  8. Efficient implementation of log2(__m256d) in AVX2
  9. Should I use double or float?
  10. Why is my integer math with std::pow giving the wrong answer?
  11. Floating point comparison revisited
  12. Building a 32-bit float out of its 4 composite bytes
  13. Why does Clang optimize away x * 1.0 but NOT x + 0.0?
  14. How computer does floating point arithmetic?
  15. Comparison of double, long double, float and float128?
  16. How to correctly and standardly compare floats?
  17. Convert a float to a string
  18. Math precision requirements of C and C++ standard
  19. Should we generally use float literals for floats instead of the simpler double literals?
  20. printf rounding behavior for doubles
  21. Why do sin(45) and cos(45) give different results? [duplicate]
  22. C++ floating point to integer type conversions
  23. How to write a Float Mat to a file in OpenCV
  24. Does one double promote every int in the equation to double?
  25. Handling overflow when casting doubles to integers in C
  26. When a float variable goes out of the float limits, what happens?
  27. Why isn’t “0f” treated as a floating point literal in C++?
  28. How is floating point conversion actually done in C++?(double to float or float to double)
  29. How to force GCC to assume that a floating-point expression is non-negative?
  30. Is there any advantage to using pow(x,2) instead of x*x, with x double?

Leave a Comment