FP16 (Half Precision) | Floating Point Format Guide

Bit Layout

An FP16 number uses 16 bits divided into three fields:

Sign

S

1 bit

Exponent

E

5 bits

Mantissa

M

10 bits

The sign bit determines positive (0) or negative (1). The exponent is stored with a bias of 15. The mantissa stores 10 bits of fractional precision.

Overview

FP16, also known as half precision, was added to the IEEE 754-2008 standard as a storage and interchange format. At just 16 bits, it uses half the memory of FP32, making it attractive for applications where memory bandwidth is a bottleneck.

With only 5 exponent bits, FP16 has a much narrower range than FP32, with a maximum value of only 65,504. This means large values (like loss values in deep learning) can easily overflow to infinity. However, for values within its range, the 10 mantissa bits provide about 3.3 decimal digits of precision.

FP16 is widely used in ML inference, computer graphics (HDR textures), and as a storage format on GPUs. NVIDIA's Tensor Cores can perform FP16 matrix multiplications at 2× the throughput of FP32.

FP16 vs BF16 FP16 and BF16 are both 16-bit formats, but they make different trade-offs. FP16 has higher precision (10 mantissa bits vs 7) but much narrower range (max ~65K vs ~3.4×10³⁸). BF16's wider range makes it better for training, while FP16's precision makes it good for inference.

Encoding Rules

Normal Numbers

value = (-1)^sign × 2^{(exponent - 15)} × (1 + mantissa / 2¹⁰)

Normal FP16 numbers have a biased exponent between 1 and 30 (actual exponent -14 to +15). The bias of 15 is calculated as 2^(e-1) - 1, where e is the number of exponent bits (5).

Subnormal Numbers

value = (-1)^sign × 2^-14 × (0 + mantissa / 2¹⁰)

Subnormals bridge the gap between the smallest normal number and zero, enabling gradual underflow.

Special Values

Zero: Exponent = 0, Mantissa = 0.
Infinity: Exponent = 31 (all ones), Mantissa = 0.
NaN: Exponent = 31, Mantissa ≠ 0.

Interactive Value Visualizer

Click any bit to flip it, drag the slider, or enter a decimal or hex value. The graphs show how values are distributed across the encoding space.

Decimal:

Hex:

Dynamic Range & Precision

Property	Value
Max Positive (Normal)	65504
Min Positive (Normal)	0.00006103515625
Max Subnormal	0.00006097555161
Min Positive (Subnormal)	5.960464478e-8
Machine Epsilon (at 1.0)	0.0009765625
Approx. Decimal Digits	~3.0
Exponent Bias	15
Exponent Range	2^-14 to 2^15
Total Bit Patterns	65,536
Supports Infinity	Yes
Supports NaN	Yes

Special Values & Bit Patterns

Value	Bit Pattern	Decimal
+0	0 00000 0000000000	0
-0	1 00000 0000000000	-0
Min Subnormal	0 00000 0000000001	5.960464478e-8
Max Subnormal	0 00000 1111111111	0.00006097555161
Min Normal	0 00001 0000000000	0.00006103515625
Max Normal	0 11110 1111111111	65504
+Infinity	0 11111 0000000000	+Infinity
-Infinity	1 11111 0000000000	-Infinity
NaN	0 11111 1000000000	NaN

Format Comparison

Property	FP16	FP32	BF16	FP8 E5M2
Total Bits	16	32	16	8
Sign Bits	1	1	1	1
Exponent Bits	5	8	8	5
Mantissa Bits	10	23	7	2
Bias	15	127	127	15
Max Value	65504	3.402823466e+38	3.389531389e+38	57344
Min Normal	0.00006103515625	1.175494351e-38	1.175494351e-38	0.00006103515625
Epsilon	0.0009765625	1.192092896e-7	0.0078125	0.25
~Decimal Digits	~3.0	~6.9	~2.1	~0.6
Has Infinity	Yes	Yes	Yes	Yes
Has NaN	Yes	Yes	Yes	Yes
Has Subnormals	Yes	Yes	Yes	Yes

Where FP16 Is Used

GPU Tensor Cores: NVIDIA GPUs from Volta onward accelerate FP16 matrix operations via Tensor Cores (.f16 / .f16x2 types in the PTX ISA). On H100, FP16 Tensor Cores deliver 1000 TFLOPS (2000 with sparsity).
Mixed-precision training: NVIDIA's mixed-precision training guide documents the FP16 recipe: forward and backward passes in FP16, FP32 master weights, and dynamic loss scaling to prevent gradient underflow.
ML frameworks: PyTorch provides torch.float16 as a core dtype. The ONNX specification defines FLOAT16 = 10 as an IEEE 754 half-precision type for model interchange.
Kernel compilers: CUTLASS defines half_t for templated MMA abstractions. Triton exports float16 as a first-class kernel dtype, and the MLIR NVVM dialect maps FP16 to Tensor Core MMA shapes.
Graphics and HDR imaging: FP16 is the standard for HDR color channels and texture storage in OpenGL and Vulkan, providing sufficient dynamic range and precision for visual data.

Limited range FP16 overflows above 65,504 and underflows below ~6×10^-8 (normal range). Loss values, gradient norms, and intermediate activations can exceed these bounds. If you encounter frequent infinities or zeros, consider BF16 (wider range) or add loss scaling.