Large integer value in numeric wave

Share icon

I encountered an issue working with large integer value in numeric wave.

It appears that when the a number larger than 2^24 is stored in numeric wave, it can be off.

For example, I was expecting the values in wave0 to be 16777216, 16777217, 16777218, and 16777219 if I run the following commands. However, they are actually 16777216, 16777216, 16777218, and 16777220.

make/n=4 wave0
wave0 = p + 2^24

My question is, how do I handle this properly if I need the last digit precision.

make without /D creates a single-precision floating point number, which is a 32-bit value.

The mantissa part of a single-precision float is only 23 bits. The other 9 bits are sign and exponent:

A google AI result is correct:


A 32-bit single precision floating-point number, defined by the IEEE 754 standard, is formed by combining a 1-bit sign, an 8-bit biased exponent, and a 23-bit mantissa. The sign bit (0 for positive, 1 for negative) indicates the number's sign, the mantissa stores the significant digits of the number, and the exponent field contains a biased value (exponent + 127) to represent the number's magnitude. 

The largest integer exactly representable in a single-precision (IEEE 754 binary32) floating-point number is 16,777,216 (2^24). While a single-precision float can store numbers up to approximately 3.4 x 10^38, it loses precision for integers larger than 2^24 (16,777,216) because its 24-bit significand (mantissa) is not large enough to hold all the bits of a larger integer precisely

Use Make/O/D to increase the mantissa size to 53 bits.

September 19, 2025 at 06:49 pm - Permalink