Large integer value in numeric wave
I encountered an issue working with large integer value in numeric wave.
It appears that when the a number larger than 2^24 is stored in numeric wave, it can be off.
For example, I was expecting the values in wave0 to be 16777216, 16777217, 16777218, and 16777219 if I run the following commands. However, they are actually 16777216, 16777216, 16777218, and 16777220.
make/n=4 wave0 wave0 = p + 2^24
My question is, how do I handle this properly if I need the last digit precision.
make without /D creates a single-precision floating point number, which is a 32-bit value.
The mantissa part of a single-precision float is only 23 bits. The other 9 bits are sign and exponent:
A google AI result is correct:
Use Make/O/D to increase the mantissa size to 53 bits.
September 19, 2025 at 06:49 pm - Permalink
Thanks for the solution. That solves my problem.
September 20, 2025 at 05:57 am - Permalink
Maybe the /L ( /U) flag is even more appropriate if you are dealing with integers.
October 21, 2025 at 06:20 am - Permalink
In general, I would not recommend using /L or /U unless you are dealing with binary logic or pure integer operations. The problem with 64-bit integers is that they are not represented exactly by DP variables (over 53 bits). Most computations in Igor are performed in DP so you really need to be careful when performing operations on 64-bit integer waves. For example:
make/n=10/L ddd=2^54 // function not available for this number typeor
matrixop/o/p=1 aa=sum(ddd) // does not support int64 waves... and,
So conversion to DP is problematic. On the other hand, integer computation without conversion to DP works:
October 21, 2025 at 11:05 am - Permalink