str2num and scientific notation behavior

We regularly import .csv files which use scientific notation in the form -1.57E-7, for example. When Igor Pro 9 imports the .csv and converts these characters to numbers stored in waves, some non-zero digits are added far to the right of the decimal place. The behavior appears to be the same when I declare string and numeric variables:

string krep = "-1.5e-07"
variable krap = str2num(krep)
print/d krap
  -1.5e-07
make/n=1 kreeb; kreeb[0] = krap
Edit/K=0 root:kreeb
print/d kreeb[0]
  -1.50000005305628e-07
print/d krap
  -1.5e-07
print/d kreeb[0]-krap
  -5.30562829905034e-15
string kreg = "-1.4e-7"
variable krag = str2num(kreg)
print/d krag
  -1.4e-07
print/d krap - krag
  -9.99999999999998e-09

I have been multiplying, rounding, and then dividing the imported waves to get around this, but I wondered if anyone can explain to me why this happens and if there is a simpler way around it.

 

Cheers

-Solomon

You are running into floating-point precision limits here, i.e., how precise you can store fractional numbers. Note that you use 'make' without the /D flag, which creates single-precision waves, further decreasing the precision of storage. I would to recommend to always use the /D flag unless you actually want single precision (for memory reasons).

make/d/o/n=1 kreeb; kreeb[0] = krap
print/d kreeb[0]
  -1.5e-07
print/d kreeb[0]-krap
  0

Still, you will run into this problem eventually if your numbers get out of hand. This is a fact of life with handling numbers on computers. In fact, if print/d could display more digits, you would still find that the result is slightly off. Interestingly, your chosen example of 0.15 is a somewhat famous example, and you often find others wondering the same.

So rounding is your friend, but not when storing the value (since you never can make the imprecision go away) but rather when working with the data such as the example of subtraction. If you are interested in the underlying reason why the number you are seeing came out like that, here is a tool to play around with: https://www.h-schmidt.net/FloatConverter/IEEE754.html

chozo,

Thank you very much for your explanation and the link to the float converter tool. I had not seen that tool before and it makes it much more clear what the computer is doing.