Str2Num rounding issue

Share icon

Mon, 01/09/2012 - 02:11 pm

I'm trying to convert eight digit numbers from strings to numbers using the str2num function. It seems like there is some sort of rounding issue though, because once they are converted the numbers are off by a digit in a seemingly random direction.

For instance, I know the result should be 22224047 and I end up with 22224048 (or 22224046). Any suggestions on how to solve this?
You haven't actually told us what the problem is. Please post the Igor commands you are using and the result Igor is giving you. Please also post your Igor version and operating system.

On the latest Igor version on Macintosh, this seems to work properly.

String myStr = "22224047"
variable myVar = str2num(myStr)
print/d myVar
  22224047

January 9, 2012 at 03:35 pm - Permalink

If your eight-digit numbers are stored in a wave, be sure the wave is double precision. A single-precision number has a precision of about seven digits.

It looks like you are converting integers; if not, also be aware that num2str gives you six-digit precision.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com

January 9, 2012 at 04:43 pm - Permalink