Str2Num rounding issue
bverdoorn
Mon, 01/09/2012 - 02:11 pm
For instance, I know the result should be 22224047 and I end up with 22224048 (or 22224046). Any suggestions on how to solve this?
Mon, 01/09/2012 - 02:11 pm
On the latest Igor version on Macintosh, this seems to work properly.
variable myVar = str2num(myStr)
print/d myVar
22224047
January 9, 2012 at 03:35 pm - Permalink
It looks like you are converting integers; if not, also be aware that num2str gives you six-digit precision.
John Weeks
WaveMetrics, Inc.
support@wavemetrics.com
January 9, 2012 at 04:43 pm - Permalink