Assigning data values from variables

Share icon

Hi there,

I'm having an issue with data values generated from a user prompt that I then assign to a wave. The basic wave assignment is: 

Prompt sampInt, "electrode sampling interval (in mm)"
DoPrompt "Select sampling interval", sampInt    
 
Make /O/N=1 eTrodeInterval = {sampInt}


It's not a problem to create the wave, but the data value is far more precise than the user input (e.g. 0.05 becomes 0.0500000007450581). I'd like to use the value in this wave as part of an algorithm in a later step, but the rounding error generated from the extra decimal places is prohibitive.

Is there a way to ensure the data value is assigned to the wave exactly as the user enters it?

Thanks for your help!
I get .5 on my Macintosh using Igor Pro 6.22A. Try this:
Function Test()
    Variable sampInt
    Prompt sampInt, "electrode sampling interval (in mm)"
    DoPrompt "Select sampling interval", sampInt
    Print/D sampInt
End


Also, in general floating point numbers do not have infinite precision so only a small subset of numbers can be represented exactly. However 0.5 is one that can be represented exactly. See What Every Computer Scientist Should Know About Floating-Point Arithmetic".

August 3, 2012 at 08:51 am - Permalink

Perfect, thanks! The reference will be very helpful as well. I appreciate your help!

August 3, 2012 at 09:10 am - Permalink