Sig. figs in IGOR PRO

Share icon

Thu, 09/03/2015 - 01:11 pm

I have a data file (attached). I opened it notepad, and focused on one of the numbers. It is 2.19376E+2 (this is just an example, it applied to all other numbers). I even open it in a hex editor, and found the same number. There are no additional significant figures in this number beyond the 6.

Now, I open the file in Igor Pro. I go to Modify Columns, select Decimal and choose to display 6 sig figs after the decimal point. and now the number is 219.376007. The '7' just came from nowhere. This is just an example, you can see yourself. DCVT001.zip
It appears your wave was created as single precision. The 7 is and artifact of asking for more digits than single precision can provide. This illustrates the same thing:
make jack= 2.19376E+2
edit jack
ModifyTable format(jack)=3,digits(jack)=6
make /d jackd= 2.19376E+2
append jackd
ModifyTable format(jackd)=3,digits(jackd)=6


note that jackd was made with double precision.


Larry Hutchinson
WaveMetrics
support@WaveMetrics.com

September 3, 2015 at 02:22 pm - Permalink