Rounding error - using floor

I have just come across this somewhat odd inconsistency:

print( floor(5 * (1 - (80 / 100) ) ) )
  0
print( floor(5 * (100 - 80) / 100 ) )
  1

My guess is that this is due to floating point arithmetic, or something similar?

Not a big deal as I am (now!) using the second expression.

My understanding is that floating point numbers are described as sums of 2^2,2^1,2^0, 2^-1, 2^-2, etc....

That means 1 is perfectly described (2^0), so is 0.5 (2^-1) and 0.25 (2^-2), but 0.8 and 0.2 are not perfectly described.

Buttom line. You should not trust either of the two equations you listed to return a perfect integer.

Maybe someone more knowledgeable than me can correct me.

In reply to by olelytken

I have looked into this more - I'm pretty sure that it is due to floating point representation. Floor (and ceil) are functions where in some circumstances the tiny imprecision of floating point can matter (0.999999999 vs 1.000000001 for example, albeit to the appropriate number of decimal places).
I have tested and found the same odd behavior occurs in both R and Python, so perhaps it is embedded in the maths engine that all these systems use?

Thank you John,

I like the idea of adding 'somesmallnumber', and appreciate and understand the need to choose its value carefully. 

While John's suggestion of floor(somenumber + somesmallnumber) might work, it's like playing with fire.

A better solution is to rewrite your equation so that you are working with integer values as much as you can. That's exactly what the second equation in the original question does. In the first equation, 80/100 is evaluated first, and that is a non-integral value which cannot be precisely represented in floating point. The further arithmetic (subtraction then multiplication) propagates and expands the error.

Igor's APMath operation can be used when you don't want to worry about rewriting a computation to avoid floating point issues. It's definitely not as fast as regular numerical computation but in many cases it doesn't matter. Here's an example that shows your first equation with APMath and without. I have removed the floor call in both cases so you can see the imprecision in the results better.

Function Test()
    String destStr
//  APMath destStr = floor(5 * (1 - (80 / 100) ) )
    APMath destStr = (5 * (1 - (80 / 100) ) )
    Variable destVar = str2num(destStr)
    printf "%.16g\r", destVar
   
//  Variable numdestVar = floor(5 * (1 - (80 / 100) ) )
    Variable numdestVar = (5 * (1 - (80 / 100) ) )
    printf "%.16g\r", numdestVar
End

 

In reply to by aclight

Thank you Adam.

The APMath option had not occurred to me.

In my particular application I will never be dealing with numbers beyond 3 to 4 decimal places, so using somesmallnumber as something like 1E-14 deals with the issue and I cannot envisage it ever causing errors. Incidentally, I only came across this when writing unit tests for relatively extreme edge cases, so I am confident the 'fire' will not cause harm.

Regards, Kurt