# Rounding error - using floor I have just come across this somewhat odd inconsistency:

print( floor(5 * (1 - (80 / 100) ) ) )
0
print( floor(5 * (100 - 80) / 100 ) )
1

My guess is that this is due to floating point arithmetic, or something similar?

Not a big deal as I am (now!) using the second expression.

My understanding is that floating point numbers are described as sums of 2^2,2^1,2^0, 2^-1, 2^-2, etc....

That means 1 is perfectly described (2^0), so is 0.5 (2^-1) and 0.25 (2^-2), but 0.8 and 0.2 are not perfectly described.

Buttom line. You should not trust either of the two equations you listed to return a perfect integer.

Maybe someone more knowledgeable than me can correct me.

I have looked into this more - I'm pretty sure that it is due to floating point representation. Floor (and ceil) are functions where in some circumstances the tiny imprecision of floating point can matter (0.999999999 vs 1.000000001 for example, albeit to the appropriate number of decimal places).
I have tested and found the same odd behavior occurs in both R and Python, so perhaps it is embedded in the maths engine that all these systems use?

https://en.wikipedia.org/wiki/Floating-point_arithmetic

You can use tricks like this: print floor(somenumber + somesmallnumber). But do it very carefully; select values of somesmallnumber very carefully!

While John's suggestion of floor(somenumber + somesmallnumber) might work, it's like playing with fire.

A better solution is to rewrite your equation so that you are working with integer values as much as you can. That's exactly what the second equation in the original question does. In the first equation, 80/100 is evaluated first, and that is a non-integral value which cannot be precisely represented in floating point. The further arithmetic (subtraction then multiplication) propagates and expands the error.

Igor's APMath operation can be used when you don't want to worry about rewriting a computation to avoid floating point issues. It's definitely not as fast as regular numerical computation but in many cases it doesn't matter. Here's an example that shows your first equation with APMath and without. I have removed the floor call in both cases so you can see the imprecision in the results better.

Function Test()
String destStr
//  APMath destStr = floor(5 * (1 - (80 / 100) ) )
APMath destStr = (5 * (1 - (80 / 100) ) )
Variable destVar = str2num(destStr)
printf "%.16g\r", destVar

//  Variable numdestVar = floor(5 * (1 - (80 / 100) ) )
Variable numdestVar = (5 * (1 - (80 / 100) ) )
printf "%.16g\r", numdestVar
End