# Rounding error - using floor

KurtB

Thu, 04/29/2021 - 06:56 am

I have just come across this somewhat odd inconsistency:

•print( floor(5 * (1 - (80 / 100) ) ) )

0

•print( floor(5 * (100 - 80) / 100 ) )

1

0

•print( floor(5 * (100 - 80) / 100 ) )

1

My guess is that this is due to floating point arithmetic, or something similar?

Not a big deal as I am (now!) using the second expression.

My understanding is that floating point numbers are described as sums of 2^2,2^1,2^0, 2^-1, 2^-2, etc....

That means 1 is perfectly described (2^0), so is 0.5 (2^-1) and 0.25 (2^-2), but 0.8 and 0.2 are not perfectly described.

Buttom line. You should not trust either of the two equations you listed to return a perfect integer.

Maybe someone more knowledgeable than me can correct me.

April 29, 2021 at 08:33 am - Permalink

In reply to My understanding is that… by olelytken

I have looked into this more - I'm pretty sure that it is due to floating point representation. Floor (and ceil) are functions where in some circumstances the tiny imprecision of floating point can matter (0.999999999 vs 1.000000001 for example, albeit to the appropriate number of decimal places).

I have tested and found the same odd behavior occurs in both R and Python, so perhaps it is embedded in the maths engine that all these systems use?

April 29, 2021 at 08:43 am - Permalink

https://en.wikipedia.org/wiki/Floating-point_arithmetic

You can use tricks like this: print floor(somenumber + somesmallnumber). But do it very carefully; select values of somesmallnumber very carefully!

April 29, 2021 at 11:09 am - Permalink

Thank you John,

I like the idea of adding 'somesmallnumber', and appreciate and understand the need to choose its value carefully.

April 29, 2021 at 12:22 pm - Permalink

While John's suggestion of floor(somenumber + somesmallnumber) might work, it's like playing with fire.

A better solution is to rewrite your equation so that you are working with integer values as much as you can. That's exactly what the second equation in the original question does. In the first equation, 80/100 is evaluated first, and that is a non-integral value which cannot be precisely represented in floating point. The further arithmetic (subtraction then multiplication) propagates and expands the error.

Igor's APMath operation can be used when you don't want to worry about rewriting a computation to avoid floating point issues. It's definitely not as fast as regular numerical computation but in many cases it doesn't matter. Here's an example that shows your first equation with APMath and without. I have removed the floor call in both cases so you can see the imprecision in the results better.

String destStr

// APMath destStr = floor(5 * (1 - (80 / 100) ) )

APMath destStr = (5 * (1 - (80 / 100) ) )

Variable destVar = str2num(destStr)

printf "%.16g\r", destVar

// Variable numdestVar = floor(5 * (1 - (80 / 100) ) )

Variable numdestVar = (5 * (1 - (80 / 100) ) )

printf "%.16g\r", numdestVar

End

April 30, 2021 at 06:15 am - Permalink

In reply to While John's suggestion of… by aclight

Thank you Adam.

The APMath option had not occurred to me.

In my particular application I will never be dealing with numbers beyond 3 to 4 decimal places, so using somesmallnumber as something like 1E-14 deals with the issue and I cannot envisage it ever causing errors. Incidentally, I only came across this when writing unit tests for relatively extreme edge cases, so I am confident the 'fire' will not cause harm.

Regards, Kurt

April 30, 2021 at 06:34 am - Permalink