Floating point less-than-equal comparisons after addition and substraction

No, there is no best practice. Unfortunately, there cannot be, because almost all floating-point calculations introduce some rounding error, and the consequences of the errors are different for different applications.

Typically, software will perform some calculations that ideally would yield some exact mathematical result x but, due to rounding errors (or other issues), produce an approximation x‘. When comparing floating-point numbers, you want to ask some question about x, such as “Is x < 1?” or “Is x = 3.1415926…?” So the problem you want to solve is “How do I use x‘ to answer this question about x?”

There is no general solution for this. Some errors may produce an x‘ that is greater than 1 even though x is less than 1. Some errors may produce an x‘ that is less than 1 even though x is greater than 1. The solution in any specific instance depends on information about the errors that were generated while calculating x‘ and the specific question to be answered.

Sometimes a thorough analysis can demonstrate that certain questions about x can be answered using x‘. For example, in some situations, we might craft calculations so that we know that, if x‘ < 1, then x < 1. Or perhaps that, if x‘ < .99875, then x < 1. Say we analyze the calculations we used to calculate x‘ and can show that the final error is less than .00125. Then, if x‘ < .99875, then we know x < 1, and, if x‘ > 1.00125, then x > 1. But, if .99875 < x‘ < 1.00125, then we do not know whether x > 1 or x < 1. What do we do in that situation? Is it then better for your application to take the path where x < 1 or the path where x > 1? The answer is specific to each application, and there is no general best practice.

I will add to this that the amount of rounding error that occurs varies hugely from application to application. This is because rounding error can be compounded in various ways. Some applications with a few floating-point operations will achieve results with small errors. Some applications with many floating-point operations will also achieve results with modest errors. But certain behaviors can lead calculations astray and produce catastrophic errors. So dealing with rounding error is a custom problem for each program.

Leave a Comment