Is the most significant decimal digits precision that can be converted to binary and back to decimal without loss of significance 6 or 7.225?

by

These are talking about two slightly different things.

The 7.2251 digits is the precision with which a number can be stored internally. For one example, if you did a computation with a double precision number (so you were starting with something like 15 digits of precision), then rounded it to a single precision number, the precision you’d have left at that point would be approximately 7 digits.

The 6 digits is talking about the precision that can be maintained through a round-trip conversion from a string of decimal digits, into a floating point number, then back to another string of decimal digits.

So, let’s assume I start with a number like 1.23456789 as a string, then convert that to a float32, then convert the result back to a string. When I’ve done this, I can expect 6 digits to match exactly. The seventh digit might be rounded though, so I can’t necessarily expect it to match (though it probably will be +/- 1 of the original string.

For example, consider the following code:

#include <iostream>
#include <iomanip>

int main() {
    double init = 987.23456789;
    for (int i = 0; i < 100; i++) {
        float f = init + i / 100.0;
        std::cout << std::setprecision(10) << std::setw(20) << f;
    }
}

This produces a table like the following:

     987.2345581         987.2445679         987.2545776         987.2645874
     987.2745972         987.2845459         987.2945557         987.3045654
     987.3145752          987.324585         987.3345947         987.3445435
     987.3545532          987.364563         987.3745728         987.3845825
     987.3945923          987.404541         987.4145508         987.4245605
     987.4345703         987.4445801         987.4545898         987.4645386
     987.4745483         987.4845581         987.4945679         987.5045776
     987.5145874         987.5245972         987.5345459         987.5445557
     987.5545654         987.5645752          987.574585         987.5845947
     987.5945435         987.6045532          987.614563         987.6245728
     987.6345825         987.6445923          987.654541         987.6645508
     987.6745605         987.6845703         987.6945801         987.7045898
     987.7145386         987.7245483         987.7345581         987.7445679
     987.7545776         987.7645874         987.7745972         987.7845459
     987.7945557         987.8045654         987.8145752          987.824585
     987.8345947         987.8445435         987.8545532          987.864563
     987.8745728         987.8845825         987.8945923          987.904541
     987.9145508         987.9245605         987.9345703         987.9445801
     987.9545898         987.9645386         987.9745483         987.9845581
     987.9945679         988.0045776         988.0145874         988.0245972
     988.0345459         988.0445557         988.0545654         988.0645752
      988.074585         988.0845947         988.0945435         988.1045532
      988.114563         988.1245728         988.1345825         988.1445923
      988.154541         988.1645508         988.1745605         988.1845703
     988.1945801         988.2045898         988.2145386         988.2245483

If we look through this, we can see that the first six significant digits always follow the pattern precisely (i.e., each result is exactly 0.01 greater than its predecessor). As we can see in the original double, the value is actually 98x.xx456–but when we convert the single-precision float to decimal, we can see that the 7th digit frequently would not be read back in correctly–since the subsequent digit is greater than 5, it should round up to 98x.xx46, but some of the values won’t (e.g,. the second to last item in the first column is 988.154541, which would be round down instead of up, so we’d end up with 98x.xx45 instead of 46. So, even though the value (as stored) is precise to 7 digits (plus a little), by the time we round-trip the value through a conversion to decimal and back, we can’t depend on that seventh digit matching precisely any more (even though there’s enough precision that it will a lot more often than not).


1. That basically means 7 digits, and the 8th digit will be a little more accurate than nothing, but not a whole lot–for example, if we were converting from a double of 1.2345678, the .225 digits of precision mean that the last digit would be with about +/- .775 of the what started out there (whereas without the .225 digits of precision, it would be basically +/- 1 of what started out there).

More Related Contents:

  1. Convert hex to float
  2. Difference between decimal, float and double in .NET?
  3. What is “2’s Complement”?
  4. How do I restrict a float value to only two places after the decimal point in C?
  5. Python JSON serialize a Decimal object
  6. How to deal with big numbers in javascript [duplicate]
  7. When should I use double instead of decimal?
  8. How To Represent 0.1 In Floating Point Arithmetic And Decimal
  9. How to get numbers after decimal point?
  10. Binary representation of float in Python (bits not hex)
  11. How to convert a Decimal to a Double in C#?
  12. Trouble with floats in Objective-C
  13. C++ decimal data types
  14. Get decimal portion of a number with JavaScript
  15. Which arithmetic operations are the same on unsigned and two’s complement signed numbers?
  16. Add zeros to a float after the decimal point in Python
  17. Float vs Decimal in ActiveRecord
  18. Rounding a number in Python but keeping ending zeros
  19. Turn float into string
  20. Is there a tool to know whether a value has an exact binary representation as a floating point variable?
  21. Convert exponential number to decimal in php
  22. How to convert Decimal to Double in C#?
  23. How to detect type of compression used on the file? (if no file extension is specified)
  24. Adding binary numbers
  25. Decimal place issues with floats and decimal.Decimal
  26. Convert decimal to binary in python [duplicate]
  27. Is there a reliable way in JavaScript to obtain the number of decimal places of an arbitrary number?
  28. Decimal to Binary
  29. How to perform round to even with floating point numbers
  30. Rounding to n significant digits

Leave a Comment