ruby floating point errors

by

If you do the calculations by hand in double-precision binary, which is limited to 53 significant bits, you’ll see what’s going on:

129.95 = 1.0000001111100110011001100110011001100110011001100110 x 2^7

129.95*100 = 1.1001011000010111111111111111111111111111111111111111011 x 2^13

This is 56 significant bits long, so rounded to 53 bits it’s

1.1001011000010111111111111111111111111111111111111111 x 2^13, which equals

12994.999999999998181010596454143524169921875

Now 129.95*10 = 1.01000100110111111111111111111111111111111111111111111 x 2^10

This is 54 significant bits long, so rounded to 53 bits it’s 1.01000100111 x 2^10 = 1299.5

Now 1299.5 * 10 = 1.1001011000011 x 2^13 = 12995.

More Related Contents:

  1. Why is division in Ruby returning an integer instead of decimal value?
  2. Can’t output currency in ruby with money-gem [closed]
  3. Very Basic Ruby puts and gets
  4. What does the (unary) * operator do in this Ruby code?
  5. How to sort an array in descending order in Ruby
  6. How to debug Ruby scripts [closed]
  7. Confusion with the assignment operation inside a falsy `if` block [duplicate]
  8. Why can I refer to a variable outside of an if/unless/case statement that never ran?
  9. diff a ruby string or array
  10. How to break outer cycle in Ruby?
  11. Ruby obfuscator [duplicate]
  12. Difference between DateTime and Time in Ruby
  13. What does send() do in Ruby?
  14. Adding a directory to $LOAD_PATH (Ruby)
  15. How does the magic comment ( # Encoding: utf-8 ) in ruby​​ work?
  16. Why does Ruby seem to hoist variable declarations from inside a case statement even if that code path is not executed? [duplicate]
  17. Where is Ruby’s string literal juxtaposition feature officially documented?
  18. Office 365 Rest API – Daemon week authentication
  19. Which style of Ruby string quoting do you favour?
  20. Access variables programmatically by name in Ruby
  21. Confusion with Atomic Grouping – how it differs from the Grouping in regular expression of Ruby?
  22. Extract number from string in Ruby
  23. How to use “RVM –default” on MacOSX
  24. ‘||=’ operator in Ruby
  25. How to check if a directory/file/symlink exists with one command in Ruby
  26. Split string by multiple delimiters
  27. RVM and thin, root vs. local user
  28. Ruby array subtraction without removing items more than once
  29. Transliteration in ruby
  30. How to match something with regex that is not between two special characters?

Leave a Comment