ruby 1.9: invalid byte sequence in UTF-8

by

In Ruby 1.9.3 it is possible to use String.encode to “ignore” the invalid UTF-8 sequences. Here is a snippet that will work both in 1.8 (iconv) and 1.9 (String#encode) :

require 'iconv' unless String.method_defined?(:encode)
if String.method_defined?(:encode)
  file_contents.encode!('UTF-8', 'UTF-8', :invalid => :replace)
else
  ic = Iconv.new('UTF-8', 'UTF-8//IGNORE')
  file_contents = ic.iconv(file_contents)
end

or if you have really troublesome input you can do a double conversion from UTF-8 to UTF-16 and back to UTF-8:

require 'iconv' unless String.method_defined?(:encode)
if String.method_defined?(:encode)
  file_contents.encode!('UTF-16', 'UTF-8', :invalid => :replace, :replace => '')
  file_contents.encode!('UTF-8', 'UTF-16')
else
  ic = Iconv.new('UTF-8', 'UTF-8//IGNORE')
  file_contents = ic.iconv(file_contents)
end

More Related Contents:

  1. How does the magic comment ( # Encoding: utf-8 ) in ruby​​ work?
  2. Set global default encoding for ruby 1.9
  3. Ruby 1.9: how can I properly upcase & downcase multibyte strings?
  4. Convert unicode codepoint to string character in Ruby
  5. How to determine if a character is a Chinese character
  6. Reassigning the value of variables in Ruby?
  7. What is attr_accessor in Ruby?
  8. Using PowerShell to write a file in UTF-8 without the BOM
  9. What is the difference between UTF-8 and Unicode?
  10. What is the difference between include and require in Ruby?
  11. What is the colon operator in Ruby?
  12. Get the name of the currently executing method
  13. What is this &:last Ruby construct called? [duplicate]
  14. Get list of a class’ instance methods
  15. Why are my basic Heroku apps taking two seconds to load?
  16. Comparing ruby hashes [duplicate]
  17. How do I write out a text file in C# with a code page other than UTF-8?
  18. Learning Insertion Sort in Ruby
  19. Can’t install RMagick 2.13.1. Can’t find MagickWand.h.
  20. What is the difference between STDIN and $stdin in Ruby?
  21. Passing a method as a parameter in Ruby
  22. Python – dealing with mixed-encoding files
  23. What is the difference between Raising Exceptions vs Throwing Exceptions in Ruby?
  24. List of installed gems?
  25. How can I find the number of days between two Date objects in Ruby?
  26. What are those pipe symbols for in Ruby?
  27. Ruby’s yield feature in relation to computer science
  28. How to convert JSON to a Ruby hash
  29. Why isn’t current directory on my Ruby path? [duplicate]
  30. Iterate over Ruby Time object with delta

Leave a Comment