Nokogiri, open-uri, and Unicode Characters

by

Summary: When feeding UTF-8 to Nokogiri through open-uri, use open(...).read and pass the resulting string to Nokogiri.

Analysis:
If I fetch the page using curl, the headers properly show Content-Type: text/html; charset=UTF-8 and the file content includes valid UTF-8, e.g. "Genealogía de Jesucristo". But even with a magic comment on the Ruby file and setting the doc encoding, it’s no good:

# encoding: UTF-8
require 'nokogiri'
require 'open-uri'

doc = Nokogiri::HTML(open('http://www.biblegateway.com/passage/?search=Mateo1-2&version=NVI'))
doc.encoding = 'utf-8'
h52 = doc.css('h5')[1]
puts h52.text, h52.text.encoding
#=> Genealogà a de Jesucristo
#=> UTF-8

We can see that this is not the fault of open-uri:

html = open('http://www.biblegateway.com/passage/?search=Mateo1-2&version=NVI')
gene = html.read[/Gene\S+/]
puts gene, gene.encoding
#=> Genealogía
#=> UTF-8

This is a Nokogiri issue when dealing with open-uri, it seems. This can be worked around by passing the HTML as a raw string to Nokogiri:

# encoding: UTF-8
require 'nokogiri'
require 'open-uri'

html = open('http://www.biblegateway.com/passage/?search=Mateo1-2&version=NVI')
doc = Nokogiri::HTML(html.read)
doc.encoding = 'utf-8'
h52 = doc.css('h5')[1].text
puts h52, h52.encoding, h52 == "Genealogía de Jesucristo"
#=> Genealogía de Jesucristo
#=> UTF-8
#=> true

More Related Contents:

  1. Why does installing Nokogiri on Mac OS fail with libiconv is missing?
  2. Nokogiri/Xpath namespace query
  3. Mac user and getting WARNING: Nokogiri was built against LibXML version 2.7.8, but has dynamically loaded 2.7.3
  4. How do I remove emoji from string
  5. How to get rid of non-ascii characters in ruby
  6. Installing Nokogiri on OSX 10.10 Yosemite
  7. How to avoid tripping over UTF-8 BOM when reading files
  8. Error to install Nokogiri on OSX 10.9 Maverick?
  9. Ruby 1.9: how can I properly upcase & downcase multibyte strings?
  10. Convert unicode codepoint to string character in Ruby
  11. Print an XML document without the XML header line at the top
  12. How to select nodes by matching text
  13. How can I get the absolute URL when extracting links using Nokogiri?
  14. How to determine if a character is a Chinese character
  15. Is this the best way to unescape unicode escape sequences in Ruby?
  16. extract single string from HTML using Ruby/Mechanize (and Nokogiri)
  17. XPath axis, get all following nodes until
  18. Compute percentage of cents total
  19. What does ||= (or-equals) mean in Ruby?
  20. Why use Ruby’s attr_accessor, attr_reader and attr_writer?
  21. Why doesn’t ruby support method overloading?
  22. In Ruby, what’s the relationship between ‘new’ and ‘initialize’? How to return nil while initializing?
  23. Why am I getting objects printed twice?
  24. How do I remove leading whitespace chars from Ruby HEREDOC?
  25. Convert to/from DateTime and Time in Ruby
  26. In Ruby, how does coerce() actually work?
  27. How to check if a Ruby object is a Boolean
  28. Invalid function in ruby
  29. When do Ruby instance variables get set?
  30. Rails ‘parse_query’ error on server in brand new app

Leave a Comment