How to avoid joining all text from Nodes when scraping

by

This is an easily solved problem that results from not reading the documentation about how text behaves when used on a NodeSet versus a Node (or Element).

The NodeSet documentation says text will:

Get the inner text of all contained Node objects

Which is what we’re seeing happen with:

doc = Nokogiri::HTML(<<EOT)
<html>
  <body>
    <p>foo</p>
    <p>bar</p>
    <p>baz</p>
  </body>
</html>
EOT

doc.search('p').text # => "foobarbaz"

because:

doc.search('p').class # => Nokogiri::XML::NodeSet

Instead, we want to get each Node and extract its text:

doc.search('p').first.class # => Nokogiri::XML::Element
doc.search('p').first.text # => "foo"

which can be done using map:

doc.search('p').map { |node| node.text } # => ["foo", "bar", "baz"]

Ruby allows us to write that more concisely using:

doc.search('p').map(&:text) # => ["foo", "bar", "baz"]

The same things apply whether we’re working with HTML or XML, as HTML is a more relaxed version of XML.

A Node has several aliased methods for getting at its embedded text. From the documentation:

#content ⇒ Object

Also known as: text, inner_text

Returns the contents for this Node.

More Related Contents:

  1. Scraping html tables into R data frames using the XML package
  2. HTML-parser on Node.js [closed]
  3. How do I pretty-print HTML with Nokogiri?
  4. How do I parse an HTML table with Nokogiri?
  5. Line Break in XML? [duplicate]
  6. Why are bullets not appearing on posts? [closed]
  7. What to do Regular expression pattern doesn’t match anywhere in string?
  8. Testing text() nodes vs string values in XPath
  9. Create HTML Table with SQL FOR XML
  10. How do I escape ampersands in XML so they are rendered as entities in HTML?
  11. How do I encode/decode HTML entities in Ruby?
  12. What is the best practice to parse html in swift?
  13. XPath to match @class value and element value?
  14. Convert a Nokogiri document to a Ruby Hash
  15. Converting HTML to XML
  16. How to HTML encode/escape a string? Is there a built-in?
  17. Space Before Closing Slash?
  18. White space inside XML/HTML tags
  19. How to select first element via XPath?
  20. How do I export html table data as .csv file?
  21. Valid content-type for XML, HTML and XHTML documents
  22. Is it possible to have HTML text or CDATA inside an XML attribute?
  23. Is it possible to insert HTML content in XML document?
  24. HtmlAgilityPack Drops Option End Tags
  25. How to handle xml/html in git feature branch workflow?
  26. Print an XML document without the XML header line at the top
  27. Convert html to plain text in VBA
  28. How can you parse HTML in android?
  29. How to get the HTML source of a webpage in Ruby [duplicate]
  30. Parse html using C

Leave a Comment