extract single string from HTML using Ruby/Mechanize (and Nokogiri)

by

Radek. I’m going to show you how to fish.

When you call Mechanize::Page::parser, it’s giving you the Nokogiri document. So your “xpath” and “at_xpath” calls are invoking Nokogiri. The problem is in your xpaths. In general, start out with the most general xpath you can get to work, and then narrow it down. So, for example, instead of this:

puts  post_page.parser.xpath('/html/body/div/div/div/div/div/table/tbody/tr/td/div[2]/text()').to_s.strip

start with this:

puts post_page.parser.xpath('//table').to_html

This gets the any tables, anywhere, and then prints them as html. Examine the HTML, to see what tables it brought back. It probably grabbed several when you want only one, so you’ll need to tell it how to pick out the one table you want. If, for example, you notice that the table you want has CSS class “userdata“, then try this:

puts post_page.parser.xpath("//table[@class="userdata"]").to_html

Any time you don’t get back an array, you goofed up the xpath, so fix it before proceding. Once you’re getting the table you want, then try to get the rows:

puts post_page.parser.xpath("//table[@class="userdata"]//tr").to_html

If that worked, then take off the “to_html” and you now have an array of Nokogiri nodes, each one a table row.

And that’s how you do it.

More Related Contents:

  1. How to avoid joining all text from Nodes when scraping
  2. Why does installing Nokogiri on Mac OS fail with libiconv is missing?
  3. Nokogiri/Xpath namespace query
  4. Mac user and getting WARNING: Nokogiri was built against LibXML version 2.7.8, but has dynamically loaded 2.7.3
  5. Installing Nokogiri on OSX 10.10 Yosemite
  6. Nokogiri, open-uri, and Unicode Characters
  7. Error to install Nokogiri on OSX 10.9 Maverick?
  8. Print an XML document without the XML header line at the top
  9. How to select nodes by matching text
  10. How can I get the absolute URL when extracting links using Nokogiri?
  11. XPath axis, get all following nodes until
  12. How to get the last character of a string in an array [closed]
  13. Compute percentage of cents total
  14. What does ||= (or-equals) mean in Ruby?
  15. How to remove RVM (Ruby Version Manager) from my system
  16. Why use Ruby’s attr_accessor, attr_reader and attr_writer?
  17. Can’t install Ruby under Lion with RVM – GCC issues
  18. Why doesn’t ruby support method overloading?
  19. What’s the difference between URI.escape and CGI.escape?
  20. Best way to escape and unescape strings in Ruby?
  21. Calling a Method From a String With the Method’s Name in Ruby
  22. In Ruby, what’s the relationship between ‘new’ and ‘initialize’? How to return nil while initializing?
  23. Why am I getting objects printed twice?
  24. Finding the cause of a memory leak in Ruby
  25. How do I remove leading whitespace chars from Ruby HEREDOC?
  26. Convert to/from DateTime and Time in Ruby
  27. Can you ask ruby to treat warnings as errors?
  28. What does ||= mean? [duplicate]
  29. Why are methods in Ruby documentation preceded by a hash sign?
  30. How can I intercept method call in ruby?

Leave a Comment