How can I get the absolute URL when extracting links using Nokogiri?

by

Nokogiri is unrelated, other than the fact that it gives you the link anchor to begin with. Use Ruby’s URI library to manage paths:

absolute_uri = URI.join( page_url, href ).to_s

Seen in action:

require 'uri'

# The URL of the page with the links
page_url="http://foo.com/zee/zaw/zoom.html"

# A variety of links to test.
hrefs = %w[
  http://zork.com/             http://zork.com/#id
  http://zork.com/bar          http://zork.com/bar#id
  http://zork.com/bar/         http://zork.com/bar/#id
  http://zork.com/bar/jim.html http://zork.com/bar/jim.html#id
  /bar                         /bar#id
  /bar/                        /bar/#id
  /bar/jim.html                /bar/jim.html#id
  jim.html                     jim.html#id
  ../jim.html                  ../jim.html#id
  ../                          ../#id
  #id
]

hrefs.each do |href|
  root_href = URI.join(page_url,href).to_s
  puts "%-32s -> %s" % [ href, root_href ]
end
#=> http://zork.com/                 -> http://zork.com/
#=> http://zork.com/#id              -> http://zork.com/#id
#=> http://zork.com/bar              -> http://zork.com/bar
#=> http://zork.com/bar#id           -> http://zork.com/bar#id
#=> http://zork.com/bar/             -> http://zork.com/bar/
#=> http://zork.com/bar/#id          -> http://zork.com/bar/#id
#=> http://zork.com/bar/jim.html     -> http://zork.com/bar/jim.html
#=> http://zork.com/bar/jim.html#id  -> http://zork.com/bar/jim.html#id
#=> /bar                             -> http://foo.com/bar
#=> /bar#id                          -> http://foo.com/bar#id
#=> /bar/                            -> http://foo.com/bar/
#=> /bar/#id                         -> http://foo.com/bar/#id
#=> /bar/jim.html                    -> http://foo.com/bar/jim.html
#=> /bar/jim.html#id                 -> http://foo.com/bar/jim.html#id
#=> jim.html                         -> http://foo.com/zee/zaw/jim.html
#=> jim.html#id                      -> http://foo.com/zee/zaw/jim.html#id
#=> ../jim.html                      -> http://foo.com/zee/jim.html
#=> ../jim.html#id                   -> http://foo.com/zee/jim.html#id
#=> ../                              -> http://foo.com/zee/
#=> ../#id                           -> http://foo.com/zee/#id
#=> #id                              -> http://foo.com/zee/zaw/zoom.html#id

The more convoluted answer here previously used URI.parse(root).merge(URI.parse(href)).to_s.
Thanks to @pguardiario for the improvement.

More Related Contents:

  1. Why does installing Nokogiri on Mac OS fail with libiconv is missing?
  2. Nokogiri/Xpath namespace query
  3. Mac user and getting WARNING: Nokogiri was built against LibXML version 2.7.8, but has dynamically loaded 2.7.3
  4. Installing Nokogiri on OSX 10.10 Yosemite
  5. Nokogiri, open-uri, and Unicode Characters
  6. Error to install Nokogiri on OSX 10.9 Maverick?
  7. Print an XML document without the XML header line at the top
  8. How to select nodes by matching text
  9. extract single string from HTML using Ruby/Mechanize (and Nokogiri)
  10. XPath axis, get all following nodes until
  11. Write regular expressions [closed]
  12. Can you supply arguments to the map(&:method) syntax in Ruby?
  13. Ruby block and unparenthesized arguments
  14. Why can I refer to a variable outside of an if/unless/case statement that never ran?
  15. Is there an efficient way to perform hundreds of text substitutions in Ruby?
  16. diff a ruby string or array
  17. Methods in Ruby: objects or not?
  18. How does the magic comment ( # Encoding: utf-8 ) in ruby​​ work?
  19. sqlite3-ruby install error on Ubuntu
  20. Need a simple explanation of the inject method
  21. DRY Ruby Initialization with Hash Argument
  22. Exporting an Environment Variable in Ruby
  23. How do I temporarily redirect stderr in Ruby?
  24. Is there a way to remove the BOM from a UTF-8 encoded file?
  25. Using layouts in HAML files independently of Rails
  26. How to implement a short URL like the URLs in Twitter?
  27. How to run untrusted Ruby code inside a safe sandbox?
  28. Why is sum so much faster than inject(:+)?
  29. Ruby: extend self
  30. The Class/Object Paradox confusion

Leave a Comment