Spider a Website and Return URLs Only

by

The absolute last thing I want to do is download and parse all of the content myself (i.e. create my own spider). Once I learned that Wget writes to stderr by default, I was able to redirect it to stdout and filter the output appropriately.

wget --spider --force-html -r -l2 $url 2>&1 \
  | grep '^--' | awk '{ print $3 }' \
  | grep -v '\.\(css\|js\|png\|gif\|jpg\)$' \
  > urls.m3u

This gives me a list of the content resource (resources that aren’t images, CSS or JS source files) URIs that are spidered. From there, I can send the URIs off to a third party tool for processing to meet my needs.

The output still needs to be streamlined slightly (it produces duplicates as it’s shown above), but it’s almost there and I haven’t had to do any parsing myself.

More Related Contents:

  1. Can grep show only words that match search pattern?
  2. Count all occurrences of a string in lots of files with grep
  3. How can I grep recursively, but only in files with certain extensions?
  4. Grep only the first match and stop
  5. How can I have grep not print out ‘No such file or directory’ errors?
  6. How to grep with a list of words
  7. How to grep for two words existing on the same line? [duplicate]
  8. Ansible shell module returns error when grep results are empty
  9. Is it valid to replace http:// with // in a ?
  10. How to check whether a string is a valid HTTP URL?
  11. How can I grep for a string that begins with a dash/hyphen?
  12. Use URI builder in Android or create URL with variables
  13. grep: show lines surrounding each match
  14. How to process each output line in a loop?
  15. how to get an uri of an image resource in android
  16. Pulling data from a webpage, parsing it for specific pieces, and displaying it
  17. How to get the process ID to kill a nohup process?
  18. Detecting ‘stealth’ web-crawlers
  19. Convert String to Uri
  20. Does lookbehind work in sed?
  21. XML Namespace URI with HTTPS?
  22. How to show only next line after the matched one?
  23. How to get Uri.EscapeDataString to comply with RFC 3986
  24. How to find all files containing specific text (string) on Linux?
  25. How do I delete files programmatically on Android?
  26. How can I get the baseurl of site?
  27. Using sed and grep/egrep to search and replace
  28. Registering an Application to a URI Scheme in windows 10
  29. Locally run all of the spiders in Scrapy
  30. How to make “grep” read patterns from a file?

Leave a Comment