Bash – Regex for HTML contents

by

Don’t parse XML/HTML with regex, use a proper XML/HTML parser.

theory :

According to the compiling theory, HTML can’t be parsed using regex based on finite state machine. Due to hierarchical construction of HTML you need to use a pushdown automaton and manipulate LALR grammar using tool like YACC.

realLife©®™ everyday tool in a shell :

You can use one of the following :

xmllint often installed by default with libxml2, xpath1

xmlstarlet can edit, select, transform… Not installed by default, xpath1

xpath installed via perl’s module XML::XPath, xpath1

xidel xpath3

saxon-lint my own project, wrapper over @Michael Kay’s Saxon-HE Java library, xpath3

or you can use high level languages and proper libs, I think of :

python‘s lxml (from lxml import etree)

perl‘s XML::LibXML, XML::XPath, XML::Twig::XPath, HTML::TreeBuilder::XPath

php‘s DOMXpath

Check: Using regular expressions with HTML tags

Example using xidel:

xidel -s "$currentURL" -e '//a/extract(@href,"id=(\d+)",1)'

More Related Contents:

  1. Awk – replace coumn 2 in table 1 from coumn 2 in table 2 based on matching data in column 1 (common between tables)
  2. bash regex with quotes?
  3. Find and kill a process in one line using bash and regex
  4. RE error: illegal byte sequence on Mac OS X
  5. Bash Regular Expression — Can’t seem to match any of \s \S \d \D \w \W etc
  6. How do I use regular expressions in bash scripts?
  7. Using the star sign in grep
  8. How to match digits in regex in bash script
  9. Regular expressions in a Bash case statement
  10. Regex matching in a Bash if statement
  11. Extract substring using regexp in plain bash
  12. Bash, grep between two lines with specified string
  13. Search and replace in bash using regular expressions
  14. Why is sed not recognizing \t as a tab?
  15. What does ‘\K’ mean in this regex?
  16. Insert linefeed in sed (Mac OS X)
  17. Does bash support word boundary regular expressions?
  18. How do I use a regex in a shell script?
  19. Remove duplicates from text file based on second text file
  20. How to match a pattern given in a variable in awk?
  21. replace a unknown string between two known strings with sed
  22. Recursively change file extensions in Bash
  23. How can I match a string with a regex in Bash?
  24. Why does BASH_REMATCH not work for a quoted regular expression?
  25. Rename files using a regex with bash [duplicate]
  26. How can I match spaces with a regexp in Bash?
  27. How to change date format in sed?
  28. How to execute a Bash command only if a Docker container with a given name does not exist?
  29. Grep and regex – why am I escaping curly braces?
  30. How to extract domain name from url?

Leave a Comment