jsoup Another html parser I really liked using was jsoup. You could get all the <p> elements in 2 lines of code. Document doc = Jsoup.connect(“http://en.wikipedia.org/”).get(); Elements ps = doc.select(“p”); Then write it out to a file in one more line out.write(ps.text()); //it will append all of the p elements together in one long string … Read more
Try this: from bs4 import BeautifulSoup from bs4.element import Comment import urllib.request def tag_visible(element): if element.parent.name in [‘style’, ‘script’, ‘head’, ‘title’, ‘meta’, ‘[document]’]: return False if isinstance(element, Comment): return False return True def text_from_html(body): soup = BeautifulSoup(body, ‘html.parser’) texts = soup.findAll(text=True) visible_texts = filter(tag_visible, texts) return u” “.join(t.strip() for t in visible_texts) html = urllib.request.urlopen(‘http://www.nytimes.com/2009/12/21/us/21storm.html’).read() … Read more
from BeautifulSoup import BeautifulSoup import re html_text = “”” <h2>this is cool #12345678901</h2> <h2>this is nothing</h2> <h1>foo #126666678901</h1> <h2>this is interesting #126666678901</h2> <h2>this is blah #124445678901</h2> “”” soup = BeautifulSoup(html_text) for elem in soup(text=re.compile(r’ #\S{11}’)): print elem.parent Prints: <h2>this is cool #12345678901</h2> <h2>this is interesting #126666678901</h2> <h2>this is blah #124445678901</h2>
NekoHTML, TagSoup, and JTidy will allow you to parse HTML and then process with XML tools, like XPath.
Remove javascript and CSS: <(script|style).*?</\1> Remove tags <.*?>
‘add prog ref too: Microsoft.mshtml ‘then on the page: Imports mshtml Function parseMyHtml(ByVal htmlToParse$) As String Dim htmlDocument As IHTMLDocument2 = New HTMLDocumentClass() htmlDocument.write(htmlToParse) htmlDocument.close() Dim allElements As IHTMLElementCollection = htmlDocument.body.all Dim allInputs As IHTMLElementCollection = allElements.tags(“a”) Dim element As IHTMLElement For Each element In allInputs element.title = element.innerText Next Return htmlDocument.body.innerHTML End Function As … Read more
I found using hpple quite useful to parse messy HTML. Hpple project is a Objective-C wrapper on the XPathQuery library for parsing HTML. Using it you can send an XPath query and receive the result . Requirements: -Add libxml2 includes to your project Menu Project->Edit Project Settings Search for setting “Header Search Paths” Add a … Read more
Use ( ) in regexp and group(1) in python to retrieve the captured string (re.search will return None if it doesn’t find the result, so don’t use group() directly): title_search = re.search(‘<title>(.*)</title>’, html, re.IGNORECASE) if title_search: title = title_search.group(1)
Try this: from bs4 import BeautifulSoup from bs4.element import Comment import urllib.request def tag_visible(element): if element.parent.name in [‘style’, ‘script’, ‘head’, ‘title’, ‘meta’, ‘[document]’]: return False if isinstance(element, Comment): return False return True def text_from_html(body): soup = BeautifulSoup(body, ‘html.parser’) texts = soup.findAll(text=True) visible_texts = filter(tag_visible, texts) return u” “.join(t.strip() for t in visible_texts) html = urllib.request.urlopen(‘http://www.nytimes.com/2009/12/21/us/21storm.html’).read() … Read more
Html Agility Pack This is an agile HTML parser that builds a read/write DOM and supports plain XPATH or XSLT (you actually don’t HAVE to understand XPATH nor XSLT to use it, don’t worry…). It is a .NET code library that allows you to parse “out of the web” HTML files. The parser is very … Read more