you can use selenium like below sample: from selenium import webdriver driver = webdriver.Firefox() driver.get(‘http://example.com’) element = driver.find_element_by_class_name(“yourClassName”) #or find by text or etc element.click()
You can use the XPath comment() function to select comment nodes, then reparse their contents as HTML: library(rvest) # scrape page h <- read_html(‘http://www.basketball-reference.com/teams/CHI/2015.html’) df <- h %>% html_nodes(xpath=”//comment()”) %>% # select comment nodes html_text() %>% # extract comment text paste(collapse=””) %>% # collapse to a single string read_html() %>% # reparse to HTML html_node(‘table#advanced’) … Read more
I’m going to show two ways of solving the problem. Whichever method you pick, don’t forget to read the bottom of my answer! First, I present a simple method which only works if the page uses jQuery. The second method looks slightly more complex, but will also work on pages without jQuery. The following examples … Read more
There is a much easier way to do this. I suggest using JSoup. With JSoup you can do things like Document doc = Jsoup.connect(“http://en.wikipedia.org/”).get(); Elements newsHeadlines = doc.select(“#mp-itn b a”); Or if you want the body: Elements body = doc.select(“body”); Or if you want all links: Elements links = doc.select(“body a”); You no longer need … Read more