Scraping javascript website in R

by

So, RSelenium is not the only answer (anymore). If you can install the PhantomJS binary (grab phantomjs binaries from here: http://phantomjs.org/) then you can use it to render the HTML and scrape it with rvest (similar to the RSelenium approach but doesn’t require java):

library(rvest)

# render HTML from the site with phantomjs

url <- "http://www.scoreboard.com/game/rosol-l-goffin-d-2014/8drhX07d/#game-summary"

writeLines(sprintf("var page = require('webpage').create();
page.open('%s', function () {
    console.log(page.content); //page source
    phantom.exit();
});", url), con="scrape.js")

system("phantomjs scrape.js > scrape.html", intern = T)

# extract the content you need
pg <- html("scrape.html")
pg %>% html_nodes("#utime") %>% html_text()

## [1] "10:20 AM, October 28, 2014"

More Related Contents:

  1. Can scrapy be used to scrape dynamic content from websites that are using AJAX?
  2. scrape html generated by javascript with python
  3. Headless Browser for Python (Javascript support REQUIRED!) [closed]
  4. Executing Javascript from Python
  5. What’s a good tool to screen-scrape with Javascript support? [closed]
  6. How to scroll down with Phantomjs to load dynamic content
  7. Simple Screen Scraping using jQuery
  8. Include a javascript file in Shiny app
  9. Screen Scraping from a web page with a lot of Javascript [closed]
  10. Scrape web pages in real time with Node.js
  11. Looping over urls to do the same thing
  12. Draw a circle around nodes groups
  13. Click on a javascript link within python?
  14. R networkD3 package: node coloring in simpleNetwork()
  15. How can I use Perl to grab text from a web page that is dynamically generated with JavaScript?
  16. R Shiny DataTable selected row color
  17. Re-enabling window.alert in Chrome
  18. Passing parameter inside $location.path in Angular
  19. Extract parameter value from url using regular expressions
  20. How do you completely remove Ionic and Cordova installation from mac?
  21. Convert non-ASCII characters (umlauts, accents…) to their closest ASCII equivalent (for slug creation)
  22. express.js – how to intercept response.send() / response.json()
  23. Generate random date between two dates and times in Javascript
  24. How to hide optgroup/option elements?
  25. Javascript Regular Expression for rgb values
  26. When to access the attribute (vs the property)?
  27. Javascript capture key
  28. Passing object from injected page script to content script
  29. How to pass some parameters to function called in addEventListener? [duplicate]
  30. How to use confirm using sweet alert?

Leave a Comment