rvest
Scraping a dynamic ecommerce page with infinite scroll
As @nrussell suggested, you can use RSelenium to programatically scroll down the page before getting the source code. You could for example do: library(RSelenium) library(rvest) #start RSelenium checkForServer() startServer() remDr <- remoteDriver() remDr$open() #navigate to your page remDr$navigate(“http://www.linio.com.co/tecnologia/celulares-telefonia-gps/”) #scroll down 5 times, waiting for the page to load at each time for(i in 1:5){ remDr$executeScript(paste(“scroll(0,”,i*10000,”);”)) … Read more
Inputting NA where there are missing values when scraping with rvest
The simplest way is to select a node that encloses both of the nodes you want for each row, then iterate over them, pulling out both of the nodes you want at once. purrr::map_df is handy for not only iterating, but even combining the results into a nice tibble: library(rvest) library(purrr) url <- “https://channel9.msdn.com/Events/useR-international-R-User-conferences/useR-International-R-User-2017-Conference?sort=status&direction=desc&page=14” page … Read more
Scraping with rvest – complete with NAs when tag is not present
Using the XML package parse the input with xmlTreeParse and then use xpathSApply to interate over the product_price class div nodes. For each such node the anonyous function gets the value of the div and p subnodes. The resulting character matrix m is reworked into a data frame DF and the columns are cleaned removing … Read more