Scraping with rvest – complete with NAs when tag is not present

by

Using the XML package parse the input with xmlTreeParse and then use xpathSApply to interate over the product_price class div nodes. For each such node the anonyous function gets the value of the div and p subnodes. The resulting character matrix m is reworked into a data frame DF and the columns are cleaned removing any character that is not a dot or digit and also removing any dot followed by a non-digit. Copnvert result to numeric. Note that no special processing for the missing p case is needed.

# input

Lines <- '<html>
<head></head>
<body>

<div class="product_price" id="product_price_186251">
  <p class="normal_encontrado">
    S/. 2,799.00
  </p>

  <div id="WC_CatalogEntryDBThumbnailDisplayJSPF_10461_div_10" class="price">
    S/. 2,299.00
  </div>    
</div>

<div class="product_price" id="product_price_232046">
  <div id="WC_CatalogEntryDBThumbnailDisplayJSPF_10461_div_10" class="price">
    S/. 4,999.00
  </div>
</div>
</body>
</html>'

# code to read input and produce a data.frame

library(XML)
doc <- xmlTreeParse(Lines, asText = TRUE, useInternalNodes = TRUE)

m <- xpathSApply(doc, "//div[@class="product_price"]", function(node) {
  list(p = xmlValue(node[["p"]]), div = xmlValue(node[["div"]])) })

DF <- as.data.frame(t(m), stringsAsFactors = FALSE) # rework into data frame
DF[] <- lapply(DF, function(x) as.numeric(gsub("[^.0-9]|[.]\\D", "", x))) # clean

The result is:

> DF
     p  div
1 2799 2299
2   NA 4999

More Related Contents:

  1. How to scrape tables inside a comment tag in html with R?
  2. Scraping a dynamic ecommerce page with infinite scroll
  3. stumped on how to scrape the data from this site (using R)
  4. R – How to make a click on webpage using rvest or rcurl
  5. Package “rvest” for web scraping https site with proxy
  6. R web scraping across multiple pages
  7. rvest Error in open.connection(x, “rb”) : Timeout was reached
  8. Inputting NA where there are missing values when scraping with rvest
  9. R: extracting “clean” UTF-8 text from a web page scraped with RCurl
  10. Submit form with no submit button in rvest
  11. How to use R to download a zipped file from a SSL page that requires cookies
  12. How do you scrape items together so you don’t lose the index?
  13. How do I scrape / automatically download PDF files from a document search web interface in R?
  14. How do I close unused connections after read_html in R
  15. How to replace a value by another within a variable in R?
  16. Relative frequencies / proportions with dplyr
  17. How to add texture to fill colors in ggplot2
  18. Cleaning up factor levels (collapsing multiple levels/labels)
  19. Split a large dataframe into a list of data frames based on common value in column
  20. Save a plot in an object
  21. ggplot, facet, piechart: placing text in the middle of pie chart slices
  22. YAML current date in rmarkdown
  23. How to parametrize function calls in dplyr 0.7?
  24. What are the differences between community detection algorithms in igraph?
  25. Find value corresponding to maximum in other column [duplicate]
  26. Split delimited single value character vector
  27. R split numeric vector at position
  28. Creating a Plot Window of a Particular Size
  29. Join two data frames in R based on closest timestamp
  30. Update/Replace Values in Dataframe with Tidyverse Join

Leave a Comment