Pulling data from a webpage, parsing it for specific pieces, and displaying it

by

This small example uses HtmlAgilityPack, and using XPath selectors to get to the desired elements.

protected void Page_Load(object sender, EventArgs e)
{
    string url = "http://www.metacritic.com/game/pc/halo-spartan-assault";
    var web = new HtmlAgilityPack.HtmlWeb();
    HtmlDocument doc = web.Load(url);

    string metascore = doc.DocumentNode.SelectNodes("//*[@id=\"main\"]/div[3]/div/div[2]/div[1]/div[1]/div/div/div[2]/a/span[1]")[0].InnerText;
    string userscore = doc.DocumentNode.SelectNodes("//*[@id=\"main\"]/div[3]/div/div[2]/div[1]/div[2]/div[1]/div/div[2]/a/span[1]")[0].InnerText;
    string summary = doc.DocumentNode.SelectNodes("//*[@id=\"main\"]/div[3]/div/div[2]/div[2]/div[1]/ul/li/span[2]/span/span[1]")[0].InnerText;
}

An easy way to obtain the XPath for a given element is by using your web browser (I use Chrome) Developer Tools:

  • Open the Developer Tools (F12 or Ctrl + Shift + C on Windows or Command + Shift + C for Mac).
  • Select the element in the page that you want the XPath for.
  • Right click the element in the “Elements” tab.
  • Click on “Copy as XPath”.

You can paste it exactly like that in c# (as shown in my code), but make sure to escape the quotes.

You have to make sure you use some error handling techniques because Web scraping can cause errors if they change the HTML formatting of the page.

Edit

Per @knocte’s suggestion, here is the link to the Nuget package for HTMLAgilityPack:

https://www.nuget.org/packages/HtmlAgilityPack/

More Related Contents:

  1. Parse JSON in C#
  2. Convert JSON String to JSON Object c#
  3. How to Format a date pass as a parameter to a function and return data is a string with “yyyy/mm/dd” format?
  4. Memory leak in Data Access layer using enterprise library
  5. How to use __doPostBack()
  6. Can’t get Czech characters while generating a PDF
  7. How to check a input IP fall in a specific IP range
  8. What is the difference between PreserveReferencesHandling and ReferenceLoopHandling in Json.Net?
  9. HttpContext.Current.Session is null when routing requests
  10. Proper JSON serialization in MVC 4
  11. Foreach loop, determine which is the last iteration of the loop
  12. Create thumbnail image
  13. Encrypt cookies in ASP.NET
  14. How to recognize if a string contains unicode chars?
  15. What happens in BeginProcessRequest()?
  16. Can’t get sql server compact 3.5 / 4 to work with ASP .NET MVC 2
  17. IIS Express Immediately shutting-down running site after stopping web application
  18. ASP.Net Core MVC – Client-side validation for custom attribute
  19. Fancybox – ASP.NET button not working
  20. Opening password-protected pdf file with iTextSharp
  21. Comparing Object properties using reflection
  22. How to update querystring in C#?
  23. Read a Registry Key
  24. Find a control on a page with a master page
  25. Get a list of all tree nodes (in all levels) in TreeView Controls
  26. Text on an Image button in c# asp.net 3.5
  27. Are there any multipart/form-data parser in C# – (NO ASP)
  28. Specify Date format in MVC5 (dd/MM/yyyy)
  29. Can’t find how to use HttpContent
  30. Amazon SES Email address is not verified

Leave a Comment