Scraping data dynamically generated by JavaScript in html document using C#

by

When you make the WebRequest you’re asking the server to give you the page file, this file’s content hasn’t yet been parsed/executed by a web browser and so the javascript on it hasn’t yet done anything.

You need to use a tool to execute the JavaScript on the page if you want to see what the page looks like after being parsed by a browser. One option you have is using the built in .net web browser control: http://msdn.microsoft.com/en-au/library/aa752040(v=vs.85).aspx

The web browser control can navigate to and load the page and then you can query it’s DOM which will have been altered by the JavaScript on the page.

EDIT (example):

Uri uri = new Uri("http://www.somewebsite.com/somepage.htm");

webBrowserControl.AllowNavigation = true;
// optional but I use this because it stops javascript errors breaking your scraper
webBrowserControl.ScriptErrorsSuppressed = true;
// you want to start scraping after the document is finished loading so do it in the function you pass to this handler
webBrowserControl.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(webBrowserControl_DocumentCompleted);
webBrowserControl.Navigate(uri);

private void webBrowserControl_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
    HtmlElementCollection divs = webBrowserControl.Document.GetElementsByTagName("div");

    foreach (HtmlElement div in divs)
    {
        //do something
    }
}

More Related Contents:

  1. how to dynamically generate HTML code using .NET’s WebBrowser or mshtml.HTMLDocument?
  2. Download file from an ASP.NET Web API method using AngularJS
  3. asp.net: Invalid postback or callback argument
  4. Scraping webpage generated by JavaScript with C#
  5. What is the best way to parse html in C#? [closed]
  6. Changing website favicon dynamically
  7. Dynamically set value of a file input [duplicate]
  8. How do I remove all HTML tags from a string without knowing which tags are in it?
  9. Generating HTML email body in C#
  10. Does C# have an equivalent to JavaScript’s encodeURIComponent()?
  11. jQuery: how to change tag name?
  12. Can itextsharp.xmlworker render embedded images?
  13. ASP.NET MVC get textbox input value
  14. DateTime to javascript date
  15. Using WebClient in C# is there a way to get the URL of a site after being redirected?
  16. Is there a standard way to encode a .NET string into JavaScript string for use in MS Ajax?
  17. C# httpwebrequest and javascript
  18. Incorrect List Model Binding indices when using HTML Helpers
  19. Can we create custom HTTP Status codes?
  20. How to make the .net HttpClient use http 2.0?
  21. Is it possible to use HTML’s .querySelector() to select by xlink attribute in an SVG?
  22. How to append text to a div element?
  23. Capturing webpage as image in c#, ensuring javascript rendered elements are visible
  24. Asynchronous Controller is blocking requests in ASP.NET MVC through jQuery
  25. send byte array by HTTP POST in store app
  26. Accessing the document object of a frame with JavaScript
  27. Opening a URL in a new tab
  28. Asp .NET Button – OnClientClick=”return function()” vs OnClientClick=”function()”
  29. ASP.Net double-click problem
  30. Can we consume a Blazor component as a Web Component within a regular non-Blazor HTML page?

Leave a Comment