How do you Screen Scrape? [closed]

by

Matt and Paul’s answers are correct. “Screen scraping” by parsing the HTML from a website is usually a bad idea because:

  1. Parsing HTML can be difficult, especially if it’s malformed. If you’re scraping a very, very simple page then regular expressions might work. Otherwise, use a parsing framework like the HTML Agility Pack.

  2. Websites are a moving target. You’ll need to update your code each time the source website changes their markup structure.

  3. Screen scraping doesn’t play well with Javascript. If the target website is using any sort of dynamic script to manipulate the webpage you’re going to have a very hard time scraping it. It’s easy to grab the HTTP response, it’s a lot harder to scrape what the browser displays in response to client-side script contained in that response.

If screen scraping is the only option, here are some keys to success:

  1. Make it as easy as possible to change the patterns you look for. If possible, store the patterns as text files or in a resource file somewhere. Make it very easy for other developers (or yourself in 3 months) to understand what markup you expect to find.

  2. Validate input and throw meaningful exceptions. In your parsing code, take care to make your exceptions very helpful. The target site will change on you, and when that happens you want your error messages to tell you not only what part of the code failed, but why it failed. Mention both the pattern you’re looking for AND the text you’re comparing against.

  3. Write lots of automated tests. You want it to be very easy to run your scraper in a non-destructive fashion because you will be doing a lot of iterative development to get the patterns right. Automate as much testing as you can, it will pay off in the long run.

  4. Consider a browser automation tool like Watin. If you require complex interactions with the target website it might be easier to write your scraper from the point of view of the browser itself, rather than mucking with the HTTP requests and responses by hand.

As for how to screen scrape in C#, you can either use Watin (see above) and scrape the resulting document using its DOM, or you can use the WebClient class [see MSDN or Google] to get at the raw HTTP response, including the HTML content, and then use some sort of text-based analysis to extract the data you want.

More Related Contents:

  1. Accessing X64 Static library in webservice through Wrapper dll
  2. How can I parse this in C# [duplicate]
  3. Regex for specific html tag in C# [duplicate]
  4. What is the overhead of creating a new HttpClient per call in a WebAPI client?
  5. Cannot open database “test” requested by the login. The login failed. Login failed for user ‘xyz\ASPNET’
  6. Setting HttpContext.Current.Session in a unit test
  7. Asp.net Core 2 API POST Objects are NULL?
  8. Best way to determine if two path reference to same file in C#
  9. Question mark and colon in statement. What does it mean?
  10. How to run unit tests (MSTest) in parallel?
  11. jQuery Ajax calls to web service seem to be synchronous
  12. How to add parameters into a WebRequest?
  13. How to get the colour of a pixel at X,Y using c#?
  14. Serialization breaks in .NET 4.5
  15. How to programmatically log in to a website to screenscape?
  16. How do I include subclasses in Swagger API documentation/ OpenAPI specification using Swashbuckle?
  17. Getting NetworkCredential for current user (C#)
  18. Could not load file or assembly exception
  19. Patterns for Compensating Lack of Inheritance in SOA
  20. Is it possible to specify proxy credentials in your web.config?
  21. asp.net asmx web service returning xml instead of json
  22. Best way to call a JSON WebService from a .NET Console
  23. How can I dynamically switch web service addresses in .NET without a recompile?
  24. Server did not recognize the value of HTTP Header SOAPAction
  25. WCF: Adding Nonce to UsernameToken
  26. Could not create SSL/TLS secure channel – Could the problem be a proxy server?
  27. My CustomAuthorizationPolicy.Evaluate() method never fires
  28. How to get the X509Certificate from a client request
  29. How To Get all ITEMS from Folders and Sub-folders of PublicFolders Using EWS Managed API
  30. Client configuration to consume WCF JSON web service

Leave a Comment