I need a Powerful Web Scraper library [closed]

by

Scraping is easy really, you just have to parse the content you are downloading and get all the associated links.

The most important piece though is the part that processes the HTML. Because most browsers don’t require the cleanest (or standards-compliant) HTML in order to be rendered, you need an HTML parser that is going to be able to make sense of HTML that is not always well-formed.

I recommend you use the HTML Agility Pack for this purpose. It does very well at handling non-well-formed HTML, and provides an easy interface for you to use XPath queries to get nodes in the resulting document.

Beyond that, you just need to pick a data store to hold your processed data (you can use any database technology for that) and a way to download content from the web, which .NET provides two high-level mechanisms for, the WebClient and HttpWebRequest/HttpWebResponse classes.

More Related Contents:

  1. Headless browser for C# (.NET)? [closed]
  2. What is a method group in C#?
  3. Get current folder path
  4. Reading 64bit Registry from a 32bit application
  5. Restarting (Recycling) an Application Pool
  6. How to check whether a string is a valid HTTP URL?
  7. In C#, how to check if a TCP port is available?
  8. Raise an event whenever a property’s value changed?
  9. Extension methods must be defined in a non-generic static class
  10. How to call .NET methods from Excel VBA?
  11. Exact time measurement for performance testing [duplicate]
  12. Printing from a .NET Service [closed]
  13. Does using parameterized SqlCommand make my program immune to SQL injection?
  14. Global variables in c#.net
  15. How to display a Windows Form in full screen on top of the taskbar? [duplicate]
  16. How is GetHashCode() implemented for Int32?
  17. Using C# 6 features with CodeDomProvider (Roslyn)
  18. System.Threading.Timer not firing after some time
  19. How can I get the name of a variable passed into a function?
  20. Custom xmlWriter to skip a certain element?
  21. How to edit .csproj file
  22. Remove Item in Dictionary based on Value
  23. Get affected rows on ExecuteNonQuery
  24. OraOLEDB.Oracle provider is not registered on the local machine
  25. Service Reference Error: Failed to generate code for the service reference
  26. How to edit CSS style of a div using C# in .NET
  27. click on a button in another application from my C# application?
  28. How to detect when laptop power cable has been disconnected?
  29. Get Length of Data Available in NetworkStream
  30. Custom Color Palette in Visual Studio Color Property Editor

Leave a Comment