How to programmatically log in to a website to screenscape?

by

You’d make the request as though you’d just filled out the form. Assuming it’s POST for example, you make a POST request with the correct data. Now if you can’t login directly to the same page you want to scrape, you will have to track whatever cookies are set after your login request, and include them in your scraping request to allow you to stay logged in.

It might look like:

HttpWebRequest http = WebRequest.Create(url) as HttpWebRequest;
http.KeepAlive = true;
http.Method = "POST";
http.ContentType = "application/x-www-form-urlencoded";
string postData="FormNameForUserId=" + strUserId + "&FormNameForPassword=" + strPassword;
byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData);
http.ContentLength = dataBytes.Length;
using (Stream postStream = http.GetRequestStream())
{
    postStream.Write(dataBytes, 0, dataBytes.Length);
}
HttpWebResponse httpResponse = http.GetResponse() as HttpWebResponse;
// Probably want to inspect the http.Headers here first
http = WebRequest.Create(url2) as HttpWebRequest;
http.CookieContainer = new CookieContainer();
http.CookieContainer.Add(httpResponse.Cookies);
HttpWebResponse httpResponse2 = http.GetResponse() as HttpWebResponse;

Maybe.

More Related Contents:

  1. Using WebClient or WebRequest to login to a website and access data
  2. How can I parse this in C# [duplicate]
  3. How do I prevent the app from terminating when I close the startup form?
  4. ASP.NET MVC Conditional validation
  5. HttpWebRequest using Basic authentication
  6. Reading file input from a multipart/form-data POST
  7. Reset the value of textarea after form submission
  8. How do I detect keyPress while not focused?
  9. Does page reload ever cause post?
  10. How to submit http form using C#
  11. ASP.NET Core 2.0 LDAP Active Directory Authentication
  12. How to access HTML form input from ASP.NET code behind
  13. Error making Azure Management Library API call when authenticating with azure active directory
  14. C# – HttpWebRequest POST (Login to Facebook)
  15. Unlock Windows programmatically
  16. how to generate a unique token which expires after 24 hours?
  17. How to implement custom authentication in ASP.NET MVC 5
  18. ASP.NET Identity Cookie across subdomains
  19. Getting NetworkCredential for current user (C#)
  20. How to get the logon SID in C#
  21. Simple JWT authentication in ASP.NET Core 1.0 Web API
  22. The HTTP request is unauthorized with client authentication scheme ‘Negotiate’. The authentication header received from the server was ‘NTLM’
  23. Closing a form and then call another one
  24. SendKey.Send() Not working
  25. Prevent from displaying of Windows Security window
  26. WS_EX_TRANSPARENT – What does it actually do?
  27. Built-in helper to parse User.Identity.Name into Domain\Username
  28. .NET 4.5 Bug in UserPrincipal.FindByIdentity (System.DirectoryServices.AccountManagement)
  29. I need a Powerful Web Scraper library [closed]
  30. How to authenticate user with Azure Active Directory using OAuth 2.0?

Leave a Comment