How to scrape a website that requires login first with Python

by

This works for me:

##################################### Method 1
import mechanize
import cookielib
from BeautifulSoup import BeautifulSoup
import html2text

# Browser
br = mechanize.Browser()

# Cookie Jar
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)

# Browser options
br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

br.addheaders = [('User-agent', 'Chrome')]

# The site we will navigate into, handling it's session
br.open('https://github.com/login')

# View available forms
for f in br.forms():
    print f

# Select the second (index one) form (the first form is a search query box)
br.select_form(nr=1)

# User credentials
br.form['login'] = 'mylogin'
br.form['password'] = 'mypass'

# Login
br.submit()

print(br.open('https://github.com/settings/emails').read())

You were not far off at all!

More Related Contents:

  1. How to use Python to login to a webpage and retrieve cookies for later usage?
  2. How to get http headers in flask?
  3. How to download a file over HTTP?
  4. urllib2.HTTPError: HTTP Error 403: Forbidden
  5. How do you send a HEAD HTTP request in Python 2?
  6. How to use cookies in Python Requests
  7. Problem HTTP error 403 in Python 3 Web Scraping
  8. How to add both file and JSON body in a FastAPI POST request?
  9. Python urllib2, basic HTTP authentication, and tr.im
  10. How to avoid HTTP error 429 (Too Many Requests) python
  11. How to send cookies in a post request with the Python Requests library?
  12. Is there an easy way to request a URL in python and NOT follow redirects?
  13. Python: HTTP Post a large file with streaming
  14. Python ‘requests’ library – define specific DNS?
  15. Putting a `Cookie` in a `CookieJar`
  16. Why am I getting this error in python ? (httplib)
  17. What should I do if socket.setdefaulttimeout() is not working?
  18. Save a large file using the Python requests library [duplicate]
  19. Python requests speed up using keep-alive
  20. BeautifulSoup: extract text from anchor tag
  21. Flask – POST Error 405 Method Not Allowed
  22. How to redirect from one domain to another and set cookies or headers for the other domain?
  23. How do I catch a specific HTTP error in Python?
  24. How to send zip files in the python Flask framework?
  25. Checking if a website is up via Python
  26. Retrieving all Cookies in Python
  27. How to catch 404 error in urllib.urlretrieve
  28. Is there any way to do HTTP PUT request in Python?
  29. How do I set HTTP headers using Python’s urllib?
  30. How to handle the popup “Accepting all cookies” when the element is data-testid – Using Selenium in Python

Leave a Comment