how to follow meta refreshes in Python

by

Here is a solution using BeautifulSoup and httplib2 (and certificate based authentication):

import BeautifulSoup
import httplib2

def meta_redirect(content):
    soup  = BeautifulSoup.BeautifulSoup(content)

    result=soup.find("meta",attrs={"http-equiv":"Refresh"})
    if result:
        wait,text=result["content"].split(";")
        if text.strip().lower().startswith("url="):
            url=text.strip()[4:]
            return url
    return None

def get_content(url, key, cert):
    
    h=httplib2.Http(".cache")
    h.add_certificate(key,cert,"")
    
    resp, content = h.request(url,"GET")
    
    # follow the chain of redirects
    while meta_redirect(content):
        resp, content = h.request(meta_redirect(content),"GET") 
            
    return content

More Related Contents:

  1. How to get the URL of a redirect with Python
  2. How to download image using requests
  3. Can I redirect the stdout into some sort of string buffer?
  4. Downloading a picture via urllib and python
  5. Output to the same line overwriting previous output?
  6. Proxy with urllib2
  7. Using an HTTP PROXY – Python [duplicate]
  8. How to fetch a non-ascii url with urlopen?
  9. Redirecting to URL in Flask
  10. Python Urllib2 SSL error
  11. Using MultipartPostHandler to POST form-data with Python
  12. Make a POST request while redirecting in flask
  13. Python urllib2 Basic Auth Problem
  14. Python-Requests close http connection
  15. How to redirect the output of print to a TXT file
  16. How do I get the IP address from a http request using the requests library?
  17. Python urllib2 Progress Hook
  18. Extracting text from script tag using BeautifulSoup in Python
  19. Making a POST call instead of GET using urllib2
  20. How can I speed up fetching pages with urllib2 in python?
  21. How to download any(!) webpage with correct charset in python?
  22. 104, ‘Connection reset by peer’ socket error, or When does closing a socket result in a RST rather than FIN?
  23. Unescape Python Strings From HTTP
  24. How to redirect stderr in Python?
  25. Python urllib2: Receive JSON response from url
  26. open url from pythonanywhere
  27. Submitting to a web form using python
  28. urllib2 HTTP Error 400: Bad Request
  29. Download and decompress gzipped file in memory?
  30. urllib2 and json

Leave a Comment