Python ‘requests’ library – define specific DNS?

by

requests uses urllib3, which ultimately uses httplib.HTTPConnection as well, so the techniques from https://stackoverflow.com/questions/4623090/python-set-custom-dns-server-for-urllib-requests (now deleted, it merely linked to Tell urllib2 to use custom DNS) still apply, to a certain extent.

The urllib3.connection module subclasses httplib.HTTPConnection under the same name, having replaced the .connect() method with one that calls self._new_conn. In turn, this delegates to urllib3.util.connection.create_connection(). It is perhaps easiest to patch that function:

from urllib3.util import connection


_orig_create_connection = connection.create_connection


def patched_create_connection(address, *args, **kwargs):
    """Wrap urllib3's create_connection to resolve the name elsewhere"""
    # resolve hostname to an ip address; use your own
    # resolver here, as otherwise the system resolver will be used.
    host, port = address
    hostname = your_dns_resolver(host)

    return _orig_create_connection((hostname, port), *args, **kwargs)


connection.create_connection = patched_create_connection

and you’d provide your own code to resolve the host portion of the address into an ip address instead of relying on the connection.create_connection() call (which wraps socket.create_connection()) to resolve the hostname for you.

Like all monkeypatching, be careful that the code hasn’t significantly changed in later releases; the patch here was created against urllib3 version 1.21.1. but should work for versions as far back as 1.9.

Note that this answer was re-written to work with newer urllib3 releases, which have added a much more convenient patching location. See the edit history for the old method, applicable to version < 1.9, as a patch to the vendored urllib3 version rather than a stand-alone installation.

More Related Contents:

  1. How to download a file over HTTP?
  2. What are the differences between the urllib, urllib2, urllib3 and requests module?
  3. urllib2.HTTPError: HTTP Error 403: Forbidden
  4. Python requests – print entire http request (raw)?
  5. Python Requests library redirect new url
  6. Python-Requests close http connection
  7. How do I set headers using python’s urllib?
  8. Only download a part of the document using python requests
  9. How to get response SSL certificate from requests in python?
  10. python how to decode http response
  11. How to implement retry mechanism into python requests library?
  12. What should I do if socket.setdefaulttimeout() is not working?
  13. Python requests speed up using keep-alive
  14. How do I catch a specific HTTP error in Python?
  15. Python – make a POST request using Python 3 urllib
  16. How to get the raw content of a response in requests with Python?
  17. How to catch 404 error in urllib.urlretrieve
  18. How to limit download rate of HTTP requests in requests python library?
  19. How do I set HTTP headers using Python’s urllib?
  20. should I call close() after urllib.urlopen()?
  21. Web scraping program cannot find element which I can see in the browser
  22. How to load all entries in an infinite scroll at once to parse the HTML in python
  23. How do I get the IP address from a http request using the requests library?
  24. Python: How do I get key/value pairs from the BaseHTTPRequestHandler HTTP POST handler?
  25. Passing cookies while logging in
  26. How to get around python requests SSL and proxy error?
  27. Python 3 urllib produces TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str
  28. Http Redirection code 3XX in python requests
  29. Django: using and POST
  30. Unable to get local issuer certificate when using requests

Leave a Comment