Read timeout using either urllib2 or any other http library

by

It’s not possible for any library to do this without using some kind of asynchronous timer through threads or otherwise. The reason is that the timeout parameter used in httplib, urllib2 and other libraries sets the timeout on the underlying socket. And what this actually does is explained in the documentation.

SO_RCVTIMEO

Sets the timeout value that specifies the maximum amount of time an input function waits until it completes. It accepts a timeval structure with the number of seconds and microseconds specifying the limit on how long to wait for an input operation to complete. If a receive operation has blocked for this much time without receiving additional data, it shall return with a partial count or errno set to [EAGAIN] or [EWOULDBLOCK] if no data is received.

The bolded part is key. A socket.timeout is only raised if not a single byte has been received for the duration of the timeout window. In other words, this is a timeout between received bytes.

A simple function using threading.Timer could be as follows.

import httplib
import socket
import threading

def download(host, path, timeout = 10):
    content = None
    
    http = httplib.HTTPConnection(host)
    http.request('GET', path)
    response = http.getresponse()
    
    timer = threading.Timer(timeout, http.sock.shutdown, [socket.SHUT_RD])
    timer.start()
    
    try:
        content = response.read()
    except httplib.IncompleteRead:
        pass
        
    timer.cancel() # cancel on triggered Timer is safe
    http.close()
    
    return content

>>> host="releases.ubuntu.com"
>>> content = download(host, '/15.04/ubuntu-15.04-desktop-amd64.iso', 1)
>>> print content is None
True
>>> content = download(host, '/15.04/MD5SUMS', 1)
>>> print content is None
False

Other than checking for None, it’s also possible to catch the httplib.IncompleteRead exception not inside the function, but outside of it. The latter case will not work though if the HTTP request doesn’t have a Content-Length header.

More Related Contents:

  1. How to set timeout on python’s socket recv method?
  2. What does Python’s socket.recv() return for non-blocking sockets if no data is received until a timeout occurs?
  3. Python socket.accept nonblocking?
  4. How to use Python to login to a webpage and retrieve cookies for later usage?
  5. Python socket not receiving without sending
  6. TypeError: a bytes-like object is required, not ‘str’
  7. What is the quickest way to HTTP GET in Python?
  8. Using an HTTP PROXY – Python [duplicate]
  9. On localhost, how do I pick a free port number?
  10. How to specify an authenticated proxy for a python http connection?
  11. Python-Requests close http connection
  12. How do I set headers using python’s urllib?
  13. Multithreaded web server in python
  14. Best way to parse a URL query string
  15. How to download a file using python in a ‘smarter’ way?
  16. Python Socket Flush
  17. How to get response SSL certificate from requests in python?
  18. How to connect to a remote Windows machine to execute commands using python?
  19. Why am I getting the error “connection refused” in Python? (Sockets)
  20. 104, ‘Connection reset by peer’ socket error, or When does closing a socket result in a RST rather than FIN?
  21. Download file using partial download (HTTP)
  22. why is python string split() not splitting
  23. Python [Errno 98] Address already in use
  24. how to get access to error message from abort command when using custom error handler
  25. How to send and receive HTTP POST requests in Python [closed]
  26. Making moves w/ websockets and python / django ( / twisted? )
  27. What is the global default timeout
  28. Is there a library for retrieving a file from a remote zip? [closed]
  29. Python seek on remote file using HTTP
  30. Sending multiple files through a TCP socket

Leave a Comment