Given a URL to a text file, what is the simplest way to read the contents of the text file?

by

Edit 09/2016: In Python 3 and up use urllib.request instead of urllib2

Actually the simplest way is:

import urllib2  # the lib that handles the url stuff

data = urllib2.urlopen(target_url) # it's a file like object and works just like a file
for line in data: # files are iterable
    print line

You don’t even need “readlines”, as Will suggested. You could even shorten it to: *

import urllib2

for line in urllib2.urlopen(target_url):
    print line

But remember in Python, readability matters.

However, this is the simplest way but not the safe way because most of the time with network programming, you don’t know if the amount of data to expect will be respected. So you’d generally better read a fixed and reasonable amount of data, something you know to be enough for the data you expect but will prevent your script from been flooded: 

import urllib2

data = urllib2.urlopen("http://www.google.com").read(20000) # read only 20 000 chars
data = data.split("\n") # then split it into lines

for line in data:
    print line

* Second example in Python 3: 

import urllib.request  # the lib that handles the url stuff

for line in urllib.request.urlopen(target_url):
    print(line.decode('utf-8')) #utf-8 or iso8859-1 or whatever the page encoding scheme is

More Related Contents:

  1. How to have one colorbar for all subplots
  2. Scope of python variable in for loop
  3. unknown error: session deleted because of page crash from unknown error: cannot determine loading status from tab crashed with ChromeDriver Selenium
  4. Python: Problem with raw_input reading a number
  5. How to change values in a tuple?
  6. How to make a surface with a transparent background in pygame
  7. Python: load variables in a dict into namespace
  8. Python argparse – Add argument to multiple subparsers
  9. Longest common substring from more than two strings
  10. Python: Open file in zip without temporarily extracting it
  11. How to open multiple webpages in separate tabs within a browser using selenium-webdriver and python
  12. Count consecutive characters
  13. How should I use the Optional type hint?
  14. join two lists of dictionaries on a single key
  15. In Python, how do I iterate over a dictionary in sorted key order?
  16. Difference between dict.clear() and assigning {} in Python
  17. python logging specific level only
  18. Python – Converting XLSX to PDF
  19. How can I provide a “callback” to an API?
  20. How to remove duplicate items from a list using list comprehension? [duplicate]
  21. How to sort multidimensional array by column?
  22. sum each value in a list of tuples
  23. Why is `with open()` better for opening files in Python?
  24. How can I set a default value for a field in a Django model?
  25. Python: How to sort a list of dictionaries by several values?
  26. Use pdb.set_trace() in a script that reads stdin via a pipe
  27. Unicode encoding for filesystem in Mac OS X not correct in Python?
  28. Search and replace for text within a pdf, in Python [duplicate]
  29. printing slowly (Simulate typing)
  30. aiohttp: rate limiting parallel requests

Leave a Comment