How to extract tables from websites in Python

by

Pandas can do this right out of the box, saving you from having to parse the html yourself. to_html() extracts all tables from your html and puts them in a list of dataframes. to_csv() can be used to convert each dataframe to a csv file. For the web page in your example, the relevant table is the last one, which is why I used df_list[-1] in the code below.

import requests
import pandas as pd

url="http://www.ffiec.gov/census/report.aspx?year=2011&state=01&report=demographic&msa=11500"
html = requests.get(url).content
df_list = pd.read_html(html)
df = df_list[-1]
print(df)
df.to_csv('my data.csv')

It’s simple enough to do in one line, if you prefer:

pd.read_html(requests.get(<url>).content)[-1].to_csv(<csv file>)

P.S. Just make sure you have lxml, html5lib, and BeautifulSoup4 packages installed in advance.

More Related Contents:

  1. urllib and “SSL: CERTIFICATE_VERIFY_FAILED” Error
  2. What are the differences between the urllib, urllib2, urllib3 and requests module?
  3. Downloading a picture via urllib and python
  4. can we use XPath with BeautifulSoup?
  5. AttributeError: ‘module’ object has no attribute ‘urlopen’
  6. should I call close() after urllib.urlopen()?
  7. Download Returned Zip file from URL
  8. Django: add image in an ImageField from image url
  9. How do I set headers using python’s urllib?
  10. Handling urllib2’s timeout? – Python
  11. SSL: CERTIFICATE_VERIFY_FAILED with Python3
  12. python save image from url
  13. Python: Get HTTP headers from urllib2.urlopen call?
  14. Making a POST call instead of GET using urllib2
  15. multiprocessing.pool.MaybeEncodingError: ‘TypeError(“cannot serialize ‘_io.BufferedReader’ object”,)’
  16. Get size of a file before downloading in Python
  17. How to download any(!) webpage with correct charset in python?
  18. Changing User Agent in Python 3 for urrlib.request.urlopen
  19. Only add to a dict if a condition is met
  20. Python 3 urllib produces TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str
  21. Python – make a POST request using Python 3 urllib
  22. Submitting to a web form using python
  23. How to save “complete webpage” not just basic html using Python
  24. no module named urllib.parse (How should I install it?)
  25. How to catch 404 error in urllib.urlretrieve
  26. Download pdf using urllib?
  27. Python 3 – urllib, HTTP Error 407: Proxy Authentication Required
  28. How do I set HTTP headers using Python’s urllib?
  29. Python3 error: initial_value must be str or None, with StringIO
  30. Python 3.6 urllib TypeError: can’t concat bytes to str

Leave a Comment