Speeding up beautifulsoup

by

Okay, you can really speed this up by:

Since this is ASP.NET generated form and due to it’s security features, things get a bit more complicated. Here’s the complete code, don’t be afraid of it – I’ve added comments and open to questions:

import re
from bs4 import BeautifulSoup, SoupStrainer
import requests

# start session and get the search page
session = requests.Session()
response = session.get('https://acadinfo.wustl.edu/Courselistings/Semester/Search.aspx')

# parse the search page using SoupStrainer and lxml
strainer = SoupStrainer('form', attrs={'id': 'form1'})
soup = BeautifulSoup(response.content, 'lxml', parse_only=strainer)

# get the view state, event target and validation values
viewstate = soup.find('input', id='__VIEWSTATE').get('value')
eventvalidation = soup.find('input', id='__EVENTVALIDATION').get('value')
search_button = soup.find('input', value="Search")
event_target = re.search(r"__doPostBack\('(.*?)'", search_button.get('onclick')).group(1)

# configure post request parameters
data = {
    '__EVENTTARGET': event_target,
    '__EVENTARGUMENT': '',
    '__LASTFOCUS': '',
    '__VIEWSTATE': viewstate,
    '__EVENTVALIDATION': eventvalidation,
    'ctl00$Body$ddlSemester': '201405',
    'ctl00$Body$ddlSession': '',
    'ctl00$Body$ddlDept': '%',
    'ctl00$Body$ddlAttributes': '0',
    'ctl00$Body$Days': 'rbAnyDay',
    'ctl00$Body$Time': 'rbAnyTime',
    'ctl00$Body$cbMorning': 'on',
    'ctl00$Body$cbAfternoon': 'on',
    'ctl00$Body$cbEvening': 'on',
    'ctl00$Body$tbStart': '9:00am',
    'ctl00$Body$tbEnds': '5:00pm',
    'ctl00$Body$ddlUnits': '0',
    'ctl00$Body$cbHideIStudy': 'on',
    'ctl00$Body$courseList$hidHoverShow': 'Y',
    'ctl00$Body$courseList$hidDeptBarCnt': '',
    'ctl00$Body$courseList$hidSiteURL': 'https://acadinfo.wustl.edu/Courselistings',
    'ctl00$Body$courseList$hidExpandDetail': '',
    'ctl00$Body$hidDay': ',1,2,3,4,5,6,7',
    'ctl00$Body$hidLevel': '1234',
    'ctl00$Body$hidDefLevel': ''
}

# get the list of options
strainer = SoupStrainer('div', attrs={'id': 'Body_courseList_tabSelect'})
options = soup.select('#Body_ddlSchool > option')
for option in options:
    print "Processing {option} ...".format(option=option.text)

    data['ctl00$Body$ddlSchool'] = option.get('value')

    # make the search post request for a particular option
    response = session.post('https://acadinfo.wustl.edu/Courselistings/Semester/Search.aspx',
                            data=data)
    result_soup = BeautifulSoup(response.content, parse_only=strainer)
    print [item.text[:20].replace('&nbsp', ' ') + '...' for item in result_soup.select('div.CrsOpen')]

Prints:

Processing Architecture ...
[u'A46 ARCH 100...', u'A46 ARCH 111...', u'A46 ARCH 209...', u'A46 ARCH 211...', u'A46 ARCH 266...', u'A46 ARCH 305...', u'A46 ARCH 311...', u'A46 ARCH 323...', u'A46 ARCH 328...', u'A46 ARCH 336...', u'A46 ARCH 343...', u'A46 ARCH 350...', u'A46 ARCH 355...', u'A46 ARCH 411...', u'A46 ARCH 422...', u'A46 ARCH 428...', u'A46 ARCH 436...', u'A46 ARCH 445...', u'A46 ARCH 447...', u'A46 ARCH 465...', u'A48 LAND 451...', u'A48 LAND 453...', u'A48 LAND 461...']
Processing Art ...
[u'F10 ART 1052...', u'F10 ART 1073...', u'F10 ART 213A...', u'F10 ART 215A...', u'F10 ART 217B...', u'F10 ART 221A...', u'F10 ART 231I...', u'F10 ART 241D...', u'F10 ART 283T...', u'F10 ART 301A...', u'F10 ART 311E...', u'F10 ART 313D...', u'F10 ART 315B...', u'F10 ART 317H...', u'F10 ART 323A...', u'F10 ART 323B...', u'F10 ART 323C...', u'F10 ART 329C...', u'F10 ART 337E...', u'F10 ART 337F...', u'F10 ART 337H...', u'F10 ART 385A...', u'F10 ART 391M...', u'F10 ART 401A...', u'F10 ART 411E...', u'F10 ART 413D...', u'F10 ART 415B...', u'F10 ART 417H...', u'F10 ART 423A...', u'F10 ART 423B...', u'F10 ART 423C...', u'F10 ART 429C...', u'F10 ART 433C...', u'F10 ART 433D...', u'F10 ART 433E...', u'F10 ART 433K...', u'F10 ART 461C...', u'F10 ART 485A...', u'F20 ART 111P...', u'F20 ART 115P...', u'F20 ART 1186...', u'F20 ART 119C...', u'F20 ART 127A...', u'F20 ART 133B...', u'F20 ART 135G...', u'F20 ART 135I...', u'F20 ART 135J...', u'F20 ART 1361...', u'F20 ART 1363...', u'F20 ART 1713...', u'F20 ART 219C...', u'F20 ART 2363...', u'F20 ART 2661...', u'F20 ART 281S...', u'F20 ART 311P...', u'F20 ART 315P...', u'F20 ART 3183...', u'F20 ART 333B...', u'F20 ART 335A...', u'F20 ART 335J...', u'F20 ART 3713...', u'F20 ART 381S...', u'F20 ART 415P...', u'F20 ART 435I...']
...

There are certainly things to improve here, like, I’ve hardcoded the other form values – you should probably parse the possible values and set them appropriately.

Another improvement would be to tie this up to grequests:

GRequests allows you to use Requests with Gevent to make asynchronous
HTTP Requests easily.

As you can see, when you are at the higher-level and interact with a browser through the webdriver – you are not worried about the actual requests coming to the server to get you the data. This makes it easy to automate, but can be painfully slow. When you go down to the low-level automation, you have more options to speed things up, but the implementation complexity grows very fast. Plus, think about how reliable could this sort of the solution be. So may be stick to the “black-box” solution and stay with selenium?

I’ve also tried to solve the problem using:

but failed because of different reasons (can provide you with the relevant error messages). Though, all these 3 tools should have helped to simplify the solution.

Also see similar threads:

More Related Contents:

  1. Beautiful Soup and Table Scraping – lxml vs html parser
  2. how to scrape web page that is not written directly using HTML, but is auto-generated using JavaScript? [closed]
  3. WebDriverWait not working as expected
  4. Difference between text and innerHTML using Selenium
  5. can we use XPath with BeautifulSoup?
  6. Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org
  7. Scraping Google Finance (BeautifulSoup)
  8. How to scrape a website which requires login using python and beautifulsoup?
  9. Parsing HTML in python – lxml or BeautifulSoup? Which of these is better for what kinds of purposes?
  10. Web scraping program cannot find element which I can see in the browser
  11. ‘list’ object has no attribute ‘get_attribute’ while iterating through WebElements
  12. Scrape website with dynamic mouseover event
  13. UnicodeEncodeError: ‘ascii’ codec can’t encode character ‘\xe9’ – -when using urlib.request python3
  14. Beautiful Soup 4 find_all don’t find links that Beautiful Soup 3 finds
  15. Dynamic dropdown doesn’t populate with auto suggestions on https://www.nseindia.com/ when values are passed using Selenium and Python
  16. How to click a link by text with No Text in Python
  17. How to get text from span tag in BeautifulSoup
  18. BeautifulSoup returns empty list when searching by compound class names
  19. unable to call firefox from selenium in python on AWS machine
  20. How to scrape only visible webpage text with BeautifulSoup?
  21. Python selenium multiprocessing
  22. Scraping dynamic data selenium – Unable to locate element
  23. How to find tag with particular text with Beautiful Soup?
  24. Can bs4 get the dynamic content of a webpage if requests can’t?
  25. How find specific data attribute from html tag in BeautifulSoup4?
  26. Exclude unwanted tag on Beautifulsoup Python
  27. Python selenium error when trying to launch firefox
  28. Scrape Dynamic contents created by Javascript using Python
  29. Python: Disable images in Selenium Google ChromeDriver
  30. How to scrape dynamic webpages by Python

Leave a Comment