Google Search Web Scraping with Python

by

You can always directly scrape Google results. To do this, you can use the URL https://google.com/search?q=<Query> this will return the top 10 search results.

Then you can use lxml for example to parse the page. Depending on what you use, you can either query the resulting node tree via a CSS-Selector (.r a) or using a XPath-Selector (//h3[@class="r"]/a)

In some cases the resulting URL will redirect to Google. Usually it contains a query-parameter qwhich will contain the actual request URL.

Example code using lxml and requests:

from urllib.parse import urlencode, urlparse, parse_qs

from lxml.html import fromstring
from requests import get

raw = get("https://www.google.com/search?q=StackOverflow").text
page = fromstring(raw)

for result in page.cssselect(".r a"):
    url = result.get("href")
    if url.startswith("/url?"):
        url = parse_qs(urlparse(url).query)['q']
    print(url[0])

A note on google banning your IP: In my experience, google only bans
if you start spamming google with search requests. It will respond
with a 503 if Google thinks you are bot.

More Related Contents:

  1. google search with python requests library
  2. Searching in Google with Python
  3. python 2.7 script to add a column
  4. How can I calculate a list modulo a numer in python?
  5. Python 2: Conditional statement behaviour of lists with greater/smaller [duplicate]
  6. Zip lists in Python
  7. Why is a list comprehension so much faster than appending to a list?
  8. Backporting Python 3 open(encoding=”utf-8″) to Python 2
  9. How to update the image of a Tkinter Label widget?
  10. 2D list has weird behavor when trying to modify a single value [duplicate]
  11. python filter list of dictionaries based on key value
  12. How to convert a string to utf-8 in Python
  13. Accessing attributes on literals work on all types, but not `int`; why? [duplicate]
  14. Is it safe to use the python word “type” in my code?
  15. What is the Python ‘buffer’ type for?
  16. Fastest way to process a large file?
  17. How to integrate Pygame and PyQt4?
  18. UnicodeEncodeError: ‘ascii’ codec can’t encode characters in position 0-5: ordinal not in range(128) [duplicate]
  19. Can’t install mysql-python (newer versions) in Windows
  20. python re.sub, only replace part of match [duplicate]
  21. How to apply a function to each sublist of a list in python?
  22. Printing on the same line with time.sleep()
  23. How to use PyCharm to debug Scrapy projects
  24. Python 2.7 on Ubuntu
  25. Enforcing python version in setup.py
  26. Type error Unhashable type:set
  27. What is the easiest way to achieve realtime plotting in pyqtgraph
  28. How to convert integers in list to string in python
  29. Upload file with Selenium Webdriver Python
  30. Python – reference inner class from other inner class

Leave a Comment