Managing Tweepy API Search

by

I originally worked out a solution based on Yuva Raj‘s suggestion to use additional parameters in GET search/tweets – the max_id parameter in conjunction with the id of the last tweet returned in each iteration of a loop that also checks for the occurrence of a TweepError.

However, I discovered there is a far simpler way to solve the problem using a tweepy.Cursor (see tweepy Cursor tutorial for more on using Cursor).

The following code fetches the most recent 1000 mentions of 'python'.

import tweepy
# assuming twitter_authentication.py contains each of the 4 oauth elements (1 per line)
from twitter_authentication import API_KEY, API_SECRET, ACCESS_TOKEN, ACCESS_TOKEN_SECRET

auth = tweepy.OAuthHandler(API_KEY, API_SECRET)
auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)

api = tweepy.API(auth)

query = 'python'
max_tweets = 1000
searched_tweets = [status for status in tweepy.Cursor(api.search, q=query).items(max_tweets)]

Update: in response to Andre Petre‘s comment about potential memory consumption issues with tweepy.Cursor, I’ll include my original solution, replacing the single statement list comprehension used above to compute searched_tweets with the following:

searched_tweets = []
last_id = -1
while len(searched_tweets) < max_tweets:
    count = max_tweets - len(searched_tweets)
    try:
        new_tweets = api.search(q=query, count=count, max_id=str(last_id - 1))
        if not new_tweets:
            break
        searched_tweets.extend(new_tweets)
        last_id = new_tweets[-1].id
    except tweepy.TweepError as e:
        # depending on TweepError.code, one may want to retry or wait
        # to keep things simple, we will give up on an error
        break

More Related Contents:

  1. Importing Tweets via Python using Tweepy
  2. How can I get tweets older than a week (using tweepy or other python libraries)
  3. How to add a location filter to tweepy module
  4. Avoid Twitter API limitation with Tweepy
  5. Get data from Twitter using Tweepy and store in csv file
  6. regex for Twitter username
  7. tweepy get tweets between two dates
  8. How to apply NLTK word_tokenize library on a Pandas dataframe for Twitter data?
  9. python: [Errno 10054] An existing connection was forcibly closed by the remote host
  10. Pandas max value index
  11. How do I extend a python module? Adding new functionality to the `python-twitter` package
  12. httplib CannotSendRequest error in WSGI
  13. syntaxError: ‘continue’ not properly in loop
  14. Convert Tweepy Status object into JSON
  15. “pip install unroll”: “python setup.py egg_info” failed with error code 1
  16. Freezing/Hanging tkinter GUI in waiting for the thread to complete
  17. “SSL: certificate_verify_failed” error when scraping https://www.thenewboston.com/
  18. Text box with line wrapping in matplotlib?
  19. Pandas read csv file with float values results in weird rounding and decimal digits
  20. pandas – filter dataframe by another dataframe by row elements
  21. How to set opacity of background colour of graph with Matplotlib
  22. What does a 4-element tuple argument for ‘bbox_to_anchor’ mean in matplotlib?
  23. threading ignores KeyboardInterrupt exception
  24. Matplotlib: how to show a figure that has been closed
  25. Detect strings with non English characters in Python
  26. Grouped Bar graph Pandas
  27. is not JSON serializable
  28. How can I draw lines into numpy arrays?
  29. subprocess call ffmpeg (command line)
  30. In Python, is read() , or readlines() faster?

Leave a Comment