How to retrieve more than 100 results using Twitter4j

Unfortunately you can’t, at least not in a direct way such as doing

query.setCount(101);

As the javadoc says it will only allow up to 100 tweets.

In order to overcome this, you just have to ask for them in batches and in every batch set the maximum ID that you get to be 1 less than the last Id you got from the last one. To wrap this up, you gather every tweet from the process into an ArrayList (which by the way should not stay generic, but have its type defined as ArrayList<Status> – An ArrayList that carries Status objects) and then print everything! Here’s an implementation:

void setup() {

  ConfigurationBuilder cb = new ConfigurationBuilder();
  cb.setOAuthConsumerKey("xxxx");
  cb.setOAuthConsumerSecret("xxxx");
  cb.setOAuthAccessToken("xxxx");
  cb.setOAuthAccessTokenSecret("xxxx");

  Twitter twitter = new TwitterFactory(cb.build()).getInstance();
  Query query = new Query("#peace");
  int numberOfTweets = 512;
  long lastID = Long.MAX_VALUE;
  ArrayList<Status> tweets = new ArrayList<Status>();
  while (tweets.size () < numberOfTweets) {
    if (numberOfTweets - tweets.size() > 100)
      query.setCount(100);
    else 
      query.setCount(numberOfTweets - tweets.size());
    try {
      QueryResult result = twitter.search(query);
      tweets.addAll(result.getTweets());
      println("Gathered " + tweets.size() + " tweets");
      for (Status t: tweets) 
        if(t.getId() < lastID) lastID = t.getId();

    }

    catch (TwitterException te) {
      println("Couldn't connect: " + te);
    }; 
    query.setMaxId(lastID-1);
  }

  for (int i = 0; i < tweets.size(); i++) {
    Status t = (Status) tweets.get(i);

    GeoLocation loc = t.getGeoLocation();

    String user = t.getUser().getScreenName();
    String msg = t.getText();
    String time = "";
    if (loc!=null) {
      Double lat = t.getGeoLocation().getLatitude();
      Double lon = t.getGeoLocation().getLongitude();
      println(i + " USER: " + user + " wrote: " + msg + " located at " + lat + ", " + lon);
    } 
    else 
      println(i + " USER: " + user + " wrote: " + msg);
  }
}

Note: The line

ArrayList<Status> tweets = new ArrayList<Status>();

should properly be:

List<Status> tweets = new ArrayList<Status>();

because you should always use the interface in case you want to add a different implementation. This of course, if you are on Processing 2.x will require this in the beginning:

import java.util.List;

Leave a Comment