Correct usage of DriveApp.continueFileIterator(continuationToken)

Like Jonathon said, you’re comparing dates wrongly. But that’s not the main issue with your script nor what you asked.

The main concept you’re getting wrong is that the continuation token can’t be saved before you do your loop. When you get the token, it saves where you were at that moment, if you continue iterating afterwards, that’s not saved and you will repeat those steps later, just like you’re experiencing.

To get the token later you cannot let your script terminate with an error. You have to measure how many files you can process under 5 minutes and stop your script manually before that, so you can have a chance at saving the token.

Here’s the correct way of doing it:

function listFilesInFolder() {
  var MAX_FILES = 20; //use a safe value, don't be greedy
  var id = 'folder-id';
  var scriptProperties = PropertiesService.getScriptProperties();
  var lastExecution = scriptProperties.getProperty('LAST_EXECUTION');
  if( lastExecution === null )
    lastExecution = '';

  var continuationToken = scriptProperties.getProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN');
  var iterator = continuationToken == null ?
    DriveApp.getFolderById(id).getFiles() : DriveApp.continueFileIterator(continuationToken);


  try { 
    for( var i = 0; i < MAX_FILES && iterator.hasNext(); ++i ) {
      var file = iterator.next();
      var dateCreated = formatDate(file.getDateCreated());
      if(dateCreated > lastExecution)
        processFile(file);
    }
  } catch(err) {
    Logger.log(err);
  }

  if( iterator.hasNext() ) {
    scriptProperties.setProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN', iterator.getContinuationToken());
  } else { // Finished processing files so delete continuation token
    scriptProperties.deleteProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN');
    scriptProperties.setProperty('LAST_EXECUTION', formatDate(new Date()));
  }
}

function formatDate(date) { return Utilities.formatDate(date, "GMT", "yyyy-MM-dd HH:mm:ss"); }

function processFile(file) {
  var id = file.getId();
  var name = file.getName();
  //your processing...
  Logger.log(name);
}

Anyway, it may be possible that a file gets created between your runs and you do not get it on your continued-iteration. Then, by saving the execution time after your the last run, you may miss it on your next run too. I do not know your use-case, if it’s acceptable to eventually reprocess some files or to miss some. If you can’t have either situations at all, then the only solution I see is to save the ids of all files you have already processed. You may need to store those on a drive file, because PropertiesService may be too small for too many ids.

Leave a Comment