I finally got all images to render by changing the value of the IsReusable property to true:
public bool IsReusable
{
get
{
return true;
}
}
Apparently, this keeps the handler in memory and able to handle multiple requests. When set to false, it had to create a new instance of the handler for each incoming request.