Try using FragmentStatePagerAdapter
http://android-developers.blogspot.com/2011_08_01_archive.html
coz it will remove the fragment as you swipe through the view(also when you call getitem), so you don’t need to explicitly remove them yourself.
I’ve been having the same problem it works for me.
I try your code and remove the if (fragment!=null)
block and it works.