It seems that you want a list structure with very fast removal and random access by index (not by key) times. An ArrayList
gives you the latter and a HashMap
or TreeMap
give you the former.
There is one structure in Apache Commons Collections that may be what you are looking for, the TreeList. The JavaDoc specifies that it is optimized for quick insertion and removal at any index in the list. If you also need generics though, this will not help you.