I found another way of handling selection in the MVVM pattern, which solved my issue. Instead of maintaining the selection in the viewmodel, the selection is retrieved from the ListView/ListBox, and passed as a parameter to the Command. All done in XAML:
<ListView
x:Name="_items"
ItemsSource="{Binding Items}" ... />
<Button
Content="Remove Selected"
Command="{Binding RemoveSelectedItemsCommand}"
CommandParameter="{Binding ElementName=_items, Path=SelectedItems}"/>
in my ViewModel:
private void RemoveSelection(object parameter)
{
IList selection = (IList)parameter;
...
}