List.Except is not working

In order to make Except method working as expected, the class AssignUserViewModel must have GetHashCode and Equals methods correctly overridden.

For example, if AssignUserViewModel objects are uniquely defined by their Id, you should define the class in this way:

class AssignUserViewModel
{
    // other methods...


    public override int GetHashCode()
    {
        return this.Id.GetHashCode();
    }
    public override bool Equals(object obj)
    {
        if (!(obj is AssignUserViewModel))
            throw new ArgumentException("obj is not an AssignUserViewModel");
        var usr = obj as AssignUserViewModel;
        if (usr == null)
            return false;
        return this.Id.Equals(usr.Id);
    }
}

Otherwise, if you can’t/don’t want to change the class implementation, you can implement an IEqualityComparer<> and pass it to the Except method, e.g. :

class AssignUserViewModelEqualityComparer : IEqualityComparer<AssignUserViewModel>
{
    public bool Equals(AssignUserViewModel x, AssignUserViewModel y)
    {
        if (object.ReferenceEquals(x, y))
            return true;
        if(x == null || y == null)
            return false;
        return x.Id.Equals(y.Id);
    }

    public int GetHashCode(AssignUserViewModel obj)
    {
        return obj.Id.GetHashCode();
    }
}

then your last line would become:

assignUsers = assignUsers.Except(assignedUsers, new AssignUserViewModelEqualityComparer()).ToList();