Objects in hashsets should either be immutable, or you need to exercise discipline in not changing them after they’ve been used in a hashset (or hashmap).
In practice I’ve rarely found this to be a problem – I rarely find myself needing to use complex objects as keys or set elements, and when I do it’s usually not a problem just not to mutate them. Of course if you’ve exposed the references to other code by this time, it can become harder.