I’ve run into that same issue as well. It’s because you’re closing your connection to the socket, but not the socket itself. The socket can enter a TIME_WAIT state (to ensure all data has been transmitted, TCP guarantees delivery if possible) and take up to 4 minutes to release.
or, for a REALLY detailed/technical explanation, check this link
It’s certainly annoying, but it’s not a bug. See the comment from @Vereb on this answer below on the use of SO_REUSEADDR
.