Unfortunately, there’s is no proper technical way to get the information you want. You might invent some tests, but those will have a very low correlation with the reality. So either you’ll not catch those you want, or you’ll have a larger number of false positives. Neither can be considered to make sense.
Generating any kind of traffic backwards from an Internet server in response to an incoming client (a port scan, or even a simple ping) is generally frowned upon. Or, in the case of a port scan, it may be even worse for you, eg when the client lives behind a central corporate firewall, the worst of which is when the client comes from behind the central government network firewall pool…
Frankly, IP-based bans (or actually, any kind of limiting focusing on people who do not exclusively possess their public IP address: proxy servers, VPNs, NAT devices, etc) have been unrealistic for a long time, and as the IPv4 pools have been getting depleted in many parts of the world, ISPs are putting more and more clients behind large NAT pools (it’s this week’s news in my country that the largest ISP, a subsidiary of Deutsche Telekom, has started handing out private IPv4 addresses as a standard way of business to its customers, and people have to ask the provider explicitly to get a public IP address), so there’s even less and less point in doing so. If you want to ban clients, you should ban them based on identity (account), and not based on IP address.