The only reliable and portable way to have the OS reclaim memory is to exit the process and restart it again, restoring any state you need to continue.
Of course, writing your own malloc/free implementation using brk/sbrk according to your needs is the other option.