Actually, under certain conditions it is an option to have the pkgs subdirs removed. As stated here by Anaconda Community Support “the pkgs directory is only a cache. You can remove it completely is you want to.
However, when creating new environments, it is more efficient to leave whatever packages are in the cache around.”
According to the documentation you can use conda clean --packages to remove unused packages in pkgs (which will move them to pkgs/.trash from which you can then safely delete them). While this does not check for packages installed using symlinks back to the package cache, this is not a topic if you don’t use such environments or work under Windows. I guess that’s why conda clean --packages is included in conda clean --all.
To more aggressively save space you can use conda clean --force-pkgs-dirs to remove all writable package caches (with the same caveat that there could be environments linked to these dirs). If you don’t use environments or use Anaconda under Windows, you’re probably safe. Personally, I use this option without issues.