I guess that because std::unordered_map needs to rehash, and therefore copy elements, the types need to be complete, whereas a map, only ever working with pointers to elements, will not exhibit that problem.
The solution here is to have an unordered map to a pointer:
std::unordered_map<uint32_t, std::shared_ptr<Test> >.