You can have a parent-child relationship with the windows in different processes. It’s tricky to get it to work right in all cases. You may have to debug various strange symptoms.
Normally, windows in separate processes would get their messages from separate input queues using separate message pumps. When you use SendMessage to a window in another process, it’s actually posted to the other window’s queue, processed there, and the return is effectively marshaled back to the original process. So if one of the processes stops handling messages, you can effectively lock up the other as well. (That’s true even within a process when the windows are created on different threads and the thread queues haven’t been attached.)
But when you set up the parent/child relationship among windows in different threads, Windows attaches those input queues together, forcing the message processing to be synchronous. You’re no longer in the normal case, but you face the same kinds of problems: a hang in the processing for one window effectively hangs the other process.
Watch out for messages that pass pointers in the params. The pointers will not be valid in the receiving process. (There are a couple of exceptions, like WM_COPYDATA, which recreates the data in the receiving process for you. But even those have limitations.)
You have to be especially careful when the windows are being destroyed. If possible, disconnect the parent-child relationship before destroying either window. If it’s not possible, then it’s probably best to manually destroy the child window before the parent is destroyed. Normally, destroying a parent will cause the children to be destroyed automatically, but it’s easy to get hangs when the child is in another process (or un-attached thread).
In newer versions of Windows (Vista+), you can also hit some security speedbumps if the processes run at different integrity levels.
Thanks to IInspectable who pointed out an error in my earlier answer.