Raymond Chen answered this a while back
(https://devblogs.microsoft.com/oldnewthing/20071008-00/?p=24863):
It’s actually pretty simple although
hardly anything you’d be able to guess
on your own. Note: The details of this
algorithm are an implementation
detail. It can change at any time, so
don’t rely on it. In fact, it already
changed with Flip and Flip3D; I’m just
talking about the Classic Alt+Tab
window here.For each visible window, walk up its
owner chain until you find the root
owner. Then walk back down the visible
last active popup chain until you find
a visible window. If you’re back to
where you’re started, then put the
window in the Alt+Tab list. In
pseudo-code:
BOOL IsAltTabWindow(HWND hwnd)
{
// Start at the root owner
HWND hwndWalk = GetAncestor(hwnd, GA_ROOTOWNER);
// See if we are the last active visible popup
HWND hwndTry;
while ((hwndTry = GetLastActivePopup(hwndWalk)) != hwndTry) {
if (IsWindowVisible(hwndTry)) break;
hwndWalk = hwndTry;
}
return hwndWalk == hwnd;
}
Follow the link to Chen’s blog entry for more details and some corner conditions.