After using Spy++ to inspect Steam’s window (its window styles, how it replies to window messages) and trying to match everything it does, combined with the DWMAPI calls from this C# borderless window behavior, I believe I figured it out.
To hide the window’s border, handle the WM_NCCALCSIZE message in your WindowProc:
case WM_NCCALCSIZE: {
if (window->is_borderless) {
return 0;
} else {
return DefWindowProc(hwnd, msg, wparam, lparam);
}
}
To enable the shadow, all you need to do is:
MARGINS borderless = {1,1,1,1};
DwmExtendFrameIntoClientArea(hwnd, &borderless);
To turn it back off, restore the default margins MARGINS windowed = {0,0,0,0};.
Perhaps throw in a SetWindowPos(hwnd, 0, 0, 0, 0, 0, SWP_NOZORDER | SWP_NOOWNERZORDER | SWP_NOACTIVATE | SWP_FRAMECHANGED | SWP_ASYNCWINDOWPOS | SWP_NOSIZE | SWP_NOMOVE ); also, to make sure the frame gets redrawn.
However, this does not seem to work with all window styles, apparently your window style must not contain a titlebar. Title bars work fine, and adding one seems to enable the minimize animation.
The simplest window style I got the shadow to work with was WS_POPUP | WS_THICKFRAME, to also get aero snap, maximizing, minimizing, and the smooth minimize animation I used WS_POPUP | WS_THICKFRAME | WS_MINIMIZEBOX | WS_MAXIMIZEBOX | WS_CAPTION.
Changing DWMWA_NCRENDERING_POLICY or DWMWA_ALLOW_NCPAINT via DwmSetWindowAttribute does not appear to be required, the default settings seem to work.
One word of caution: DwmExtendFrameIntoClientArea does exactly what the name suggests, so if you are drawing an image with an alpha channel directly into your client area (say with opengl, direct3d/2d), a small frame will be visible through it:
So you might have to put a non transparent widget, brush or something behind the transparent element.
If all goes well, it should then look like this:
Here is a small example project, F11 toggles borderless/windowed mode, F12 toggles the borderless shadow on and off.