win32 GUI app that writes usage text to stdout when invoked as “app.exe –help”

Microsoft designed console and GUI apps to be mutually exclusive.
This bit of short-sightedness means that there is no perfect solution.
The most popular approach is to have two executables (eg. cscript / wscript,
java / javaw, devenv.com / devenv.exe etc) however you’ve indicated that you consider this “cheating”.

You’ve got two options – to make a “console executable” or a “gui executable”,
and then use code to try to provide the other behaviour.

• GUI executable:

cmd.exe will assume that your program does no console I/O so won’t wait for it to terminate
before continuing, which in interactive mode (ie not a batch) means displaying the next (“C:\>“) prompt
and reading from the keyboard. So even if you use AttachConsole your output will be mixed
with cmd‘s output, and the situation gets worse if you try to do input. This is basically a non-starter.

• Console executable:

Contrary to belief, there is nothing to stop a console executable from displaying a GUI, but there are two problems.

The first is that if you run it from the command line with no arguments (so you want the GUI),
cmd will still wait for it to terminate before continuing, so that particular
console will be unusable for the duration. This can be overcome by launching
a second process of the same executable (do you consider this cheating?),
passing the DETACHED_PROCESS flag to CreateProcess() and immediately exiting.
The new process can then detect that it has no console and display the GUI.

Here’s C code to illustrate this approach:

#include <stdio.h>
#include <windows.h>

int main(int argc, char *argv[])
{
    if (GetStdHandle(STD_OUTPUT_HANDLE) == 0) // no console, we must be the child process
    {
        MessageBox(0, "Hello GUI world!", "", 0);
    }
    else if (argc > 1) // we have command line args
    {
        printf("Hello console world!\n");
    }
    else // no command line args but a console - launch child process
    {
        DWORD dwCreationFlags = CREATE_DEFAULT_ERROR_MODE | DETACHED_PROCESS;
        STARTUPINFO startinfo;
        PROCESS_INFORMATION procinfo;
        ZeroMemory(&startinfo, sizeof(startinfo));
        startinfo.cb = sizeof(startinfo);
        if (!CreateProcess(NULL, argv[0], NULL, NULL, FALSE, dwCreationFlags, NULL, NULL, &startinfo, &procinfo))
            MessageBox(0, "CreateProcess() failed :(", "", 0);
    }
    exit(0);
}

I compiled it with cygwin’s gcc – YMMV with MSVC.

The second problem is that when run from Explorer, your program will for a split second
display a console window. There’s no programmatic way around this because the console is
created by Windows when the app is launched, before it starts executing. The only thing you can
do is, in your installer, make the shortcut to your program with a “show command” of
SW_HIDE (ie. 0). This will only affect the console unless you deliberately honour the wShowWindow field of STARTUPINFO
in your program, so don’t do that.

I’ve tested this by hacking cygwin’s “mkshortcut.exe”. How you accomplish
it in your install program of choice is up to you.

The user can still of course run your program by finding the executable in Explorer and
double-clicking it, bypassing the console-hiding shortcut and seeing the brief black flash of a console window. There’s nothing you can do about it.