[EDIT: 4/2021 – new answer…]
Due to a recent change in the Visual Studio debugger, my original answer stopped working correctly when debugging. To remedy this, I’m providing an entirely different approach. The text of the original answer is included at the bottom.
1. Just the code, please…
To determine if a .NET application is running in GUI mode:
[DllImport("kernel32.dll")] static extern IntPtr GetModuleHandleW(IntPtr _);
public static bool IsGui
{
get
{
var p = GetModuleHandleW(default);
return Marshal.ReadInt16(p, Marshal.ReadInt32(p, 0x3C) + 0x5C) == 2;
}
}
This checks the Subsystem value in the PE header. For a console application, the value will be 3 instead of 2.
2. Discussion
As noted in a related question, the most reliable indicator of GUI vs. console is the “Subsystem” field in the PE header of the executable image. The following C# enum lists the allowable (documented) values:
public enum Subsystem : ushort
{
Unknown /**/ = 0x0000,
Native /**/ = 0x0001,
WindowsGui /**/ = 0x0002,
WindowsCui /**/ = 0x0003,
OS2Cui /**/ = 0x0005,
PosixCui /**/ = 0x0007,
NativeWindows /**/ = 0x0008,
WindowsCEGui /**/ = 0x0009,
EfiApplication /**/ = 0x000A,
EfiBootServiceDriver /**/ = 0x000B,
EfiRuntimeDriver /**/ = 0x000C,
EfiRom /**/ = 0x000D,
Xbox /**/ = 0x000E,
WindowsBootApplication /**/ = 0x0010,
};
As easy as that code is, our case here can be simplified. Since we are only interested in our the running process–which is necessarily loaded, it isn’t necessary to open any file or read from the disk to obtain the subsystem value. Our executable image is guaranteed to be already mapped into memory. It is simple to retrieve the base address for any loaded file image by calling the following function:
[DllImport("kernel32.dll")]
static extern IntPtr GetModuleHandleW(IntPtr lpModuleName);
Although we might provide a filename to this function, again things are easier and we don’t have to. Passing null, or in this case, default(IntPtr.Zero) (which is the same as IntPtr.Zero), returns the base address of the virtual memory image for the current process. This eliminates the extra steps (alluded to earlier) of having to fetch the entry assembly and its Location property, etc. Without further ado, here is the new and simplified code:
static Subsystem GetSubsystem()
{
var p = GetModuleHandleW(default); // PE image VM mapped base address
p += Marshal.ReadInt32(p, 0x3C); // RVA of COFF/PE within DOS header
return (Subsystem)Marshal.ReadInt16(p + 0x5C); // PE offset to 'Subsystem' value
}
public static bool IsGui => GetSubsystem() == Subsystem.WindowsGui;
public static bool IsConsole => GetSubsystem() == Subsystem.WindowsCui;
[end of official answer]
3. Bonus Discussion
For the purposes of .NET, Subsystem is perhaps the most—or only—useful piece of information in the PE Header. But depending on your tolerance for minutiae, there could be other invaluable tidbits, and it’s easy to use the technique just described to retrieve additional interesting data.
Obviously, by changing the final field offset (0x5C) used earlier, you can access other fields in the COFF or PE header. The next snippet illustrates this for Subsystem (as above) plus three additional fields with their respective offsets.
NOTE: To reduce clutter, the
enumdeclarations used in the following can be found here
var p = GetModuleHandleW(default); // PE image VM mapped base address
p += Marshal.ReadInt32(p, 0x3C); // RVA of COFF/PE within DOS header
var subsys = (Subsystem)Marshal.ReadInt16(p + 0x005C); // (same as before)
var machine = (ImageFileMachine)Marshal.ReadInt16(p + 0x0004); // new
var imgType = (ImageFileCharacteristics)Marshal.ReadInt16(p + 0x0016); // new
var dllFlags = (DllCharacteristics)Marshal.ReadInt16(p + 0x005E); // new
// ... etc.
To improve things when accessing multiple fields in unmanaged memory, it’s essential to define an overlaying struct. This allows for direct and natural managed access using C#. For the running example, I merged the adjacent COFF and PE headers together into the following C# struct definition, and only included the four fields we deemed interesting:
[StructLayout(LayoutKind.Explicit)]
struct COFF_PE
{
[FieldOffset(0x04)] public ImageFileMachine MachineType;
[FieldOffset(0x16)] public ImageFileCharacteristics Characteristics;
[FieldOffset(0x5C)] public Subsystem Subsystem;
[FieldOffset(0x5E)] public DllCharacteristics DllCharacteristics;
};
NOTE: A fuller version of this struct, without the omitted fields, can be found here
Any interop struct such as this has to be properly setup at runtime, and there are many options for doing so. Ideally, its generally better to impose the struct overlay “in-situ” directly on the unmanaged memory, so that no memory copying needs to occur. To avoid prolonging the discussion here even further however, I will instead show an easier method that does involve copying.
var p = GetModuleHandleW(default);
var _pe = Marshal.PtrToStructure<COFF_PE>(p + Marshal.ReadInt32(p, 0x3C));
Trace.WriteLine($@"
MachineType: {_pe.MachineType}
Characteristics: {_pe.Characteristics}
Subsystem: {_pe.Subsystem}
DllCharacteristics: {_pe.DllCharacteristics}");
4. Output of the demo code
Here is the output when a console program is running…
MachineType: Amd64 Characteristics: ExecutableImage, LargeAddressAware Subsystem: WindowsCui (3) DllCharacteristics: HighEntropyVA, DynamicBase, NxCompatible, NoSeh, TSAware
…compared to GUI (WPF) application:
MachineType: Amd64 Characteristics: ExecutableImage, LargeAddressAware Subsystem: WindowsGui (2) DllCharacteristics: HighEntropyVA, DynamicBase, NxCompatible, NoSeh, TSAware
[OLD: original answer from 2012…]
To determine if a .NET application is running in GUI mode:
bool is_console_app = Console.OpenStandardInput(1) != Stream.Null;