The reason for the large memory footprint is that the JIT compiler and Windows Forms engine are being loaded with your process. To reduce this, you can do the following:
[DllImport("psapi.dll")]
static extern int EmptyWorkingSet(IntPtr hwProc);
static void MinimizeFootprint()
{
EmptyWorkingSet(Process.GetCurrentProcess().Handle);
}
This should remove as much as possible from your memory footprint. There may be a way that you can also reduce the amount of memory that is set aside for runtime memory allocation.