Not quite sure what was causing the problem, I deleted my bin folder and then ran a build and it now seems to be working. System.Diagnostics.Debugger.Launch()
does work correctly it needs to be contained within an #if DEBUG
as @Stein Åsmul stated. Once built in DEBUG run the outputed .msi, you will be prompted to open an instance of visual studio when you hit your custom action during install.
[CustomAction]
public static ActionResult CustomAction(Session session)
{
#if DEBUG
System.Diagnostics.Debugger.Launch();
#endif
MessageBox.Show("Hello World!" + session[IISSessions.AppPoolName], "External Managed CA");
return ActionResult.Success;
}