If you have authored a custom action into your installer package and are annoyed by the console window that always pops up when the custom action runs, it’s because the custom action executable is running under the console subsystem.

When you link your object files with the VC++ linker you can specify the subsystem using the /SUBSYSTEM switch. If you’re using the C# or VB.NET compilers the /target switch allows you to specify which subsystem to use for the executable. However you specify the subsystem or however it’s determined depending on which entry point you use, you shouldn’t use the console subsystem for custom actions if you don’t want the console window to appear, unless you have a good reason for it. For example, maybe you just want to reuse a console application that you’re installing such as for custom action type 18 or any custom action using msidbCustomActionTypeExe (0x01).

In most cases you should consider running under the Windows subsystem. No window is created unless you create a window yourself and you can still allocate a console if necessary.

If you already have a console EXE you want to use, you can cause the window not to be displayed by passing certain parameters to APIs like the CreateProcess function. When calling CreateProcess pass CREATE_NO_WINDOW (0x08000000). WiX defines the CAQuietExec custom action function that does this and redirects standard handles as well.