A reader who happened across my post on Windows Installer on 64-bit Platforms mentioned a problem with running 64-bit managed custom actions using the Visual Studio 2005 Windows Installer project. This also recently cropped up in an internal discussion alias.

The issue is that if you build a managed class library project targeting a 64-bit platform using /platform:x64 or /platform:Itanium and install a Windows Installer package built in Visual Studio 2005 on a 64-bit machine a System.BadImageFormatException is thrown. The reason is because the native shim packaged with the .msi file is a 32-bit executable.

Let’s step back a minute, though, to how to build a Windows Installer setup project with managed custom actions. I won’t go into details, but basically you create a new Class Library project that contains one or more derivatives from the System.Configuration.Install.Installer class. In the Custom Actions editor for your Windows Installer project you can right-click on a specific phase (Install, Commit, Rollback, or Uninstall) or, preferably, the root node (which adds the custom action to all phases with the appropriate custom action types) and add whatever you want from project output to a specific file in your file system. If your class library is in the same solution I recommend clicking “Add Output” in the custom action editor dialog.

You should also click on the Windows Installer project and change the TargetPlatform property to either x64 or Itanium, depending on what you’re targeting. This makes sure that 64-bit components are installed to the 64-bit folders like [ProgramFiles64Folder]. If you don’t set this according to what binaries you’re installing (which can be a mix of both 32- and 64-bit) 64-bit files will be installed into [ProgramFilesFolder] which, on 64-bit platforms, is, for example, C:Program Files (x86).

Back to the problem. When you build the Windows Installer project in Visual Studio 2005 it embeds the 32-bit version of InstallUtilLib.dll into the Binary table as InstallUtil. When Windows Installer executes your managed custom action it actually is calling the ManagedInstall entry point function from InstallUtilLib.dll as a type 1 deferred custom action (1025) which creates an instance of the CCW System.Configuration.Install.IManagedInstaller interface and runs your Installer classes. Since the native InstallUtilLib.dll is 32-bit it loads the 32-bit Framework which will throw the BadImageFormatException since your managed class library is 64-bit.

To workaround this issue you either need to import the appropriate bitness of InstallUtilLib.dll into the Binary table for the InstallUtil record or – if you do have or will have 32-bit managed custom actions add it as a new record in the Binary table and adjust the CustomAction table to use the 64-bit Binary table record for 64-bit managed custom actions.

To replace the 32-bit InstallUtilLib.dll with the 64-bit bitness,

1. Open the resulting .msi in Orca from the Windows Installer SDK
2. Select the Binary table
3. Double click the cell [Binary Data] for the record InstallUtil
4. Make sure “Read binary from filename” is selected and click the Browse button
5. Browse to %WINDIR%Microsoft.NETFramework64v2.0.50727
6. Select InstallUtilLib.dll
7. Click the Open button
8. Click the OK button

Note that the Framework64 directory is only installed on 64-bit platforms and that it corresponds to the 64-bit processor type. That is, you won’t find the x64 flavor of InstallUtilLib.dll on an IA64 machine.

If you already have or anticipate having 32-bit custom actions in future patches – and I recommend this approach because the future is difficult to predict – you should add a new record.

1. Open the resulting .msi in Orca from the Windows Installer SDK
2. Select the Binary table
3. Click the Tables menu and then Add Row
4. Enter, for example, InstallUtil64 for the Name
5. Select the Data row and click the Browse button
6. Browse to %WINDIR%Microsoft.NETFramework64v2.0.50727
7. Select InstallUtilLib.dll
8. Click the Open button
9. Click the OK button
10. Select the CustomAction table
11. For each custom action where the Source column is InstallUtil and only those custom actions that are 64-bit managed custom actions (or that were built with /platform:anycpu, the default, where you want to run as 64-bit custom actions), change the value to, for example, InstallUtil64

This only affects DLLs build with /target:library. Managed EXEs will run correctly according to what platform they target.