The Windows Installer does not recommend using SelfReg to register at install time. Instead, adding the registration to your .wxs code or capturing the registration at build time is highly recommended.
To add the registration manually, you don’t use EnsureTable, you use the COM related elements (like Class, ProgId, TypeLib). It can be tedious but will be far more robust than trying to selfreg during installation.
Unfortunately, the alternative to capture the registration during build using heat.exe (provided in the WiX toolset) does not support capturing from executables. If you are open to a commercial solution, we (FireGiant) developed an alternative to heat.exe that can capture executable registration (and much more). That advanced harvesting solution has more documentation on the FireGiant site.