ActiveX, installation doesn’t work

The problem might be that you try to run msiexec.exe and this exe is not in the cab file. See this question (be sure to scroll down to the incredibly helpful sample code posted by Roey 5 Aug 2009).
Try to either create a setup.exe that will run process msiexec.exe and install your msi or make an installer with a bootstrap setup.exe file and include both in the cab.

Also, you might want to read about non-admin activex installations.

Your activex dll has to be signed and your activeX should implement this interface

/// <summary>
/// Options supported for the IObjectSafety interface 
/// </summary>
[Serializable]
[ComVisible(true)]
public enum ObjectSafetyOptions
{
    /// <summary>
    /// Indicates that the caller of the interface identified by riid might be untrusted.
    /// </summary>
    INTERFACESAFE_FOR_UNTRUSTED_CALLER = 0x00000001,
    /// <summary>
    /// Indicates that the data passed into the interface identified by riid might be untrusted.
    /// </summary>
    INTERFACESAFE_FOR_UNTRUSTED_DATA = 0x00000002,
    /// <summary>
    /// Indicates that the caller of the interface identified by riid knows to use IDispatchEx.
    /// </summary>
    INTERFACE_USES_DISPEX = 0x00000004,
    /// <summary>
    /// Indicates that the data passed into the interface identified by riid knows to use IInternetHostSecurityManager.
    /// </summary>
    INTERFACE_USES_SECURITY_MANAGER = 0x00000008
};

/// <summary>
/// Provides methods to get and set safety options.
/// The IObjectSafety interface should be implemented by objects that have interfaces which support "untrusted" clients, such as scripts.
/// It allows the owner of the object to specify which interfaces must be protected from "untrusted" use.
/// </summary>
[ComImport()]
[Guid("CB5BDC81-93C1-11CF-8F20-00805F2CD064")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IObjectSafety
{
    /// <summary>
    /// Gets the safety options supported by an object and the safety options that are currently set for that object.
    /// </summary>
    /// <param name="iid">An interface identifier for a given object</param>
    /// <param name="pdwSupportedOptions">Receives the address of a DWORD representing all the options supported for the interface identified by riid.</param>
    /// <param name="pdwEnabledOptions">Receives the address of a DWORD representing all the options currently enabled for the interface identified by riid.</param>
    /// <returns>Returns one of the following values:
    /// S_OK - the object is safe for loading
    /// E_NOINTERFACE - the riid parameter specifies an interface that is unknown to the object</returns>
    [PreserveSig]
    long GetInterfaceSafetyOptions(ref Guid iid, out int pdwSupportedOptions, out int pdwEnabledOptions);

    /// <summary>
    /// Returns whether an object is safe for initialization or scripting, as specified.
    /// </summary>
    /// <param name="iid">An iInterface identifier for the object to be made safe.</param>
    /// <param name="dwOptionSetMask">A mask representing the options to be validated.</param>
    /// <param name="dwEnabledOptions">A DWORD representing all the options currently enabled for the interface identified by riid. </param>
    /// <returns>Returns one of the following values:
    /// S_OK - the object is safe for loading
    /// E_NOINTERFACE - the riid parameter specifies an interface that is unknown to the object
    /// E_FAIL - the dwOptionSetMask parameter specifies an option that is not supported by the object</returns>
    [PreserveSig]
    long SetInterfaceSafetyOptions(ref Guid iid, int dwOptionSetMask, int dwEnabledOptions);
};

I used advanced installer to create msi installer and I set dll registration. Advanced installer then generated all necessary registry keys.

In my test the final inf file was

[version]
signature="$CHICAGO$"
AdvancedINF=2.0
[Deployment]
InstallScope=user|machine
[Setup Hooks]
install=install
[install]
run="""%EXTRACT_DIR%\runmsi.exe""" """%EXTRACT_DIR%\simpleactivex.msi"""

Please note the triple quotes. They are important.

I used this ddl

.Set DiskDirectoryTemplate=.
.Set CabinetNameTemplate=simpleactivex.cab
runmsi.exe
simpleactivex.msi
simpleactivex.inf

And I build cab using this commands

"c:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\signtool.exe" sign /sha1 9A15DC8F51773C557BA2F75CF155F8CBD367A8E1 /tr http://tsa/tsa /d SimpleActiveX /du "http://yourcompany.com" /v runmsi.exe simpleactivex.msi

makecab /V3 /F make.ddl

"c:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\signtool.exe" sign /sha1 9A15DC8F51773C557BA2F75CF155F8CBD367A8E1 /tr http://tsa/tsa /d simpleactivex /du "http://yourcompany.com" /v simpleactivex.cab

runmsi.exe is a dummy exe file that runs msiexec with given parameters. Alternatively you could use exe installer or bootstrap exe and msi installer. The important part to note is that IE will not allow to run anything outside the cab file. Therefore you have to do this hack.

When debugging I used this dummy html page

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" > 

<html>
  <head>
    <title>WebForm1</title>
  </head>
  <body style="margin-top: 0px; margin-left: 0px;">
  <OBJECT id="SimpleActiveXCtrl" classid="clsid:C0082E22-8A19-4600-8332-D31C4055291A" codebase="SimpleActiveX.CAB"></OBJECT>    

<script language="javascript">
    // da sa volat z JS metody ActiveXu alebo nastavit property
function OpenActiveX()
{
    try
    {
        alert(document.SimpleActiveXCtrl.HelloWorld("hello"));
    }
    catch(Err)
    {
        alert(Err.description);
    }
}   


</script>

  </body>
</html>