What you said above is incorrect:
Custom actions can only be executable
files. A batch file is not executable.
Custom Actions (even without tools like InstallShield) can be .EXE, VBScript, JScript, or DLL based. If you write a DLL, you can write whatever code you want to call a batch file or make any changes you want to the system — there is no limit.
Update: an example that worked for me: (entry in CustomAction table)
Action Test
Type 34
Source SystemFolder
Target cmd.exe /c c:\test.bat
ExtendedType <blank>