Rundll32 only supports running DLL exports with the following signature:
void CALLBACK
EntryPoint(HWND hwnd, HINSTANCE hinst, LPSTR lpszCmdLine, int nCmdShow);
It does not support running arbitrary entry points. Since ShellExecute does not have that signature, clearly bad things will happen.
INFO: Windows Rundll and Rundll32 Interface has more info on the rundll32 interface.
If you want to do the equivelent of ShellExecute from the command line, just use start:
C:\>start "C:\Documents and Settings\admin\Desktop\tmp"