Convert inline assembly code to C++

The problem isn’t solvable in general; that’s because the comment,

// call the function & handle the return value.  use __stdcall calling convention

indicates reliance on 32bit calling conventions.

In 32bit x86, stdcall means that all arguments are passed on the stack, in reverse order (i.e. the last arg is pushed first. That is, if arg[0] is at addr then arg[1], no matter it’s type, is at addr + sizeof(arg[0])). That’s the reason why the following code in your sample:

// just copy the args buffer to the stack (it's already layed out correctly)
int* begin = m_argsBegin;
int* ptr = m_argsEnd;
int arr[1000], i=0;
while (ptr > begin) {
    int val = *(--ptr);

    __asm push val
}

actually can work – because it simply doesn’t matter what exactly is there, what type the arguments are; all that’s relevant is that each and every one of them is in a known memory location, and known-to-be-consecutive-in-memory. If you know that argument N is at addr, then you can know argument N+1 is at addr + sizeof(arg[N]).

That’s what the comment says, “it’s already layed out correctly” – unfortunately, this is not true in 64bit mode. The code therefore cannot be “ported”; there is no equivalent to port to.

Calling conventions that are at least partially register based – as is Win64 on x64 (64bit x86) behave differently. For those, it depends on what types of arguments the called functions take (you can, in Windows, pass four integer-type args in general-purpose registers plus some float-type args in XMM registers). You therefore need to know more about the signature (prototype) of the function you call than simply “it takes N arguments” in order to be able to correctly marshal args from an “anycall” type wrapper like the above. In 64bit mode, for every function you’d wish to call through your wrapper, you need to know not just how many args there are in total, but also how many are in general-purpose regs, how many in XMM regs, and how many on the stack.

The “call a func through a pointer and copy the return value to a known location” part is portable and can be expressed in plain C/C++. But the part that gets the arguments for this, as per above, does not port in any straightforward way between 32bit stdcall and any 64bit x86 calling convention I know of (neither Win64/x64 nor the UN*X x86_64 conventions allow to predict both the locations and total stack memory usage of all arguments for a function given just the number and types of the args but not their ordering).

What exactly you need to do depends much more on the callers/users of the above class ExternalFunctionCall than on the small little sample of inline assembly you’ve shown. It’s particularly necessary to know how the members m_argsBegin and m_argsEnd are initialized and where. Can you provide a few more details on how the class looks like (all member variables/functions), as well as examples on its actual use ?

Leave a Comment