Which calling convention to use for EAX/EDX in IDA

Question

I have some code (I assume Delphi) which uses only the EAX and EDX register for passing the arguments (and of course the stack if more are required). I looked which calling conventions would match, but I haven't found one which uses only EAX and EDX. AFAIK Borland fastcall/register is using EAX and EDX, but also ECX, which is not the case here.

Can I tell IDA somehow about this calling convention? How would I do this?

Answer 1

you can add a function type by editing it (Key Y) and adding the name.

I wrote a page to remind me about calling conventions at ASM level.

Introduction

the original call is myfunc(0,1,2,3,4).

• standard order is first argument is pushed last.
• standard stack adjusting is 'callee cleanup' - after returning, the stack should be without its calling arguments.

Note: the stack looks vertically like the call order.

stdcall (stack only)

push    4
push    3
push    2
push    1
push    0
call    myfunc
xor     eax,eax
retn    10

Fastcall (ecx, edx)

This is actually Microsoft's fastcall.

push    4
push    3
push    2
mov     edx,1
xor     ecx,ecx
call    myfunc
xor     eax,eax
retn    10

CDECL & syscall (caller cleanup)

push    4
push    3
push    2
push    1
push    0
call    myfunc
add     esp,014
xor     eax,eax
retn    10

Pascal (reverse order, ebx saved, even if ebx is unused...)

push    ebx
push    0
push    1
push    2
push    3
push    4
call    myfunc
xor     eax,eax
pop     ebx
retn    10

Fortran/watcall (eax, edx, ebx, ecx, then stack - ebx is saved)

Apparently it's not so clear what the fortran calling convention is, and this one is even different from raymond's post's The __fortran calling convention isn't the calling convention used by FORTRAN.

push    ebx
push    4
mov     ecx,3
mov     ebx,2
mov     edx,1
xor     eax,eax
call    myfunc
xor     eax,eax
pop     ebx
retn    10

Delphi 'registers' calling convention (default)

Uses eax, ecx, edx as first 3 arguments. Other arguments are pushed on stack in reverse order.

push    3
push    4
mov     ecx, 2
mov     edx, 1
xor     eax,eax
call    myfunc
xor     eax,eax
retn    10

Answer 2

If you run into a calling convention which is not covered by any of the standard calling conventions you can use the __usercall or __userpurge calling convention which allows you to specify which arguments are passed where. The syntax is

return_type __usercall function_name<registers>(arg0_type arg0<registers>, arg1_type arg1<registers>, ...)

Where registers can be a grouping of registers separated by the ':' character if the argument occupies more than one register.

If you're in a really shitty spot calling convention wise you can use the full syntax as described here. Where you instead of simply typing the register name holding the argument you can describe arguments being passed in parts of registers or stack elements. The syntax is as above only with registers replaced by

<argoff:register^regoff.size>

or if the argument is passed on the stack

<argoff:^stkoff.size>

Answer 3

Delphi and Borland C++ Builder use EAX, EDX and ECX for the first three arguments in their variant of the __fastcall calling convention. So if you choose "Delphi" or "C++ Builder" in Options-Compiler, you can just use __fastcall in the function prototype - no need to resort to __usercall.