What compiler uses a calling convention that uses EAX as the first argument, then pushes onto the stack?

Question

I recently asked to question: How can an IA-32 program seemingly compiled with MSVC send its sole argument via EAX? After posting the question, I found that another function passed the first argument in EAX and then pushed its remaining argument. The caller then cleans up the stack.

The calling code:

.text:00402465                 lea     eax, [ebp+var_4]
    ...
.text:00402469                 push    eax
.text:0040246A                 mov     eax, [ebp+hWnd]
.text:0040246D                 call    openFileDialog
.text:00402472                 add     esp, 4

And the function itself:

.text:00411730 openFileDialog  proc near
.text:00411730
    ...
.text:00411730 arg_0           = dword ptr  8
.text:00411730
.text:00411730                 push    ebp
.text:00411731                 mov     ebp, esp
.text:00411733                 sub     esp, 18h
.text:00411736                 cmp     byte_42AE1D, FALSE
.text:0041173D                 push    ebx
.text:0041173E                 push    esi
.text:0041173F                 push    edi
.text:00411740                 mov     esi, eax
    ...
.text:00411789                 mov     eax, [ebp+arg_0]
.text:0041178C                 push    eax
.text:0041178D                 push    esi
.text:0041178E                 call    openFileDialog_Compat
.text:00411793                 add     esp, 8

As you can see, in the function, the value of EAX is saved before anything can affect it, so it is definitely being used as a parameter. Later, the pushed argument is passed to a normal __cdecl function.

The program is linked to use msvcr100.dll and uses MSVC style throughout (Such as __security_cookie, MSVC name mangling, etc.), so it would appear to have been compiled with Visual C++, but this unusual calling convention makes me question that.

When using "Whole Program Optimization", the compiler may use whatever calling convention fits for functions with internal linkage. — newgre, Dec 06 '17 at 08:10

score 7 · Accepted Answer · answered Dec 06 '17 at 19:58

This is probably a program compiled with "Whole Program Optimization" or "Link-time code generation". From MSDN:

When /LTCG is used to link modules compiled with /Og, /O1, /O2, or /Ox, the following optimizations are performed:

Cross-module inlining

Interprocedural register allocation (64-bit operating systems only)

Custom calling convention (x86 only)

Small TLS displacement (x86 only)

Stack double alignment (x86 only)

Improved memory disambiguation (better interference information for global variables and input parameters)

Thank you. This is something I have not encountered before. I've only ever thought from the compiler point of view and the described standards. — Ben Jaguar Marshall, Dec 07 '17 at 01:10

score 0 · Answer 2 · edited Jun 17 '20 at 09:54

0

check Calling Conventions that msvc support yet. as for the function name openFileDialog, it's a c# func? combining C# and C++ via CLR?? strange things may happen. MS dose not need to expose these details to public.

edited Jun 17 '20 at 09:54

Community

1

answered Dec 06 '17 at 07:05

yufeng

55
3

What compiler uses a calling convention that uses EAX as the first argument, then pushes onto the stack?

2 Answers2

Related