Calling conventions

The calling convention of a function answers the following questions,

• How parameters are passed into a function
  
• Who cleans up the stack on function return

The following lists common x86 function calling conventions.

Win32 (Stdcall): The parameters are pushed onto the stack from right to left. The called function cleans up the stack.

01003540 push esi
01003541 lea eax,[ebp-62Ch]
01003547 push eax
01003548 call module!func (01001058)

Native C++ (Thiscall): "this" pointer is passed via ECX; the rest of parameters are pushed onto the stack from right to left. The called function cleans up the stack.

00413586 mov eax,dword ptr [ebp+0Ch]
00413589 push eax
0041358a mov ecx,dword ptr [ebp+8]
0041358d push ecx
0041358e mov ecx,dword ptr [ebp-14h] ; store "this" to ECX
00413591 call module!func (00411505)

COM (Stdcall for C++): The parameters are pushed onto the stack from right to left, include "this" pointer, i.e. "this" pointer is pushed onto the stack as the first parameter. The called function cleans up the stack.

01002ffe mov ecx,dword ptr [eax] ; ecx="this"->lpvtbl
01003000 push ebx
01003001 push offset 01002c7c
01003006 push eax ; "this" as 1st parameter
01003007 call dword ptr [ecx] ; [ecx]=QueryInterface, [ecx+4]=AddRef, [ecx+8]=Release

Fastcall: First two parameters are passed in via ECX and EDX; the rest are pushed onto the stack from right to left. The called function cleans up the stack.

0100248e mov edx,eax ; 2nd parameter
01002490 mov ecx,edi ; 1st parameter
01002492 call module!func (01002445)

Cdecl: The parameters are pushed onto the stack from right to left. The calling function cleans up the stack.

01002490 push eax
01002491 push offset 0100118c
01002496 call module!func (010026c5)
0100249b pop ecx ; pop 1st parameter
0100249c pop ecx ; pop 2nd parameter

.NET Framework: The .NET Framework uses the fastcall calling convention. The first two paramters are passed via ECX and EDX. For an instance method, "this" pointer is passed via ECX as the first paramter.

There are three methods to dispatch a method call.

Interface-based dispatch:
mov ecx,edi ; move "this" pointer into ecx
mov eax,dword ptr [ecx] ; move "TypeHandle" into eax
mov eax,dword ptr [eax+0Ch] ; move IVMap into eax at offset 12
mov eax,dword ptr [eax+30h] ; move the ifc impl start slot into eax
call dword ptr [eax] ; call method

Direct dispatch:
mov ecx,esi ; move "this" pointer into ecx
cmp dword ptr [ecx],ecx ; compare and set flags
call dword ptr ds:[009552D8h] ; directly call method

Virtual dispatch:
mov ecx,esi ; move "this" pointer into ecx
mov eax,dword ptr [ecx] ; acquire the MethodTable address
call dword ptr [eax+44h] ; dispatch to the method at offset 0x44

(Excerpted from Drill Into .NET Framework Internals to See How the CLR Creates Runtime Objects)

Debug a service application from startup

The article -- Debugging a Service Application provides pretty detailed information on debugging a service application. However, besides adding a ProgramName registry key under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options, where ProgramName is the name of the service application's executable file, and adjusting the service application timeout by adding or modifying a DWORD value called ServicesPipeTimeout under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control, in my recent experience, I have found that the following two items should be aware of as well.

• For WinDBG GUI to pop up, Allow service to interact with desktop option should be enabled for the service application.

• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\ServicesPipeTimeout could affect the Windows startup time. For instance, when I changed its value to 86,400,000 (24 hours), it took my XP box about one day to fully start up! When I changed its value to 600,000 (10 minutes), it took my XP box about 10 minutes to fully start up.
  

Static data members in inheritance

A static data member means that one copy of data would be shared among all instances of the same class. It is true even when inheritance is involved.

class base
{
public: static wchar_t * s_pName;
};

wchar_t * base::s_pName = L"base";

class derived1 : public base { };

class derived2 : public base { };

int _tmain(int argc, _TCHAR* argv[])
{
derived1 d1;
d1.s_pName = L"derived1";

derived2 d2;
d2.s_pName = L"derived2";

wprintf(L"%s\n", d1.s_pName);
wprintf(L"%s\n", d2.s_pName);

return 0;
}

Since s_pName is a static data member in the base class, all the instances of the derived classes share one copy of the data member. Therefore, the output of the above program would be as following,

derived2
derived2.

Obviously, it is not a desirable output for the instances of derived1 class. To overcome this limitation, we could use a so-called "mixin-style" base class. I replace the angle brackets for a template with square brackets.

template[class T]
class base
{
public: static wchar_t * s_pName;
};

template[class T] wchar_t * base[T]::s_pName = L"base";

class derived1 : public base[derived1] { };

class derived2 : public base[derived2] { };

int _tmain(int argc, _TCHAR* argv[])
{
derived1 d1;
d1.s_pName = L"derived1";

derived2 d2;
d2.s_pName = L"derived2";

wprintf(L"%s\n", d1.s_pName);
wprintf(L"%s\n", d2.s_pName);

return 0;
}

The base class is turned into a template. The template part of the base class ensures that each derived class gets a different s_pName. The output from the above program would be as following,

derived1
derived2.

Why did PEM_read_bio_PrivateKey fail to load a key in FIPS mode?

My former manager told me that PEM_read_bio_PrivateKey failed to load a private key generated via OpenSSL commandline in FIPS mode. So I tried to load the key he sent to me using a simple test application. When not in FIPS mode, PEM_read_bio_PrivateKey worked fine. However, when in FIPS mode, I got the error -- "error:0608008D:digital envelope routines:EVP_DigestInit:disabled for fips". The error was consistent to what he had told me. Then I asked him whether he used any non-FIPS algorithm during the key generation. The answer was no.

Then I set a breakpoint at PEM_read_bio_PrivateKey and stepped into the function. After several rounds of code tracing, I had the following stacktrace.

0:000> kb
ChildEBP RetAddr Args to Child
0012f0d8 0fb43d0c 0fbc9ab0 0fbbef38 0012f5ec LIBEAY32!EVP_BytesToKey+0x10
0012f5cc 0fb43503 0012f5e8 0091ee40 0012f610 LIBEAY32!PEM_do_header+0xcc
0012f614 0fb46281 0012fa50 0012fa58 0012fa4c LIBEAY32!PEM_bytes_read_bio+0xe3
0012fa60 0041146d 0091c258 00000000 00000000 LIBEAY32!PEM_read_bio_PrivateKey+0x51
0012ff68 00411b18 00000001 003d4c58 003d59d8 openssl_loadkey!wmain+0x7d
0012ffb8 0041195f 0012fff0 7c817077 0115f6f2 openssl_loadkey!__tmainCRTStartup+0x1a8
0012ffc0 7c817077 0115f6f2 0115f77e 7ffdf000 openssl_loadkey!wmainCRTStartup+0xf
0012fff0 00000000 00411082 00000000 78746341 kernel32!BaseProcessStart+0x23

EVP_BytesToKey would return error. It caused PEM_read_bio_PrivateKey to fail to load the key. The second parameter of EVP_BytesToKey is a pointer to EVP_MD.

0:000> ln 0fbbef38
(0fbbef38) LIBEAY32!md5_md | (0fbbef80) LIBEAY32!sha_md
Exact matches:
LIBEAY32!md5_md = struct env_md_st

For some reason, MD5 was passed into the function. Since MD5 is a non-FIPS algorithm, in FIPS mode, it would be disabled in OpenSSL library. Then the question was where MD5 came from. When looking at the OpenSSL source code, I found that when a key file contains a header, PEM_read_bio_PrivateKey would read the header and retrieve the encryption cipher, then it would try to generate the key using EVP_BytesToKey by passing MD5 as its second parameter. Since MD5 is hard-coded as the second parameter, any key file containing a header which specifies an encryption cipher would fail to be loaded in FIPS mode. It is definitely a defect in the OpenSSL code. Until the fix, the workaround is not to generate a key with a header containing an encryption cipher. The OpenSSL versions we used in testing are openssl-0.9.7m and openssl-0.9.8j.