Why did DllRegisterServer return 80070006?

I tried to register an in-proc COM server using regsvr32. However, regsvr32 kept returning the error code 80070006.


Driven by the curiosity, I decided to figure out the reason. At the DOS command prompt, I typed "windbg -o cmd.exe /c regsvr32 08comps.dll" to attach WinDBG to the child process (regsvr32.exe).

Before going forward, I checked the error message for 80070006.

1:001> !error 80070006
Error code: (HRESULT) 0x80070006 (2147942406) - The handle is invalid.

Below the display of registers for each step is yanked out for clarity.

1:001> sxe ld:08comps* ; break when 08comps.dll is loaded
1:001> x 08comps!*DllRegisterServer*
009723f0 08comps!DllRegisterServer (void)
1:001> bp 08comps!DllRegisterServer ; break at DllRegisterServer

08comps!DllRegisterServer:
009723f0 mov edi,edi
009723f2 push ebp
009723f3 mov ebp,esp
009723f5 push ecx
009723f6 mov eax,dword ptr [08comps!aProxyFileList (00973040)] ds:0023:00973040={08comps!srv_ProxyFileInfo (009721d8)}
009723fb mov ecx,dword ptr [eax+4] ds:0023:009721dc={08comps!_srv_StubVtblList (00973038)}
009723fe cmp dword ptr [ecx],0 ds:0023:00973038={08comps!_ICalculatorStubVtbl (00972198)}
00972401 je 08comps!DllRegisterServer+0x25 (00972415) [br=0]
00972403 mov edx,dword ptr [08comps!aProxyFileList (00973040)] ds:0023:00973040={08comps!srv_ProxyFileInfo (009721d8)}
00972409 mov eax,dword ptr [edx+4] ds:0023:009721dc={08comps!_srv_StubVtblList (00973038)}
0097240c mov ecx,dword ptr [eax] ds:0023:00973038={08comps!_ICalculatorStubVtbl (00972198)}
0097240e mov edx,dword ptr [ecx] ds:0023:00972198={08comps!IID_ICalculator (00972200)}
00972410 mov dword ptr [ebp-4],edx ss:0023:0007e86c=7c917de9
00972413 jmp 08comps!DllRegisterServer+0x2c (0097241c)
0097241c mov eax,dword ptr [ebp-4] ss:0023:0007e86c={08comps!IID_ICalculator (00972200)}
0097241f push eax
00972420 push offset 08comps!aProxyFileList (00973040)
00972425 mov ecx,dword ptr [08comps!hProxyDll (00973360)] ds:0023:00973360=00000000 ; hProxyDll is NULL. It seems suspicious.
0097242b push ecx ; hProxyDll is the first parameter
0097242c call dword ptr [08comps!_imp__NdrDllRegisterProxy (00972058)] ds:0023:00972058={RPCRT4!NdrDllRegisterProxy (77ed3211)}

RPCRT4!NdrDllRegisterProxy:
77ed3211 mov edi,edi
77ed3213 push ebp
77ed3214 mov ebp,esp
77ed3216 sub esp,140h
77ed321c mov eax,dword ptr [RPCRT4!__security_cookie (77efb2ac)] ds:0023:77efb2ac=0000cf46
77ed3221 mov ecx,dword ptr [ebp+10h] ss:0023:0007e868={08comps!IID_ICalculator (00972200)}
77ed3224 push ebx
77ed3225 push esi
77ed3226 mov dword ptr [ebp-4],eax ss:0023:0007e854=00000000
77ed3229 mov eax,dword ptr [ebp+8] ss:0023:0007e860=00000000 ; store hProxyDLL in EAX
77ed322c xor esi,esi ; esi = 0
77ed322e cmp eax,esi ; check if hProxyDll is NULL
77ed3230 push edi
77ed3231 mov edi,dword ptr [ebp+0Ch] ss:0023:0007e864={08comps!aProxyFileList (00973040)}
77ed3234 mov dword ptr [ebp-13Ch],ecx ss:0023:0007e71c=00972f58
77ed323a mov ebx,80070000h
77ed323f je RPCRT4!NdrDllRegisterProxy+0x68 (77ed3279) [br=1] ; jump since hProxyDLL is NULL
77ed3279 mov dword ptr [ebp-134h],80070006h ss:0023:0007e724=00000000 ; store error code 80070006 in a local variable
77ed3283 xor esi,esi ; esi = 0
77ed3285 cmp dword ptr [ebp-134h],esi ss:0023:0007e724=80070006 ; check if error code is less than 0
77ed328b jl RPCRT4!NdrDllRegisterProxy+0x254 (77ed3464) [br=1] ; jump since error code is less than 0
77ed3464 mov ecx,dword ptr [ebp-4] ss:0023:0007e854=0000cf46
77ed3467 mov eax,dword ptr [ebp-134h] ss:0023:0007e724=80070006 ; store error code in EAX
77ed346d pop edi
77ed346e pop esi
77ed346f pop ebx
77ed3470 call RPCRT4!__security_check_cookie (77e7602d)
77ed3475 leave
77ed3476 ret 0Ch

From the above code tracing, it seemed that the error code 80070006 returned from DllRegisterServer was because 08comps!hProxyDll was NULL. The next question was why hProxyDll was NULL. I restarted the debugging session and set break on access to hProxyDll -- ba w4 08comps!hProxyDll. However, the code did not break. It indicated that no code ever tried to set a value to hProxyDll. Then I found out from rpcproxy.h that hProxyDll is set to the module handle in DllMain. So I tried "x 08comps!*dllmain*" and "x 08comps!*" and searched for "DllMain". However, there was no matches. It seemed 08comps.dll used DllMain provided by the runtime instead of its own. Then I checked the source code. It turned out that DLLDATA_ROUTINES macro was used. The macro does not provide the code for DllMain. To fix the issue, REGISTER_PROXY_DLL_ROUTINES macro should be used. Since it provides its own DllMain, where hProxyDll is set to the module handle.