Construct a stack trace manually

When investigating a crash dump from the web server, I found the thread that caused the crash displayed a pretty wild stack trace.

0:008> kb
ChildEBP RetAddr Args to Child
WARNING: Frame IP not in any known module. Following frames may be wrong.
0191f9f9 b876f9f6 f0022608 e6022fa7 f076f9b7 0x30d5f5d
0191f9fd f0022608 e6022fa7 f076f9b7 e0022fa7 0xb876f9f6
0191fa01 e6022fa7 f076f9b7 e0022fa7 080314d8 0xf0022608
0191fa05 f076f9b7 e0022fa7 080314d8 9b0224d9 0xe6022fa7
0191fa09 e0022fa7 080314d8 9b0224d9 38000002 0xf076f9b7
0191fa0d 080314d8 9b0224d9 38000002 580191fa 0xe0022fa7
0191fa11 9b0224d9 38000002 580191fa 0076f940 0x80314d8
0191fa15 38000002 580191fa 0076f940 08000000 0x9b0224d9
0191fa19 580191fa 0076f940 08000000 780224d9 0x38000002
0191fa1d 0076f940 08000000 780224d9 18003f83 0x580191fa
0191fa21 08000000 780224d9 18003f83 000016c1 0x76f940
0191fa25 780224d9 18003f83 000016c1 50000000 0x8000000
0191fa29 18003f83 000016c1 50000000 4b0191fa 0x780224d9
0191fa2d 00000000 50000000 4b0191fa 0176f9b7 0x18003f83

To ensure that I had loaded the correct symbols, I issued the lm command to check the loaded symbol files and displayed stack traces from some other threads. Nothing seemed to be out of ordinary.

0:008> r
eax=02f8e248 ebx=02260b78 ecx=06c67458 edx=030d5f50 esi=05bbffd8 edi=a49c6c00
eip=030d5f5d esp=0191f9bf ebp=0191f9f9 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010202
030d5f5d 0000 add byte ptr [eax],al ds:0023:02f8e248=38

Among registers, while ESP and EBP looked normal, EIP seemed suspicious.

0:008> !address 030d5f5d
02ce0000 : 030d3000 - 00003000
Type 00020000 MEM_PRIVATE
Protect 00000004 PAGE_READWRITE
State 00001000 MEM_COMMIT
Usage RegionUsageHeap
Handle 008c0000

The EIP was from RegionUsageHeap instead of RegionUsageImage. It seemed even more suspicious.

0:008> !heap -x 030d5f5d
Entry User Heap Segment Size PrevSize Unused Flags
-----------------------------------------------------------------------------
030d5f38 030d5f40 008c0000 02ce0000 28 48 8 busy

0:008> dc 030d5f40 L10
030d5f40 00320030 004d0038 00430049 0053004a 0.2.8.M.I.C.J.S.
030d5f50 004a0043 0033004b 00480045 00000055 C.J.K.3.E.H.U...
030d5f60 00050003 03720073 04510c78 02ec3d90 ....s.r.x.Q..=..
030d5f70 61507265 00006567 00030005 030801ce erPage..........

0:008> du 030d5f40
030d5f40 "028MICJSCJK3EHU"

It seemed that EIP had been overwritten by a string, more specifically, the second byte of the last not-NULL character (in red).

Since the EIP had been corrupted, the debugger could not display a correct stack trace. So I had to manually construct the stack trace.

There are two ways to construct a stack trace. One is introduced in the WinDBG help or http://msdn.microsoft.com/en-us/library/cc267826.aspx, the other is in http://www.dumpanalysis.org/blog/index.php/2007/07/25/reconstructing-stack-trace-manually. The former can be used when the function call does not use the frame pointer. The latter can be used when the function call uses the frame pointer.

0:008> !teb
TEB at 7ffae000
ExceptionList: 0191fac0
StackBase: 01920000
StackLimit: 01913000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ffae000
EnvironmentPointer: 00000000
ClientId: 00000598 . 00000684
RpcHandle: 00000000
Tls Storage: 00000000
PEB Address: 7ffde000
LastErrorValue: 0
LastStatusValue: c0150008
Count Owned Locks: 0
HardErrorMode: 0

0:008> dds 01913000 01920000

[...] ; I do not need to look at the stack above ESP (0191f9bf)
0191f9bc 78260b78 mfc80!COleDocument::OnEditChangeIcon+0x6b
0191f9c0 5002260b
0191f9c4 50030d5f
0191f9c8 50030d5f
0191f9cc 02030d5f
0191f9d0 05bbffd8
0191f9d4 02a49ca8 IFtpSrvAPI!ATL::CComObject[CFTPLicense]::`scalar deleting destructor'+0x8
0191f9d8 00000000
0191f9dc 02a49beb IFtpSrvAPI!ATL::CComObject[CFTPLicense]::Release+0x1b
0191f9e0 00000001
0191f9e4 022608b8
0191f9e8 77d04239 oleaut32!VariantClear+0xb1
0191f9ec 05bbffd8
0191f9f0 022608b8
0191f9f4 022608b8
0191f9f8 0191fa1c ; last EBP and previous EBP pattern
0191f9fc 76f9f619 jscript!VAR::Clear+0x5d
0191fa00 022608b8
0191fa04 022fa7f0
0191fa08 76f9b7e6 jscript!GcAlloc::ReclaimAll+0x40
0191fa0c 022fa7f0
0191fa10 0314d8e0
0191fa14 0224d908
0191fa18 0000029b
0191fa1c 0191fa38
0191fa20 76f94058 jscript!GcContext::Reclaim+0x89
0191fa24 00000000
0191fa28 0224d908
0191fa2c 003f8378
0191fa30 0016c118
0191fa34 00000000
0191fa38 0191fa50
0191fa3c 76f9b74b jscript!IScavengerBase::UnlinkFromGc+0x5b
0191fa40 00000001
0191fa44 003f81e4
0191fa48 003f8178
0191fa4c 0224d928
0191fa50 0191fa58
0191fa54 76f98753 jscript!ScavVarList::Init+0x2d
0191fa58 0191fa80
0191fa5c 76f9b0d3 jscript!CSession::Close+0xb5
0191fa60 00000000
0191fa64 00000000
0191fa68 00000010
0191fa6c 00000000
0191fa70 0191fadc
0191fa74 003f7e78
0191fa78 00000000
0191fa7c 00000000
0191fa80 0191faa0
0191fa84 76f9b57d jscript!COleScript::Close+0x7c
0191fa88 0191fadc
0191fa8c 0191fb08
0191fa90 008ca5d8
0191fa94 0191fb08
0191fa98 008ca5d8
0191fa9c 0191fad8
0191faa0 0191fad8
0191faa4 6400189b Core!CActiveScriptEngine::~CActiveScriptEngine+0x5b
0191faa8 003f7e78
0191faac 80720457
0191fab0 0191fadc
0191fab4 0191fad0
0191fab8 008ca5d8
0191fabc 0191fb08
0191fac0 0191fba8
0191fac4 6406b521 Core!_WSAFDIsSet+0xed
0191fac8 00000002
0191facc 003635c5 WSFTPWebServer!CWebActiveScriptEngine::~CWebActiveScriptEngine+0x85
0191fad0 0191fae0
0191fad4 0191fb08
0191fad8 0191fbb4
0191fadc 0036fd70 WSFTPWebServer!CWebTcp::_ProcessScript+0x270
0191fae0 80721a46
0191fae4 0191fc40
0191fae8 0191fbcc
0191faec 0191faf8
0191faf0 0191faf8
0191faf4 00000001
0191faf8 6407638c Core!CResult::`vftable'
0191fafc 00000000
0191fb00 782b8d2c mfc80!afxStringManager+0x14
0191fb04 cccccccc
0191fb08 640762d4 Core!CActiveScriptEngine::`vftable'
0191fb0c 640762c0 Core!CActiveScriptEngine::`vftable'
0191fb10 00000000
0191fb14 781d7f94 mfc80!CMapStringToPtr::`vftable'
0191fb18 01fa7f40
[...]
0191fb58 01f75008
0191fb5c 01f75040
0191fb60 cccccccc
0191fb64 cccccccc
0191fb68 6407638c Core!CResult::`vftable'
0191fb6c 00000000
0191fb70 782b8d2c mfc80!afxStringManager+0x14
0191fb74 cccccccc
0191fb78 cccccccc
0191fb7c 6407aabc Core!CIoMemFile::`vftable'
0191fb80 00000001
0191fb84 00000000
0191fb88 6407aab0 Core!CIoMemFile::`vftable'
0191fb8c 0003440f
0191fb90 0705c008
0191fb94 0004d94c
0191fb98 0003440f
0191fb9c 00000000
0191fba0 cccccccc
0191fba4 01f75008
0191fba8 0191fc40
0191fbac 0038eed3 WSFTPWebServer!CreateErrorInfo+0x3f2d
0191fbb0 00000002
0191fbb4 0191fc4c
0191fbb8 003713d7 WSFTPWebServer!CWebTcp::_ProcessFile+0x1e7
0191fbbc 0191fc88
0191fbc0 05ee8700
[...]
0191fbfc 00000000
0191fc00 00000001
0191fc04 cccccccc
0191fc08 781d7db8 mfc80!CStringArray::`vftable'
0191fc0c 02174368
0191fc10 00000003
0191fc14 00000003
0191fc18 00000000
0191fc1c cccccccc
0191fc20 00000001
0191fc24 06c6757e
0191fc28 cccccccc
0191fc2c 6407638c Core!CResult::`vftable'
0191fc30 00000000
0191fc34 782b8d2c mfc80!afxStringManager+0x14
0191fc38 cccccccc
0191fc3c 01f75008
0191fc40 0191fc9c
0191fc44 0038f285 WSFTPWebServer!CreateErrorInfo+0x42df
0191fc48 00000003
0191fc4c 0191fca8
0191fc50 003710b4 WSFTPWebServer!CWebTcp::ProcessFile+0x64
0191fc54 0191fc88
0191fc58 05ee8700
[...]
0191fc80 cccccccc
0191fc84 cccccccc
0191fc88 6407638c Core!CResult::`vftable'
0191fc8c 00000000
0191fc90 782b8d2c mfc80!afxStringManager+0x14
0191fc94 cccccccc
0191fc98 01f75008
0191fc9c 0191fcfc
0191fca0 0038f21d WSFTPWebServer!CreateErrorInfo+0x4277
0191fca4 ffffffff
0191fca8 0191fd08
0191fcac 0036f94a WSFTPWebServer!CWebTcp::_ProcessRequest+0x30a
0191fcb0 0191fd6c
0191fcb4 05ee8700
[...]
0191fcf8 01f75008
0191fcfc 0191ff04
0191fd00 0038ee4b WSFTPWebServer!CreateErrorInfo+0x3ea5
0191fd04 00000002
0191fd08 0191ff10
0191fd0c 003696b5 WSFTPWebServer!CWebTcp::DoTask+0x6e5
0191fd10 0191fd6c
0191fd14 05ee8700
[...]
0191fdc8 03037ba0
0191fdcc cccccccc
0191fdd0 cccccccc
0191fdd4 6407638c Core!CResult::`vftable'
0191fdd8 00000000
0191fddc 782b8d2c mfc80!afxStringManager+0x14
0191fde0 cccccccc
0191fde4 cccccccc
0191fde8 640763ac Core!CNetAddress::`vftable'
0191fdec 00000001
0191fdf0 dc070002
0191fdf4 25c4a8c0
[...]
0191fedc cccccccc
0191fee0 05ee8700
0191fee4 cccccccc
0191fee8 cccccccc
0191feec 003af0e0 WSFTPWebServer!oWebCounters
0191fef0 cccccccc
0191fef4 003af0e0 WSFTPWebServer!oWebCounters
0191fef8 00a9f0aa
0191fefc 01f75018
0191ff00 80721ee2
0191ff04 0191ff40
0191ff08 0038e5e6 WSFTPWebServer!CreateErrorInfo+0x3640
0191ff0c 00000009
0191ff10 0191ff4c
0191ff14 6404cd10 Core!CThreadPoolThread::_Run+0x80
0191ff18 807201ab
0191ff1c 00000000
0191ff20 008ca5b0
0191ff24 008ca5d8
0191ff28 ffffffff
0191ff2c 7c829f59 ntdll!RtlFreeHeap+0x70f
0191ff30 78134c39 msvcr80!free+0xcd
0191ff34 01f75018
0191ff38 008ca5b0
0191ff3c 0191ff18
0191ff40 0191ff6c
0191ff44 640722f0 Core!_WSAFDIsSet+0x6ebc
0191ff48 00000000
0191ff4c 0191ff78
0191ff50 6405b17e Core!CWorkerThread::_ThreadProc+0x3e
0191ff54 8072019f
0191ff58 00000000
0191ff5c 00000000
0191ff60 008ca5d8
0191ff64 0191ffb0
0191ff68 0191ff54
0191ff6c 0191ffa0
0191ff70 64073650 Core!_WSAFDIsSet+0x821c
0191ff74 00000000
0191ff78 0191ffb0
0191ff7c 781329bb msvcr80!_endthreadex+0x3b
0191ff80 008ca5b0
0191ff84 807d38bd
0191ff88 00000000
0191ff8c 00000000
0191ff90 008ca5d8
0191ff94 c0000096
0191ff98 0191ff84
0191ff9c 0191f5d8
0191ffa0 0191ffdc
0191ffa4 78138ced msvcr80!_except_handler4
0191ffa8 f9f78cb5
0191ffac 00000000
0191ffb0 0191ffec
0191ffb4 78132a47 msvcr80!_endthreadex+0xc7
0191ffb8 00000000
0191ffbc 77e64829 kernel32!BaseThreadStart+0x34
0191ffc0 008ca5d8
0191ffc4 00000000
0191ffc8 00000000
0191ffcc 008ca5d8
0191ffd0 c0000096
0191ffd4 0191ffc4
0191ffd8 0191f5f0
0191ffdc ffffffff
0191ffe0 77e61a60 kernel32!_except_handler3
0191ffe4 77e64830 kernel32!`string'+0x98
0191ffe8 00000000
0191ffec 00000000
0191fff0 00000000
0191fff4 781329e1 msvcr80!_endthreadex+0x61
0191fff8 008ca5d8
0191fffc 00000000
01920000 ????????

By following the pattern among EBP, previous EBP, and return address, I constructed the stack trace up to jscript!VAR::Clear+0x5d. Beyond that, the pattern could not be applied anymore. Since not all the function calls saved the frame pointer, I had to manually unassemble backward the return address to see if there was a call to the function above it.

0:008> ub 76f9f619 l1
jscript!VAR::Clear+0x3d:
76f9f613 ff155411f976 call dword ptr [jscript!_imp__VariantClear (76f91154)]

0:008> ub 77d04239 l1
oleaut32!VariantClear+0xae:
77d04236 ff5108 call dword ptr [ecx+8]

0:008> ub 02a49beb l1
IFtpSrvAPI!ATL::CComObject[CFTPLicense]::Release+0x19
02a49be9 ffd2 call edx

0:008> ub 02a49ca8 l1
IFtpSrvAPI!ATL::CComObject[CFTPLicense]::`scalar deleting destructor'+0x3:
02a49ca3 e898ffffff call IFtpSrvAPI!ATL::CComObject[CFTPLicense]::~CComObject[CFTPLicense] (02a49c40)

Therefore, mfc80!COleDocument::OnEditChangeIcon+0x6b did not belong to the stack trace. The constructed stack trace was as following,

0191f9d0 02a49ca8 IFtpSrvAPI!ATL::CComObject[CFTPLicense]::~CComObject[CFTPLicense]
0191f9d8 02a49beb IFtpSrvAPI!ATL::CComObject[CFTPLicense]::`scalar deleting destructor'+0x8
0191f9e4 77d04239 IFtpSrvAPI!ATL::CComObject[CFTPLicense]::Release+0x1b
0191f9f8 76f9f619 oleaut32!VariantClear+0xb1
0191fa1c 76f94058 jscript!VAR::Clear+0x5d
0191fa38 76f9b74b jscript!GcContext::Reclaim+0x89
0191fa50 76f98753 jscript!IScavengerBase::UnlinkFromGc+0x5b
0191fa58 76f9b0d3 jscript!ScavVarList::Init+0x2d
0191fa80 76f9b57d jscript!CSession::Close+0xb5
0191faa0 6400189b jscript!COleScript::Close+0x7c
0191fad8 0036fd70 Core!CActiveScriptEngine::~CActiveScriptEngine+0x5b
0191fbb4 003713d7 WSFTPWebServer!CWebTcp::_ProcessScript+0x270
0191fc4c 003710b4 WSFTPWebServer!CWebTcp::_ProcessFile+0x1e7
0191fca8 0036f94a WSFTPWebServer!CWebTcp::ProcessFile+0x64
0191fd08 003696b5 WSFTPWebServer!CWebTcp::_ProcessRequest+0x30a
0191ff10 6404cd10 WSFTPWebServer!CWebTcp::DoTask+0x6e5
0191ff4c 6405b17e Core!CThreadPoolThread::_Run+0x80
0191ff78 781329bb Core!CWorkerThread::_ThreadProc+0x3e

From the stack trace, the crash seemed to be another case of Invalid instruction pointer.