Interpreting Windows Crash Reports: Diagnosing Issues in FileMaker Server
On Windows if an application like FileMaker Server crashes, you’ll find a dmp file for the crash. For example in C:\WINDOWS\MiniDump folder. Once you got the file, you wonder how to read it. It seems like the best was to use WinDbg application from Microsoft.
First you download the Windows 10 SDK, mount the disk image and run the installer. From all the options in WinSDKSetup, you only need the entry for “Debugging Tools for Windows”. All the other entries can be unchecked. So instead of installing several GB of data, only 160 MB are installed.
You can now run WinDbg (X64), the debugger version for 64-bit Windows. Use the menu command “Open Crash Dump…” In the File menu to open the dmp file. You are welcome with a screen like this:
Microsoft (R) Windows Debugger Version 10.0.18362.1 AMD64Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\Users\Christian\Desktop\Logs\fmserver_1571076915310.DMP]User Mini Dump File: Only registers, stack and portions of memory are available
Symbol search path is: srv*Executable search path is:Windows 8.1 Version 9600 MP (8 procs) Free x64Product: Server, suite: TerminalServer SingleUserTS6.3.9600.18217 (winblue_ltsb.160124-0053)Machine Name:Debug session time: Mon Oct 14 20:15:15.000 2019 (UTC + 2:00)System Uptime : not availableProcess Uptime: 0 days 0:00:55.000……………………………This dump file has an exception of interest stored in it.The stored exception information can be accessed via .ecxr.(68.1610): Access violation – code c0000005 (first/second chance not available)For analysis of this file, run !analyze -v*** WARNING: Unable to verify timestamp for ntdll.dllntdll+0x9079a:00007ffe`5061079a c3 ret
We already see this is a crash report from a Windows Server with an Access violation (highlighted for you in bold). The WinDbg window has a console line where you can enter commands. Or we just click on the analyze command in the text.
If you see “ntdll.wrong.symbols.dll” in the logs, you don’t have symbols loaded.
We run the command “.symfix c:\myCache”. Once we got internet connection and symbols fixed, the analysis command takes a bit longer and the names show up.
The full output follows here:
**************************************************** ***************************** ** Exception Analysis ** ************** **************************************************** ****************
DEBUG_FLR_EXCEPTION_CODE(c0000374) and the “.exr -1” ExceptionCode(c0000005) don’t match
KEY_VALUES_STRING: 1
Key : AV.FaultValue: Read
Key : Timeline .Process.Start.DeltaSecValue: 55
PROCESSES_ANALYSIS: 1
SERVICE_ANALYSIS: 1
STACKHASH_ANALYSIS: 1
TIMELINE_ANALYSIS: 1
Timeline: !analyze.StartName:Time: 2019-10-15T06:50:07.79ZDiff: 45292079 mSec
Timeline: Dump.CurrentName:Time: 2019-10-14T18:15:15.0ZDiff: 0 mSec
Timeline: Process.StartName:Time: 2019-10-14T18:14:20.0ZDiff: 55000 mSec
DUMP_CLASS: 2
DUMP_QUALIFIER: 400
CONTEXT: (.ecxr)rax=00000000ffffffff rbx=0000000000010027 rcx=0000003d47db3b30 rdx=0000000000800029 rsi=0000003d47db4880 rdi=0000003d47db3b50rip=00007ffe505b5bb8 rsp =0000003d4911d930 rbp=0000000000000000r8=0000000000000000 r9=0000000000008000 r10=0000003d47db4890r11=0000003d4911d67c r12=0000000000000001 r13=0000003d47d918 b0r14=0000000000000029 r15=0000003d47db3b00iopl=0 nv up ei pl nz na po nccs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206ntdll!RtlFreeHeap+0x4e8:00007ffe`505b5bb8 8b50fc mov edx,dword ptr [rax-4] ds:00000000`fffffffb=?????????Resetting default scope
FAULTING_IP:ntdll!RtlFreeHeap+4e800007ffe`505b5bb8 8b50fc mov edx,dword ptr [rax-4]
EXCEPTION_RECORD: (.exr -1)ExceptionAddress: 00007ffe505b5bb8 (ntdll!RtlFreeHeap+0x00000000 000004e8)ExceptionCode: c0000005 (Access violation)ExceptionFlags : 00000000NumberParameters: 2Parameter[0]: 0000000000000000Parameter[1]: 00000000fffffffbAttempt to read from address 00000000fffffffb
DEFAULT_BUCKET_ID: HEAP_CORRUPTION
PROCESS_NAME: fmserver.exe
FOLLOWUP_IP:ntdll!RtlFreeHeap+4e800007ffe`505b5bb8 8b50fc mov edx,dword ptr [rax-4]
READ_ADDRESS: 00000000fffffffbERROR_CODE: (NTSTATUS) 0xc0000005 – The instruction at 0x%p referenced memory at 0x%p. The operation %s could not be performed in memory.EXCEPTION_CODE: (NTSTATUS) 0xc0000374 – A heap was corrupted.EXCEPTION_CODE_STR: c0000005EXCEPTION_PARAMETER1: 0000000000000000EXCEPTION_PARAMETER2: 00000000fffffffbWATSON_BKT_PROCSTAMP: 4e5dcbf3WATSON_BKT_PROCVER: 3.0.0.10PROCESS_VER_PRODUCT: Bonjo urWATSON_BKT_MODULE: ntdll.dllWATSON_BKT_MODSTAMP: 5d6aa558WATSON_BKT_MODOFFSET: 35bb8WATSON_BKT_MODVER: 6.3.9600.19478BUILD_VERSION_STRING: 6.3.9600.18217 (winblue_ltsb.160124-0053)MODLIST_WITH_TSCHKSUM_HASH: 93af2701fca4ac53ef261c8a670b18fc371029b1MODLIST_SHA1_HASH: 90c96f3438de4cda49c9e1ab686ac4455189e52aYOU MP_FLAGS: 8DUMP_TYPE: 1ANALYSIS_SESSION_HOST: DESKTOP-CK2F9O6ANALYSIS_SESSION_TIME: 10-15-2019 08:50:07.0079ANALYSIS_VERSION: 10.0.18362.1 amd64fre
THREAD_ATTRIBUTES:ADDITIONAL_DEBUG_TEXT: Enable Pageheap/AutoVerifer ; Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD]FAULTING_THREAD: 00001610THREAD_SHA1_HASH_MOD_FUNC: e13263ae6e1446925b0795010b356214878a79e31_HASH_MOD_FUNC_OFFSET: 98972a2fd4f89208ff78e81e8a86bfe496f8ef26BUGCHECK_STR: APPLICATION_FAULT_INVALID_POINTER_READ_HEAP_CORRUPTIONPRIMARY_PROBLEM_CLASS: APPLICATION_FAULTPROBLEM_CLASSES:
ID: [0n262]Type: [HEAP_CORRUPTION]Class: PrimaryScope: DEFAULT_BUCKET_ID (Failure Bucket ID prefix)BUCKET_IDName: AddData: OmitPID: [0x68]TID: [0x1610]Frame: [0] : ntdll!RtlFreeHeap
ID: [0n262]Type: [HEAP_CORRUPTION ]Class: PrimaryScope: BUCKET_IDName: AddData: OmitPID: [0x68]TID: [0x1610]Frame: [0] : ntdll!RtlFreeHeap
ID: [0n313]Type: [@ACCESS_VIOLATION]Class: AddendumScope: BUCKET_IDName : OmitData: OmitPID: [Unspecified]TID: [0x1610]Frame: [0] : ntdll!RtlFreeHeap
ID: [0n285]Type: [INVALID_POINTER_READ]Class: PrimaryScope: BUCKET_IDName: AddData: OmitPID: [Unspecified]TID: [0x1610]Frame: [0] : ntdll!RtlFreeHeap
LAST_CONTROL_TRANSFER: from 00007ffe323869d8 to 00007ffe505b5bb8
STACK_TEXT:0000 0000`00000000 00000000`00000000 heap_corruption! fmserver.exe+0x0
THREAD_SHA1_HASH_MOD: ca4e26064d24ef7512d2e94de5a93c38dbe82fe9SYMBOL_STACK_INDEX: 0SYMBOL_NAME: heap_corruption!fmserver.exeFOLLOWUP_NAME: MachineOwnerMODULE_NAME: heap_corruptionIMAGE_NAME: heap_corruptionDEBUG_FLR_IMAGE_TIMESTAMP: 0STACK_COMMAND: ** Pseudo Context ** ManagedPseudo ** Value: 2c5ec68dae0 ** ; kbFAILURE_BUCKET_ID: HEAP_CORRUPTION_c0000005_heap_corruption!fmserver.exeBUCKET_ID: APPLICATION_FAULT_INVALID_POINTER_READ_HEAP_CORRUPTION_heap_corruption!fmserver.exeFAILURE_EXCEPTION_CODE: c0000005FAILURE_IMAGE_NAME: heap_corruptionB UCKET_ID_IMAGE_STR: heap_corruptionFAILURE_MODULE_NAME: heap_corruptionBUCKET_ID_MODULE_STR: heap_corruptionFAILURE_FUNCTION_NAME: fmserver.exeBUCKET_ID_FUNCTION_STR: fmserver.exeBUCKET_ID_OFFSET: 0BUCKET_ID_MODTIMEDATESTAMP: 0BUCKET_ID_MODCHECKSUM: 0BUCKET_ID_MODVER_STR: 0.0.0.0BUCKET_ID_PREFIX_STR: APPLICATION_FAULT_INVALID_POINTER_READ_ FAILURE_PROBLEM_CLASS: APPLICATION_FAULTFAILURE_SYMBOL_NAME: heap_corruption!fmserver.exeWATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/fmserver.exe/3.0.0.10/4e5dcbf3/ntdll.dll/6.3.9600.19478/5d6aa558/c0000005/00035bb8.htm?Retriage=1TARGET_TIME: 2019-10-14T18:15 :15.000ZOSBUILD: 9600OSSERVICEPACK: 19358SERVICEPACK_NUMBER: 0OS_REVISION: 0SUITE_MASK: 272PRODUCT_TYPE: 3OSPLATFORM_TYPE: x64OSNAME: Windows 8.1OSEDITION: Windows 8.1 Server TerminalServer SingleUserTSOS_LOCALE:USER_LCID: 0OSBUILD_TIMESTAMP: 2019-05-06 04:41:50BUILDDATESTAMP_STR: 160124-0053BUILDLAB_STR: winblue_ltsbBUILDOSVER_STR: 6.3.9600.18217ANALYSIS_SESSION_ELAPSED_TIME: 8d8ANALYSIS_SOURCE: UM_STRING: um:heap_corruption_c0000005_heap_corruption!fmserver.exeFAILURE_ID_HASH: {d7a5d9a8-6e0b-efa6-ace0-ce9b8a5e9fb6 }
What we read is about a heap corruption detected in runtime library free memory function (RtlFreeHeap).
Sadly for this crash report we can’t help as a heap corruption can happen due to overwriting memory by some other unrelated function and a buffer overrun. 15 10 19 – 08:58
