Server issues - Isolate Issue

2021-10-14T15:06:53+00:00

Introduction

There are many reasons why BI can crash. Support often requests:

Verify all Security Software Exemptions. See Windows Tuning article for details.
Monitor RAM and CPU usage to verify these are stable and not leading to the issue.
If RAM is continuing to grow, it is often due to incorrect Security Software exclusions.
If CPU utilization is >65%, re-add your cameras using best practices.
The biggest CPU savings are dual streams on all cameras / Direct to disc recording / hardware acceleration (e.g. Intel QuickSync).
The Connect cameras correctly section in the Checklist walks through connecting cameras correctly.
Attempt to isolate any source of instability by temporarily disabling specific cameras.

This article is about how do you isolate any source of instability by temporarily disabling specific cameras.

Isolate the Issue

Recently, an update caused instability to my machine. The symptom was switching Profiles caused BI to crash.
This is often where a user stops with ticket information.

Below is the information I provided which led to resolution.
Engineering started working on the issue at 1:50 PM.
Engineering resolved the issue at 2:50 PM.

With the right information, it is easy to resolve software issues that result in a crash.

Can you provide steps to reproduce the issue?
Can you isolate the issue to a particular camera?

I started with a video showing the steps to reproduce the issue. Sometimes, there are enough clues in the video for engineering to have ideas.
This time they needed more information.

Engineering response:

Hello Sam ... same as for a customer:

See troubleshooting in help for instructions on locating/reporting error details.
Does this happen if not running as a service?
Does it happen if camera X is enabled?

My responses as I investigated to isolate the issue.

Crashes as a service. Still crashes when not running as a service.

Faulting application name: BlueIris.exe, version: 5.5.1.8, time stamp: 0x61659f5b
Faulting module name: mfc140u.dll, version: 14.26.28720.3, time stamp: 0x5e74b0d1
Exception code: 0xc0000005
Fault offset: 0x00000000002c34b0
Faulting process id: 0x1670
Faulting application start time: 0x01d7bf92f0a6f7e3
Faulting application path: C:\Program Files\Blue Iris 5\BlueIris.exe
Faulting module path: C:\Windows\SYSTEM32\mfc140u.dll
Report Id: 744bd896-188f-4447-ab99-3ba5384b2adf
Faulting package full name:
Faulting package-relative application ID:

Testing cameras. Standby.

Provided visual to engineering so EASY to understand.

With one camera. Switching from Profile 0 -> 2
I can now reproduce the issue with just one camera.

Faulting application name: BlueIris.exe, version: 5.5.1.8, time stamp: 0x61659f5b
Faulting module name: ntdll.dll, version: 10.0.19041.1202, time stamp: 0x4f115fac
Exception code: 0xc0000374
Fault offset: 0x00000000000ff199
Faulting process id: 0x1494
Faulting application start time: 0x01d7bf935d7d72f7
Faulting application path: C:\Program Files\Blue Iris 5\BlueIris.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 253e4276-0f40-4ed5-888d-3d902572255f
Faulting package full name:
Faulting package-relative application ID:

More test cases to isolate the issue: Is the Profile switch the problem or something else?
Is the Profile switch just a red herring for something else?
Confirmed the active cameras + profile change is causing the issue.

Disabling camera.
Switching profiles.
Enabling camera.
Software runs.

Turn stuff off on the one camera. Are the camera settings for this one camera causing the issue?

Camera settings: H/W Accel = Intel.
H/W Accel on/off has no impact.
Still crashes on profile switch.

Visuals always help understanding.

Suspicious of clones. Enabled non-cloned camera.
No issues switching profiles so far.

Another visual for better understanding.

Two non-cloned cameras.
No issues switching profiles.
Let me know when you can reproduce the error.

Reconfirmed my suspicion that clones are causing the crash.

Enabled bottom left camera (also a clone).
Switch from Profile 2 -> 0 caused a crash.

Faulting application name: BlueIris.exe, version: 5.5.1.8, time stamp: 0x61659f5b
Faulting module name: mfc140u.dll, version: 14.26.28720.3, time stamp: 0x5e74b0d1
Exception code: 0xc0000005
Fault offset: 0x00000000002c34b0
Faulting process id: 0xd48
Faulting application start time: 0x01d7bf94c030fef0
Faulting application path: C:\Program Files\Blue Iris 5\BlueIris.exe
Faulting module path: C:\Windows\SYSTEM32\mfc140u.dll
Report Id: 24b89ecf-c541-418f-be7f-fe3232d7f47c
Faulting package full name:
Faulting package-relative application ID:

In Event Viewer Crash Event always followed by:

Fault bucket 1677643144606994550, type 4
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: BlueIris.exe
P2: 5.5.1.8
P3: 61659f5b
P4: mfc140u.dll
P5: 14.26.28720.3
P6: 5e74b0d1
P7: c0000005
P8: 00000000002c34b0
P9:
P10:

Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERC6EA.tmp.mdmp
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERC891.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERC8A2.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERC8B0.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERC8D0.tmp.txt

These files may be available here:
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_BlueIris.exe_c687f6ab5d0f4b5206f43e9c6e677773abae78_9f5cc1ae_23a5bd04-3fd0-497a-a405-30d5b2f15853

Engineering Addresses Issue

Engineering requested BI Settings.

Some more feedback I shared:

FYI, under profile 2, the clone masters become inactive. That could be the key.
The clones however become active.

An hour later, Engineering found the issue.

I asked engineering for details on how they resolved the issue:

Good work. Thanks!

I'm assuming you had to replicate the crash before resolving the issue.
What helped you get to the root cause.
Did any of my symptoms help?
Did you simply import my registry, assign valid IP addresses to my cameras and reproduce the crash?

I want to document this example so users know what it takes to resolve crashes.

Engineering response:

I just started eliminating things.

Deleted all of the groups.
Disabled other cameras.
It was the clone.
Disabled deepstack on it then no crash

Conclusion

I was able to Isolate the issue by identifying a single camera (clone) with steps to reproduce the crash.
From there, Engineering turned off settings on the camera to identify the root cause.

The core issue was incorrectly handling static object detection when camera streams turn off.

Statistics: Posted by varghesesa — Thu Oct 14, 2021 3:06 pm