Isolate the issue

2021-12-08T06:16:33+00:00

Introduction

This article provides a framework on how to isolate the issue so BI support can replicate the problem and resolve the root cause. This is the final step in resolving instability as described in the Instability and Troubleshooting article.

Isolate the issue

Reinstall the version causing instability if you reverted back to an older version. Below is an overview of the key components to getting to resolution.

Verify all Security Software Exemptions.
Monitor RAM and CPU usage to verify these are stable leading to the issue.
Attempt to isolate any source of instability by temporarily disabling specific camera, functionality etc.

Windows or Third Party Software
Are updates to Windows or other third party software the issue.

The biggest question to ask yourself when observing instability is what has changed?

Windows update
Was there a Windows update?
While most users (including myself) never suspect a Windows update can cause issues, it is very possible. Common ticket:

I have a blue iris setup, exclusive windows host running for the last couple years without issue. In the last few weeks it has been totally unreliable. I have 3 times wiped the computer and started from scratch, once replacing the hard drive.

The fix was rolling back a Windows update.

Yesterday, I did an uninstall of the latest windows updates. The computer has since been running fine for the last 6 hours or so, versus crashing every15 minutes.

Another testimonial from another user.

This user had RAM issues after an update.

UH-Oooo what happened ? this was before the latest, Ver xxx? I’ve never ran over 4 gig RAM. Hey this all started as a CPU thing.. we should have never said RAM. (HAHA).

What I am seeing is something happens and the RAM grows a bit.. then it falls back but stabilizes at a higher amount. Then something happens again and it rises,, then it falls back to a higher amount again..

OK about my memory issue.. seems it was a windows update fault?
The KB5005539 (.net update) I think was the issue, and strangely KB5006670 seems to have fixed it.

If you suspect a Windows update is causing instability, use System Restore to remove the update.
See Windows Tools article for details.

Anti-virus & Firewall
Was there an update to your Antivirus or Firewall software?

Common framework

This section explains how to isolate BI functionality in order to identify root cause. A detailed description of your setup is not isolating the issue. The cool part is you can leverage BI functionality to isolate any source of instability by temporarily disabling specific camera(s), functionality etc.

Rule out hardware acceleration (HA) by turning it off
Rule out working cameras
Rule out camera features that are working fine.

Hardware acceleration
The usual culprit. Turn off hardware acceleration for all cameras.
Global Settings -> Cameras tab. Set Hardware accelerated decode = No.
Do the same for each camera. Camera settings -> Video tab.

Does stability return? If so, scroll down to Isolate Camera & HA section to determine whether there is HA incompatibility with all cameras or some cameras.

Isolate camera(s)
At this point, server is still crashing even though HA is disabled.
If not, skip to Isolate Camera & HA section below.

New camera
Did you add a new camera? Maybe the conflict is with the hardware and that one particular camera.
Simply disable the camera. Does stability return?

If the software crashes too soon after startup, you may need to leverage the registry to isolate the problematic camera.

Camera feeds
This section determines whether the issue is with the video feeds coming from the camera or features in BI.
Turn the shield to red. This turns off everything besides streaming. If the server still crashes it is one or more camera feeds.
While the server is down, use the registry to disable all the cameras.
Start the server and re-enable cameras one by one to identify the offending camera(s).

For the offending camera(s), go through all the settings on the camera and self-correct by simplifying the streams.
The Camera setup article has details.

BI Camera settings
If the server is stable with the shield red, the camera feeds are fine.
If the video feed is not causing the crash, then it must be the camera settings on a camera(s).
Time to identify the offending camera(s) and the camera setting(s) in BI causing instability.

Disable all the cameras.
Turn Shield to green.
Enable cameras one by one. Isolate the bad camera(s)

Restart BI if crashed.
Isolate the bad setting(s) on the camera(s) that causes the crash by turning features off, e.g. DeepStack.
The issue often lies with settings in the Trigger, Record or Alerts tab in Camera settings.
Report issue to support. See Submitting ticket.

Isolate Camera & HA
At this point, we know HA is causing crashes. However, we have not identified an offending camera.
See if incompatibility between HA and a camera(s) is causing the instability.
If you turned off HA for all cameras and stability returned, follow steps below.

Disable all your cameras.
Turn HA back on for all the cameras at the global level.
Global Settings -> Cameras tab -> Hardware accelerated decode. Choose appropriate setting.
Enable HA for each camera one by one and confirm stability.

The camera(s) that causes instability is the culprit. Turn off HA for those camera(s).

If all the cameras are unstable, HA is the problem. Turn HA off for all the cameras.

Crashes are frequently due to updated hardware drivers that introduced bugs. If HA is suddenly causing instability for all cameras, was there a Windows update recently? If a Windows update is the issue, then you may need to leave hardware acceleration (HA) off until there is a Windows resolution. Or you may have to revert back to the previous Windows version. See System Restore section in Windows Tools article.

Additional questions

The common framework helps identify common causes for instability.

Is hardware acceleration causing instability?
Which camera(s) are causing the issue?

Did you recently add a camera?
If you turn off cameras from a particular vendor, does stability return?

If a camera causes instability, is the issue with the video feed?
If not, is the issue with a particular camera setting? What is the root cause?

However, many issues are due to the combination of features plus your environment. Below are additional questions to ask yourself to get to the root cause.

Think of Blue Iris as blocks or modules that work together. The point of below questions is NOT to answer every question. The point of below is to help you think of relevant questions and investigate further to identify how to reproduce the error.

If stability does not return after turning the shield to red what else is on? This is a big clue.
Are you streaming to YouTube or website?
Are you streaming to your Home Automation system?
Are you posting to a website?
Are there inputs to BI besides cameras? Any DIO devices?
Is auto-cycle on? Group cycle or camera cycle? Turn it off. Does stability improve?
Are you uploading files to the cloud via FTP?
Are you constantly streaming video to a Web Interface, iPad, via mqtt or Webcasting? Does stability improve if you turn it off?
Do you notice crashes when someone connects remotely via the web interface or mobile apps?

In summary, we love customer support that leads to the narrowing down of the problem (usually with reproducible steps) so engineering can then determine where in the software to look and identify the problem.

Submitting ticket

Support info: Include support info in body of email. Global settings -> About tab -> Email support button. Copy/paste information in body of email.
Describe the issue.

Were you able to isolate the issue? Can you reproduce the crash? If so, what are the reproducible steps?

Server health

Confirm any and all errors in the logs were addressed and resolved prior to submitting a ticket.
Logs from previous month. Need to see a full month of logs.
Camera stats. Status button -> Cameras tab -> Export
Storage. Screenshot of Status button -> Storage tab.

Windows Information

Include the event viewer crash info. The Troubleshooting and FAQ section in Help provides details. Copy / paste information into body of email. Any other clues seen in Event viewer such as other frequent Errors.

Statistics: Posted by varghesesa — Wed Dec 08, 2021 6:16 am