Status log errors

varghesesa · Post by **varghesesa** » Tue Feb 23, 2021 5:25 pm

Introduction

The Status -> log provides the clues as to what may be causing issues with Blue Iris. This article provides a list of problems/error messages with an explanation of the underlying issue and the remedy.

The Status->Log tab shares good summary information regarding the health of the server. Keep in mind the log tab and the log file are two separate things. The log file contains a chronological list of all events executed on the server. See details on the differences between the log tab and the log file here and when to use either.

The errors are divided into categories:

DeepStack
Storage
Database
Recording
GPU
Memory

DeepStack

The DeepStack sub-forum has many articles to get DeepStack running correctly.

The DeepStack + Blue Iris article is a great resource to understand the DeepStack integration and get it running.
The DeepStack Gotchas article documents learnings from past tickets.

DeepStack: Server Error 100

: deepstack server error.PNG (14.77 KiB) Viewed 23332 times

The DeepStack Server Error 100 means DeepStack is responding with a 100 error instead of processing the image.

A common culprit is DeepStack is NOT responding by the timeout. Global settings -> AI tab.

: status log errors_deepstack_server 100 error.png (2.35 KiB) Viewed 21237 times

Gotcha 14 has more details. Annecdotal from a customer.

I had, when running DS in my virtualized win10 without gpu/tpu, a lot of DeepStack 100 errors under "cancelled" alerts. Nothing found on the web about any possible reasons. Then, looking in the logs, I found that DS was taking more time than the timeout I set, so this error 100 is caused by no reply in the given time, I suppose.

Other suggestions for next steps:

Restart BI
If DeepStack was working before, consider reinstalling DeepStack.
Troubleshoot your DeepStack server by isolating the BI server from the DeepStack server by running each independently. The "Run DeepStack Manually to Confirm Installation" section in the DeepStack article provides details. See article link above.

DeepStack: Not running or unreachable

: deepstack unreachable.PNG (32.01 KiB) Viewed 23247 times

Issue: Error message is obvious. BI cannot talk to DeepStack

Fix:

Double check your server settings. Global settings -> AI. Confirm the server address and port number.
If your settings are correct, you probably installed the wrong version of the software from DeepStack.
See the Gotcha in the DeepStack article -> DeepStack AI Installation section.

Storage

Storage errors pertain to the Clips and archiving settings. These errors usually pop-up when BI is dependent on a NAS drive or an external drive which is causing problems. This is best illustrated with an example.

The Storage / Record Issues article/webinar walks through examining log files to identify root cause. The logs start with MoveFile errors, i.e. moving files from one drive to another based on your Clips and archiving tab was having issues. Eventually the New folder filled up resulting in Clip write errors (Clip: Disk full).

If you find errors, try to troubleshoot yourself first. Each error provides next steps / fixes. In addition, make sure you are following best practices as well. See Storage best practices in Windows Tuning article.

Move / Delete events

Code: Select all

0 	7/19/2021 7:32:01.679 PM		New                 	Move: 2 items 3.66G [602.1G/600.0G, 326.2G free] 126 locked
0 	7/19/2021 7:32:01.788 PM		Stored              	Delete: 2 items 4.24G [2.93T/2.92T, 1.99T free]

The Move event is saying 2 recordings were moved from New to Stored (assuming Stored based on 2 items were moved which then resulted in 2 items being deleted in Stored to make room for the files moved from New).
602.1/600 means 600 GB was allocated and the folder is currently over by 2.1 GB thus the move.
The drive holding New has 326GB free which is good since New and the Database are often stored on the same drive.
126 locked means you have 126 locked files in BI.
Click on Help button in the Storage tab of the Status button for more information.

MoveFile Errors

: MoveFile Error.JPG (32.68 KiB) Viewed 24228 times

MoveFile Error 0
BI is having problems accessing/copying files to a drive and is receiving a Windows error. This is usually due to NAS permissions. Check the NAS login exactly matches the username/password used by the account running BI or the service account running BI as a service.
Confirm Windows is not in conflict with BI by walking through checks in the Windows Tuning article.

MoveFile Error 2
The database has records of files that no longer exist on your Windows filesystem. This is usually due to users accessing / deleting files manually.
Fix:

A database repair/regenerate will cause BI to rescan the storage folders and synchronize. From the clip list, right click -> Database -> Repair/Regenerate.
If the warnings continue, there may be issues with your hard drive. Monitor logs for Record/Storage errors. See routine maintenance section for best practices.

MoveFile Error 3
Similar to above, Error 3 means Folder is not found. The usual issue is an external drive such as a USB drive or a NAS device has lost connection to the BI server. Double-check all your external hard drives are functional and accessible by BI.
Another common cause is you renamed BI storage directories through Windows Explorer instead of the Clips and archiving tab which is a major no, no.
Fix: Go through your Clips and archiving tab and confirm all drives that are being used by BI exist and are functional.

Check your Clips and archiving settings so that you are not over-allocating disk space. The Status log -> Clip Storage tab provides a quick visual if storage was allocated correctly. There should be no red regions. Click on the Help button within the Clips and archiving dialog to get dialog specific help. The Storage and Recordings webinar also walks through these settings. YouTube link.
Are you running BI as a service? Running the service under the LocalSystem account can cause issues. Disable and re-enable the service on settings/startup to run with your own local Windows account (w/ admin privileges). See Windows Tuning article for details.
Are you using a NAS device for storage? Are you also running BI as a service? If so, the Windows username/password used to run the service has to be identical to the credentials on the NAS device otherwise writing to the NAS device will not work. See Windows Tuning article for details.
Double check all security software exemptions for BlueIris.exe and the recording folders. See Windows Tuning article for details.

NAS Access Errors
When BI identifies NAS access errors, common error codes returned by the NAS device are:

Code: Select all

ERROR_LOGON_FAILURE              	1326L
ERROR_WRONG_PASSWORD             	1323L
ERROR_GEN_FAILURE          		31L 	// A device attached to the system is not functioning
ERROR_SHARING_VIOLATION          	32L	// BI cannot access the file because it is being used by another process
ERROR_REQUEST_ABORTED            	1235L	// The request was aborted by the NAS device
ERROR_INTERNAL_DB_ERROR          	1383L
ERROR_FILE_READ_ONLY             	6009L	// The requested file is read only

The Status->Log entry will look like below with various error code numbers.

1 4/6/2021 09:55:14.025 Clips MoveFile Error 1235: c:\BlueIris\New\Cam2\Cam2.26052021_180000.bvr

Fix:

Run through Windows Tuning article. Following sections are important.

Local admin account section
Antivirus / Firewall sections
Storage section

Storage Drive
Issue: BI takes a long time to start

If your BI takes a long time to start, i.e. 5+ seconds, you could have corrupt files. Run Microsoft's System File Checker on your hard drive to see if there are any damaged files. From command prompt, run "sfc /scannow".

Database

Gotcha 1: DB Compact Error

BI performs a daily database compact/repair usually at 2 am. See Global settings -> Clips and archiving tab.
The database compact only takes as much time as it takes to copy the DB folder, usually seconds. The start and stop of the DB operations is logged.
Most of the Compact/Repair is just to make sure the list of recordings listed in the Clips list matches the files actually existing on the file system. For example, if you deleted a recording on the file system, Compact/repair will pick this up and delete this record from the database.
Any alerts associated with that clip/recording would also be wiped out.

This DB Compact Error (00000070) means "DISK FULL". Please ensure the drive with the DB always has enough space for a complete copy of that folder.

: db compact error.PNG (91.86 KiB) Viewed 23778 times

: status log errors_database.png (59.25 KiB) Viewed 21463 times

Gotcha 2: AddtoBIDB failed <error number>

: AddtoBIDB failed.PNG (56.6 KiB) Viewed 23321 times

Issue: This error means an alert was not written to the database. The consequence is a missed alert on the Alerts list as well as missed Alert actions (e.g. Push notification, email, SMS etc).

The database error is most probably caused by improper antivirus exemptions.
Run through the Windows Tuning article just to make sure Windows is not in conflict with BI.
In addition, confirm the drive containing the database is setup correctly.

Gotcha 3: Database errors - Alerts

: database delete errors.png (35.63 KiB) Viewed 23148 times

Issue: The logs may list errors after the Delete / Move commands for the Alerts folder.
The actual issue is the CLIPS.DAT file is somehow corrupted. This could be something like an item in an unexpected folder/view or it could be something worse like corruption in the headers.

Problem: The error number represents inconsistencies between the file system and the db records. BI uses the db to keep track of file recordings that were created and deleted. The errors mean there is a record in the database of a recording file. However, when BI searched the file system for the file, the file does not exist. This inconsistency is often due to users manually deleting camera recordings on the file system.

Fix:

Please right-click in the clips list and select Database -> Compact/Repair
Deletes "holes" in the database, e.g. deleting records of deleted items.
If errors remain, right-click in the clips list and select Database -> Repair/Regenerate.
Repair/regenerate will try to synchronize the database with content on the file system.
If errors remain, choose Database -> Delete & Regenerate.
The function will delete (start from scratch) and recreate the database by rescanning your hard drive and rediscover all your recordings in the All clips folder.
The Alerts folder will also be rescanned to regenerate the Alerts list. However, for most users, the Alerts folder is empty which means the Alerts list will be deleted. If you choose to save Hi-res JPEG files, the Alerts list can be recreated with the saved images. However, keep in mind the alert list will JUST have a snapshot of the alert. To actually playback the alert, you need to go to the corresponding Clip and scan to the time of the incident.
Since the Database is completely recreated, data that is ONLY stored in the DB may be lost, such as memos, alert images and other flags that exist only in the database.
If still not resolved, you may be having hardware issues (RAM, Drive where DB resides).
Make sure the database folder resides on a local, fast (and error free) drive
Double check the folder (containing the database) is excluded from antivirus software. See Windows Tuning article for details.
Make sure there is enough free space on the drive containing the database to duplicate/copy the Database folder at all times.
Check health of storage drives. Status button -> Storage tab. There should be no red spaces.
Windows comes with sfc /scannow (system file checker). Hard drive vendors, e.g. WD Purple, also comes with software to check their hard drives.
If this issue happens repeatedly, zip up the database folder and send to us via a file sharing service, e.g. google drive.

Recording

Recording errors have to do with dumping video to files or saving JPEGs based on your record settings. The New folder is usually the issue/focus, however, BI provides the flexibility to choose any folder. The errors here are a mixed bag. Since the content being stored is live video, the issue could be with the camera settings resulting in record errors. Or the issue could be with Windows or Server hardware (RAM / disk drive) and not being able to write to the drive.

If you find errors, try to troubleshoot yourself first. Each error provides next steps / fixes. In addition, make sure you are following best practices as well. See details in Windows Tuning article.

Encoder missing error

Rarely seen today because 32-bit Windows is rarely used anymore. The issue is Windows 32 does not support H.265.
Thus, if you set your camera settings to H.265 encoding, this problem may occur.

Fix:

Upgrade Windows to 64 bit
Change camera encoding to H.264

Access denied creating file.

BI was not able to create a recording file because of Windows privileges.

Fix:

Are you running BI as a service? Running the service under the LocalSystem account can cause issues. Disable and re-enable the service on settings/startup to run with your own local Windows account (w/ admin privileges). See Windows Tuning article for details.
Are you using a NAS device for storage? Are you also running BI as a service? If so, the Windows username/password used to run the service has to be identical to the credentials on the NAS device otherwise writing to the NAS device will not work. See Windows Tuning article for details.
Double check all security software exemptions for BlueIris.exe and the recording folders. See Windows Tuning article for details.

Out of Memory

BI ran out of memory.

Largely seen when running 32-bit Windows. If BI is using more than 1 GB of RAM on a Windows 32 machine, this is a very good indicator that you need a 64-bit system.

Fix:

Upgrade to a 64-bit Windows machine.
If you are using the server running BI for other applications, you may need to reassign this machine as a dedicated BI machine with no other applications running.
The amount of cameras connected to BI may be causing the memory issue. Disable cameras and see if stability returns.
Recording Direct to Disk (D2D) may decrease RAM usage as well.

Clip: Disk full

Hard drive is full. Recordings from your cameras can no longer be written to storage.

: status log_clip disk full.PNG (22.86 KiB) Viewed 23291 times

Quick fixes:

One user stated right click in Clips list -> Database -> Repair/Regenerate got Storage working again.
Always good to double check your Antivirus exemptions.

If the quick fixes do not work, you will need to figure out what broke with your storage setup.

Check your Clips and archiving settings so that you are not over-allocating disk space. The Status log -> Clip Storage tab provides a quick visual if storage was allocated correctly. There should be no red regions. The Help button provides details.
Another common issue is your archiving setup moves content from the C: drive to a NAS drive (New -> Stored). For some reason, BI is no longer able to connect to the NAS drive. Storage on New fills up. Before the New folder fills up, the logs will have MoveFile storage errors.

I document this scenario in the Storage / Record Issues case study.
Otherwise, you will need to look for Recording / Storage errors in the logs. Each documented error listed in this article describes the issue and possible fixes.
Finally, understand how Clips and archiving works in BI. Click on the Help button within the Clips and archiving dialog to get dialog specific help. The Storage and Recordings section of the Checklist shares articles/webinars for better understanding.

Clip: Disk Not Ready

: status log_clip disk not ready.PNG (36.5 KiB) Viewed 23289 times

One symptom is BI stops recording at night and after accessing the console directly in the morning, BI starts recording again.
The underlying issue is BI cannot access the hard drive because Windows has possibly put the PC to sleep (common for laptops).

Another cause could be the BI Server has lost connection to the NAS storage device and thus stopped recording. This is most often due to the username/password used to run the BI service is not identical to the username/password on the NAS devices. Storage section has details.

Fix:

Check Windows Power and sleep settings.
If using a NAS, make sure NAS is setup correctly to work with BI. Storage section has details.

AVI 1GB limitation

Just a warning message. AVI files have a 1GB file size limit. So if a user chooses to record in AVI format (rarely used today) and the camera record settings choose the default parameters, i.e. Combine or cut video with size equal to 4 GB, then there is a conflict. The warning message appears to notify user of the conflict but no action needs to be taken.

Cannot append; possibly corrupt

Two possible issues. The file is open but for some reason, BI is not able to write to the file. Possible issue could be the the disk is full.

Fix:

Make sure you are on the most recent release.
Double check free space in your storage directories. The Status log -> Clip Storage tab provides a quick visual if storage was allocated correctly. There should be no red regions. Click on the Help button within the Clips and archiving dialog to get dialog specific help. The Storage and Recordings webinar also walks through these settings. YouTube link.
Check Status->log for errors.

The second issue is a trigger event has occurred, but the camera stream is sending very few key frames resulting in no files being created.
If you do not know what are key frames, review the Camera Streams Optimization article to set your camera settings to work well with BI.
This issue occurs when the software is attempting to append to the recording file, yet there are no key frames in the stream. If you are recording direct to disk, make sure you have adequate key frames to start recording. Key frame value above 0.50/sec is recommended. The Status -> Cameras tab will provide camera stats. 5.3.9+ should resolve this issue in software.

Fix:

Upgrade to 5.3.9+
Review the Camera Streams Optimization article to set your camera settings to work well with BI.

Format change

BVR files have different formats based on certain BI settings. If those settings are changed, the BVR recording for that camera will change resulting in a new recording file being created. This is just an informational message. If you are usually expecting 3 recording files per day (default recording settings) you may unexpectedly see 3+ on occasion. A BVR file format change is the culprit due to BI camera setting changes.

Clip: write error

2 2/6/2021 4:40:01.700 PM Basement Clip: write error 80000000, undefined
2 5/8/2021 1:36:00.782 AM Kitchen Clip: write error 80000057, undefine2 5/14/2021 4:51:09.384 PM App

Generic error when BI attempts writing a recording to file. Error code 80000057 provides a little more clarity. The 57 error means "invalid format".

Fix:

Confirm the "New" directory is using a local SSD drive (ideally).
Reboot machine
Check physical hard drive. File write errors could very well indicate hardware issues.
Double check all security software exemptions for BlueIris.exe and the recording folders. See Windows Tuning article for details.

WriteBlueAlert no bitmap 0

Code: Select all

2 	28/10/2021 8:27:58.233 AM	Cam2                	WriteBlueAlert no bitmap 0

This error means there is no key frame available in the pre-trigger buffer in order to decode the camera feed. The key frame ratio has to be >0.5 for BI to work well. The usual culprits are:

You may have Limit decoding unless required selected. Camera settings -> Video tab.
You selected a proprietary encoding for the camera. H.264+ / H.265+ / Smart Codec

Fix:
Change the camera encoding. See Camera settings for details.
Turn off Limit decoding unless required. See Video tab for best practices.
Increase pre-trigger buffer size. See key frame / pre-trigger section for details on setting proper pre-trigger buffer size based on the key frame ratio.

The WriteBlueAlert error is problematic because they occur after a camera triggers and before the event is written to the database. The user will perceive missed alerts on the alerts list because of this error.

GPU

Error: CGadSocket thread

: cgadsocket.PNG (143.68 KiB) Viewed 23869 times

Issue: We suspect this issue is due to an incompatibility between the cameras encoding and your hardware acceleration. If hardware acceleration is not the culprit, the issue has to lie with software decoding.

Fix:

Turning off hardware acceleration for problematic cameras is the quick fix. If your CPU utilization is ok, then no worries.
Review camera settings and make adjustments to your camera encoding settings. The article provides tips on simplifying the camera encoding which may make the camera compatible with your hardware acceleration.
Update your drivers for the graphics card.

Warning: 4/17/2021 9:57:35.064 AM HW encode Setup failed: <error code> for 'recording 0'

Error codes: -1, -22

This error is saying you want to stream the saved camera video using your HWVA accelerator but it's causing problems so BI does so with software. Keep in mind streaming to file (i.e. exporting to an mp4) or streaming to a website is the same functionality for the software. This is a warning message because if the streaming fails in hardware, BI will continue by using software encoding. BI will continue future attempts with HWVA and will continue to switch to software encoding with each failure.
It is possible that the hardware can only support one stream at a time and is busy when BI makes the streaming request.
It's also possible the hardware can only decode streams. In that case, it's best to turn HWVA off when trying to stream.
Streaming occurs in many places. One common place is with the web server. Global settings -> Web server -> Advanced -> Configure. Change Hardware acceleration to No.
Streaming is also used when exporting to mp4. For example, right click on an alert in the clip list -> Convert/Export -> Configure.

Fix: Turn off HWVA as seen below.

: encoding_no_hw.png (73.99 KiB) Viewed 23528 times

"live sem stuck" error

Issue: Blue Iris is getting errors when accessing your hardware acceleration (gpu) to decode a camera stream.
Fix:

Turn off HA for that particular camera and confirm issue goes away. You may leave HA off if CPU utilization is ok.
Otherwise, you may need to update the firmware on the graphics card or the camera until there is compatibility.
Otherwise, simplify the camera encoding and observe whether issue goes away, e.g. H265 -> H264. Switch encoding profile from Main -> Baseline.

HW VA not compatible: <error number>

: hwva.png (28.4 KiB) Viewed 23250 times

Issue: "HW VA not compatible" are just warnings or messages not errors. HW VA stands for Hardware Video Acceleration. Basically, the software now turns off HA automatically for a camera if it determines the stream is not compatible and this is logged to Status->log as well.

Fix: If you want to fix the problem, you can go to the camera settings on the camera and adjust the encoding to a more simple format that may start working with your hardware. A simple example is if the camera was set to H.264 High encoding, adjust the setting to H.264 Baseline. If you were using H.265, switch to H.264. See Camera setup article for details.

Alternatively, driver updates may occasionally resolve the issue. And at times, the only solution is to leave the setting at No and let BI decode the camera stream via software. As long as the CPU utilization is reasonable, the No setting should be fine.

Memory

8000007a Error

: 8000007a error.PNG (13.14 KiB) Viewed 23665 times

The 7A error during clip/alert playback means "insufficient buffer", a type of "out of memory" error.

Fix:

Dial back number of cameras. If successful, your server needs more RAM to operate your cameras.
Switching to D2D can reduce RAM requirements.

If the error is reproducible on specific clips, please send a BVR example for review.

Clip: H264 encoder missing (X x Y)

X x Y are image resolutions: 856 x 480, 704 x 480, 512 x 512, 640 x 480 etc

Code: Select all

2       12/17/2021 6:14:28.436 AM      Side_Cam			Clip: H264 encoder missing (856x480)
2	 1/5/2022    1:00:20 PM		Garbage             	Clip: H264 encoder missing (640x480)
2	 1/5/2022    1:00:37 PM		Aisle16             	Clip: MJPG encoder missing (704x480)
2	 1/5/2022    1:00:39 PM		FrontFacingShop     	Clip: H264 encoder missing (856x480)
2	 1/5/2022    1:00:43 PM		Aisle38-40-360      	Clip: H264 encoder missing (512x512)
2	 1/5/2022    1:00:47 PM		Aisle16             	Clip: MJPG encoder missing (704x480)

Issue: This error is usually seen when the PC or Windows is running out of memory.

Fix:
Antivirus exemptions
Most common fix is to setup correct Antivirus exclusions so BI can run uninhibited.

DeepStack
This issue seems to pop up more when running DeepStack. Details from a ticket below.

User wanted to use 2 custom models. Running one model was fine: objects:0,dark. Running two models resulted in the issue: objects:0,dark,combined.

User test:
I set my AI preferences to use objects:0,dark,combined. I then restarted BI and walked outside to cause 12 triggers in 2 minutes. It looks like BI bumped the RAM but never released it. So I went back outside and did another walk around. The Physical and Commit Charge both increased slightly and never released. So it appears that you are correct. After the course of several alerts, I would run out of memory.

Process Explorer

: status log error_H264 encoder error.png (60.25 KiB) Viewed 20417 times

Event Viewer

Event Viewer will also confirm resource depletion when BI crashes.

Code: Select all

Log Name:      System
Source:        Microsoft-Windows-Resource-Exhaustion-Detector
Date:          12/28/2021 8:12:59 AM
Event ID:      2004
Task Category: Resource Exhaustion Diagnosis Events
Level:         Warning
Keywords:      Events related to exhaustion of system commit limit (virtual memory).
User:          SYSTEM
Computer:      DESKTOP-4SS4SEQ
Description:
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: python.exe (9392) consumed 8961937408 bytes, python.exe (9420) consumed 4247379968 bytes, and python.exe (9384) consumed 4187844608 bytes.