Record troubleshooting

2022-01-09T02:34:18+00:00

Introduction

This article is about narrowing down record issues to root cause. Common ticket states I am missing recordings or recordings have skipped. This is not actionable.

If you are new to BI, best to review Record tab article / webinar.

The Next steps / Submitting a ticket section at the bottom of the article states needed information to move forward.

Troubleshooting

Record settings

Understand your record settings. The When triggered and When alerted selections will ONLY record when the camera triggers/alerts.

When triggered
Camera triggers (trigger tab settings) -> Record
Trigger: Camera triggered (motion event in logs). Easy to confirm whether camera triggered (besides checking the logs). In the Trigger tab, set Add to alerts list: = Hi-res JPEG files. If the camera triggers, a corresponding alert image is created in your Alerts folder! Easy way to use BI to tell you what is going on.

--> Record: Recording occurs until the trigger ends

When alerted
Camera triggers (trigger tab settings) -> AI (if enabled) -> Alert filters (if any. set in Alerts tab) -> Record
Trigger: Camera triggered (motion event in logs). Easy to confirm whether camera triggered (besides checking the logs). In the Trigger tab, set Add to alerts list: = Hi-res JPEG files. If the camera triggers, a corresponding alert image is created in your Alerts folder! Easy way to use BI to tell you what is going on.

-> AI: If using AI, DeepStack analysis will tell you if an object was found, i.e. confirmed alert. Fortunately a DeepStack event will also be registered in the logs.

-> Alerts tab: Filters in the AIerts tab can block an alert. Easy to check. Do you see the alert in the Alerts list. If not, then you know the Alerts tab is somehow blocking the Alert!

-> Record: Recording occurs until the trigger ends

BI does NOT record continuously with When triggered/alerted settings. In addition, Combine or cut video each (selected by default) places triggers/alerts into a combined recording file. Thus, playing back a recording (i.e. a clip) will result in the appearance of gaps when in actuality, BI is just jumping to the next alert.

So, the time lapse when playing back a file that was created with When triggered/alerted settings is due to BI playing alert 1 and then alert 2 and so on. The time between alerts are NOT recorded, thus the possible appearance of a gap in the recording.

See Recording article / webinar for details.

Cameras

Camera streams

Garbage in = Garbage recorded.
Are the cameras functioning?
Is a particular camera not recording? May have a camera hardware issue.
Is a subset of camera(s) not recording? May have a network hardware issue.
Do you have good healthy live video? Not choppy? Not lagging? Not glitching etc.

If the camera streams are poor, choppy etc then of course the recordings will have issues. The camera streams section has links to articles to resolve streaming issues if any.

Camera stats

Know your camera stats and how to fix. The key frames and pre-trigger buffer settings are so crucial for BI to perform well when saving D2D. See details in key frame section.

Below is an example from a past ticket.
Issue: I had a dark room and a door opened, someone moved a box and left and the recording never showed any of it. Not even a single idea the door had ever opened for 25 seconds.

Your camera stats tell you how the camera is performing.

The camera stats are terrible. The key frame ratio is 0.11, i.e. one key frame every 9.09 s. The person could have entered the dark room and cleaned out everything in 10s.

Fix: User also stated he was experimenting with H.265+/H.264+ encoding which is a major no, no. H265+ / H264+ are all proprietary encoding which seem to optimize network bandwidth (for 4k+ cameras) by minimizing key frames and thus the issue.

Network congestion

Surveillance equipment puts a heavy load on the network. Need to know your network stats. If the network is congested, BI can lose connection to a camera or frames may be lost on the network.

Storage

The biggest cause of record issues is when recording is set to D2D, but the key frame ratio is low or no pre-trigger buffer is created, as stated above in bold.

The second cause is BI does NOT have disk space to write the recording. Confirm your storage is functioning.

BI Server

Windows
Make sure BI and Windows are working well together.
See Windows Tuning article for details.
Focus on the Storage sections.
Always good to review Firewall and Anti-virus sections as well.

Windows power settings
Power savings: Make sure BI is running full throttle all the time.
If the disk drive or the network card is put to sleep, then obviously you will have poor recordings.
On a BI server, power saver settings should only put the monitor to sleep and nothing more.

BI Software - Check logs
The logs are a great place to start because it identifies errors associated with hardware BI is dependent on to function: Hard drives, NAS devices, DeepStack etc.
Look for Storage and Recording errors.

Server running
If you have a gap in your recordings, first make sure the server was running.
Search for Restarted after unexpected shutdown. Error code 2 (first column).
The timestamp of the preceding line tells you when the server went downs: 10:37:41.516.

Code:

0 9/16/2021 10:37:41.516 PMAI                  Delete: nothing to do [62/168 hrs, 2.00M/5.00G, 256.0G free]2 9/16/2021 10:38:03.393 PMApp                 Restarted after unexpected shutdown (5.5.0.11/1500/240)

Other software checks

Threads that can temporarily stop recordings.

Archive thread
The archive thread is responsible for moving files based on your Clips and archiving settings and has the ability to stop recordings temporarily. If you have Log clip move/delete details selected in Global settings -> Clips and archiving tab, then below log entries every 5 minutes indicates the thread is running properly.

Code:

0 1/1/2022 12:07:22.765 AMNew                 Delete: nothing to do [719/720 hrs, 11.0G/100.0G, 297.1G free] 1 locked0 1/1/2022 12:07:22.766 AMAlerts              Delete: nothing to do [167/168 hrs, 50.2M/20.0G, 297.1G free]

The actual details will be different based on your Clips and archiving setup. The 0 code event starting with Delete:, however, are the identifiers associated with archive thread events.

Compact / Repair thread
Compact / repair is a UI feature initiated by right clicking on the clip list -> Database -> Repair/regenerate. If there were issues with its functionality, it would be immediately visible in the UI feedback.

CPU Utilization
If the CPU is overloaded (>80%), then BI may be too busy to fulfill all the camera settings.
See link for details.

Hardware overloaded

You may have so many cameras connected to your server that you overloaded the hardware. Disable working cameras and see if the problematic cameras start working again. This is a telltale sign that you have overloaded hardware resources.

Hardware acceleration
To determine whether the hardware acceleration (HA) is overloaded, enable all the cameras and turn HA off for the non-working cameras. By doing so, you have moved the decoding load from HA to the CPU. See if the non-working cameras start working again.

Ethernet card

The Network congestion section above examines whether the server ethernet card is overloaded.

Hard drives

Customer Anecdotal:
Symptoms: The cameras keep freezing and stop recording after so long. Keeps getting network/signal retry but the network is fine and if I plug the old machine in it works fine.

The server suddenly ‘freezes’ along with most of the cameras. There’s doesn’t appear to be one ‘particular model’ of camera that is an issue. This can happen after half an hour or after 10 hours, it is intermittent. It can affect 2-3 cameras, or all of them. Once it freezes the iphone app will connect but says all the cameras are unreachable, the UI3 interface won’t connect/refresh at all, the blueiris cameras on screen are frozen and despite the Server saying it is at 4% RAM, the app on the iphone says it’s at 96%?!

Once Blue Iris was started it kept creeping up to 100% usage and then maxing the RAM out.

Issue/Fix: Faulty hard drive which was replaced.

Google search for many articles on "windows faulty hard drives" to determine whether you have a bad drive. Here is a link to one finding.
System File Checker (SFC) and Check Disk (chkdsk) are popular tools provided by Windows.

BI Software Settings

Below are accidental software settings that could have stopped recordings.

The following are ways recordings could have stopped based on BI settings.

Record settings

Camera settings -> Record tab

If the video checkbox is unselected, you turned off recordings for the camera.
If you choose one of the "When" settings, the cameras will NOT record unless the camera triggered.
Make sure your trigger settings are correct.

Camera Paused
Was the camera temporarily paused in BI?

Shield red
Was the shield red?

Profile
Was the camera not active for certain profiles?

The image above shows the camera is set to inactive for Profile 2. Camera settings -> General tab.
The current active profile is 2 (Work hours).
In the camera border, BI tells you the camera is running Profile 0 (Inactive) instead of Profile 2, which is a great visual reminder of what is going on.

Camera schedule

Similar to above, the camera schedule can also force a camera to the Inactive profile. Camera settings -> Schedule.

If you uncheck, Use global schedule when..., all the blank space in the schedule is filled with the Inactive profile.

Motion sensor
Is there a blue border around the camera? The blue border means the motion sensors are off for the camera.
If you chose one of the When+ record settings, then nothing will record because the camera will not trigger because the motion sensors are off.
The Alerts list / Timeline will always be empty because the camera is not triggering.

Either your Schedule is running the Inactive profile as seen in the image above. This would result in ALL the cameras not recording.
Or you paused the camera as stated above
Or somehow the motion sensor is turned off. Camera settings -> Trigger tab.

Gotchas
Learnings from past tickets.

Gotcha 1: Recordings stopped

You first need to understand the scope of the issue. Did recordings stop across ALL your cameras or do you have issues with one camera? Easy way to know is by opening the filter by date feature in the Clips list.

The red dots indicate days with recordings. Notice in example below, no red dots for Jan 4 - Jan 21. This implies the issue exists for all cameras.

The question to ask yourself is whether Recordings do not exist because BI is not recording or whether the Database is running into errors and not inserting clips / alerts into the database tables.

Check for database issues.
If not the database, you probably have a storage issue.

Submitting a ticket

Log files

Fix any server errors. The Log files framework explains how to use excel to dissect log files quickly.
The Understanding Log Files article/webinar is a case study on how I discovered NAS issues for a user.

If you really believe the issue is with BI software, provide the supporting evidence.

Describe the issue.
When you discover a gap, send the bvr to us for review.
We need a screenshot of the Alert in the Alerts list. This is how we confirm the camera was triggered/alerted and was supposed to continue recording. Does not apply to Continuous or Periodic record settings.
Specify the time when the gap occurred.
Share your camera settings. Camera settings -> General tab -> Export
Share the logs coinciding with bvr file. ** Very Important **
Support info

What we will do with the information.

Confirm recordings stopped in the BVR file.
Confirm your camera settings are not causing the issue
Check the logs to see what could have caused recordings to stop.
Were there any errors associated with writing to disk?

Before sending to us, you should first do the above to confirm for yourself the issue is NOT with your settings or your server hardware.

Statistics: Posted by varghesesa — Sun Jan 09, 2022 2:34 am