Blue Iris

Frameworks • Creating short BVR files

2022-02-24T17:28:17+00:00

Introduction

We sometimes require the original bvr file in order to investigate certain issues such as Record, Export or Playback issues. Many users use the popular Combine/cut feature in Record settings to reduce the number of files/recordings that are created on your file system. However, these BVRs often turn out to be quite large, usually 1 GB+ and has a lot of content that is irrelevant to the issue.

This article explains how to create a short BVR file that isolates the issue.

Small BVR files

Couple of options to make smaller bvr files. In Camera settings -> Record tab.

If you uncheck Combine or Cut Video, that will result in BI cutting the BVR file after each motion trigger.
Or you can leave Combine or Cut Video, and change the time length to something small like 15 min.

Statistics: Posted by varghesesa — Thu Feb 24, 2022 5:28 pm

Frameworks • Camera audio

2022-01-28T01:40:06+00:00

Introduction

If you were directed to this page you are having audio issues.

Audio or 2-way audio
I cannot hear audio from the camera.
I cannot speak to the camera.

Below are the settings to double check.

If you need support to get involved, the Next steps / Submit a ticket section explains exactly what is needed to move forward.

Troubleshooting

Step 1: Camera connector

To get audio to work, especially 2-way, you have to choose the Make / Model appropriate for your camera in the IP Config dialog. Generic/RTSP will likely not work. See Camera Connector article (Step 3) for best practices when connecting cameras with advanced functionality.
If your camera happens to use port 34567 for the RTSP/Media port, then Make/Model = Cantonk/H.264... may work.

Step 2: Confirm BI Server settings.

Confirm audio is enabled. Camera settings -> Audio tab.

If you cannot hear audio from a remote endpoint (UI3, mobile app), confirm webcasting is also selected. Below is a screenshot of default settings.

Step 3: Console test

If Audio does not work from the console, it will not work from any remote endpoints.
Live view
While testing from the desktop, make sure audio is first turned on for BI and the computer.

Computer audio is on and loud enough to hear.
BI audio is on and loud enough to hear. Top right of screen.
Select the camera. Confirm audio is active for the camera. Top left of camera mosaic window.

If console test fails, check Setup RTSP back-channel for talk support (PCM-U format) in the IP Config dialog.
Unlikely it will work, but a simple test so why not try.

Playback test
After you get live view working, confirm playback also works.
Trigger the camera to force a recording.
Can you hear the audio on playback?
If not, uncheck/check Also BVR. Camera settings -> Video tab.
Hardware acceleration can have adverse affects on audio.

MP4 test
A simple way to confirm the connector is pulling in the camera audio is to record a sample from the camera. Export the clip to mp4. If something is wrong with audio on the console machine (live view / playback tests above), this test at least confirms the BI connector is able to pull the audio from the camera.

Export settings can be tricky. Selecting mp4 format and Include audio track is a must for this test. Use sub stream if available is optional. If you are recording D2D and have Re-encode video unselected, it's very possible your media player cannot play the video format provided by your camera. Selecting Re-encode video almost always works.

Basically, the settings above are the safest and almost always works. The point of the test is to just confirm audio is being sent from the camera and being ingested in the BI server.

Step 4: User permissions

Before testing remote endpoints, confirm user logged into mobile app/UI3 has audio privileges.
Global settings -> Users tab.

Step 5: Browser test

From a mobile browser, go to the web interface (UI3) and confirm audio works for both the LAN and WAN.
If not go to the Encoder settings. Confirm HA is off. Hardware encoders are notoriously bad.

Step 6: Test from the mobile app

Before testing on the mobile app, update to the latest version of the app and the BI server.
Features/functionality on the mobile app are not guaranteed to run with older versions of the server.

Confirm audio is turned on for the mobile app.

Step 7: Two-way audio

If your camera has a speaker and you want to talk to the camera (e.g. doorbell camera and someone's at the door):

Mic and speaker can't be live at the same time and mic wins ties.
Users turn on the speaker and hold down the mic when they want to speak and it will turn back off when you let go.

Next steps / Submitting a ticket

Camera settings. Camera settings -> General tab -> Export.
If the issue is with a remote endpoint,

Confirmation or screenshot showing user permissions are set correctly. (Step 4)
UI3 audio. Works / does not work. (Step 5)
Mobile app audio. Works / does not work. (Step 6)

You may need to put the camera on the WAN for remote testing. We will let you know.

Statistics: Posted by varghesesa — Fri Jan 28, 2022 1:40 am

Frameworks • Database troubleshooting

2022-01-26T23:50:10+00:00

Introduction

Having a functioning database is critical to BI performing correctly. If there are database read/write errors, then bad things happen like recordings/alerts go missing.

Many users do not realize this, but the Clips list, Alerts list and Timeline are visual representations of records stored in the database.

Troubleshooting

Basics

Make sure you followed best practices. FYI, reviewing entire Windows tuning article is always beneficial.

Clips missing

Does BI indicate it is no longer recording? In the filter by date image below, it appears as though BI stopped recording from Jan 4 - Jan 21 (no red dots).

Not sure yet if BI actually stopped recording or the Database is no longer being updated. An easy check is to run Repair/Regenerate (see below) and see if BI can resolve the issue. Another easy check is to open your New folder and see if new recordings are being created by BI. If so, the issue is with writing to the Database. Look for database errors in the logs.

If new recordings are not being created, you probably have a storage issue or recording issue.

Database tools

If you right click in the Clips list -> Database section, you have access to the database maintenance functions. Help provides details on each function. Repair/regenerate is often used when users notice clips/alerts are missing from the Clips list/Alerts list.

Log files

The logs will tell you if BI is having issues with your Database. In order to know which errors to look for, we are documenting all the Status log errors. The ones concerning the database are in the Database category. Each log error provides details and possible fixes to self-correct.

The Log files article explains how to easily deal with log files using excel.

Camera stats

If particular cameras are missing alerts, the usual issue is lack of key frames. See key frame ratio section for details.

Submitting a ticket

Issue

Clearly describe the issue. If the issue resides with a remote endpoint, confirm whether there is also an issue in the console. If you tried the Database maintenance commands, please share. Anything else you tried.

Storage

Screenshot of Alerts folder settings.
Screenshot of the Status button -> Storage tab

Camera stats

Status button -> Cameras tab -> Export

Logs

What do the logs say? Copy/paste the exact error events from the log file or screenshot the Logs tab.

Statistics: Posted by varghesesa — Wed Jan 26, 2022 11:50 pm

Frameworks • CPU Utilization

2022-01-26T21:27:24+00:00

Introduction

Issue: CPU utilization > 60-75% is not good.

Troubleshooting

The CPU management section in the Help file provides a thorough explanation of CPU utilization. Below are the most common issues/fixes that alleviate CPU load.

Antivirus and firewall exemptions
BI will not perform well if Windows is in conflict with BI. Make sure your Anti-virus and Firewall settings are set correctly.

Cameras
CPU utilization is highly dependent on the number of cameras and how the cameras are connected to BI. Walking through the Connect cameras correctly section in the Checklist documents best practices for adding cameras. The below bullets highlight some of the key settings to reduce CPU load. Reviewing the documents listed in the Connect cameras correctly section in its entirety is recommended.

For all the high res (2MP+) cameras you need to have dual streams.
Record direct to disk not re-encode.
Turn on hardware acceleration.
Your GPU may not support H.265 or complex H.264 profiles. Always good to simplify the camera encoding as much as possible to work with your hardware.

Streaming
Streaming to many endpoints (UI3, mobile apps), web sites, YouTube etc adds load to the server and the network. A simple test is to reduce the number endpoints (turn UI3, phones, tablets) temporarily and see if CPU decreases. There is not much that can be done besides getting better hardware. The Networking article -> Network tests section does provide some insight on diagnosing server performance.

One possible CPU issue due to dynamic group layout is documented in the 5.5 Release Notes -> Gotcha 8.

Statistics: Posted by varghesesa — Wed Jan 26, 2022 9:27 pm

Frameworks • RAM Utilization

2022-01-24T16:43:53+00:00

Introduction

We occasionally receive tickets that RAM keeps growing while BI is running. Occasionally there is a RAM leak introduced by BI and if so we will notify you. If not, more often than not, hardware issues or drivers cause the leak.

Troubleshooting

Version control
If BI created a memory leak, we usually know quickly and notify users to update the software. If you believe a recent update has caused the issue, go back to a previous version and see if leak goes away. This is very helpful information.

Anti-virus software
Otherwise, the usual culprit is Anti-virus software.
All the content in this article is good to review. Pay attention to the Anti-virus and Firewall sections.

Hardware acceleration
Other sources for leaks are drivers. Turn off hardware acceleration and see if RAM stops growing. You may need to update your drivers. Alternatively, go to each of your cameras and simplify the encoding and observe if the leaks go away.

Other hardware
Below are examples of RAM and disk drive failures that lead to RAM issues.

Customer anecdotals:

Antivirus update and software reinstall.

The Memory issue seems to have gone. I did some cleanup in BI, rebooted some cameras, and checked Firewall / AV. I’m not sure what fixed it. But… till next time. Thanks.

Reinstalled software

OK, so I deleted everything BI off the computer, I also reformatted the drive I save footage to. I then reinstalled BI and it seems to be working... below are the data from about an hour apart. So is the RAM issue (and it appears the CPU usage dropped as well) related to the storage drive filling up? or was it the fresh install of BI that fixed it?

Update for you. The storage has been maxed out for 5 weeks now. Small increase of CPU usage which dropped the last 2 weeks. RAM usage has stayed below 3gs.
I believe that would eliminate everything other than the corrupted update possibility which re installing would have solved.

Thanks for your help

Faulty hard drive

I have recently upgraded my PC (specification attached) and haven’t really been able to get Blue Iris to work properly since ☹
In the most recent logs I am getting out of memory errors (Exception code: 0xc0000005). However this is a brand new top spec gaming PC with 16GB RAM. The CPU Never goes over 15% and the RAM 5G as far as I can see.

Blue Iris also regularly crashing out, I have attached the Application Error that seems to cause it (Exception code: 0xc0000005). So far since purchasing the PC I haven’t been able to get Blue Iris to work properly on it ☹

Thanks for the attempt at helping, but upon further investigation by me this has actually turned out to be a faulty hard drive. Once BlueIris was started it kept creeping up to 100% usage and then maxing the RAM out.

New PC company is sending replacement disk.

I have been trying to troubleshoot a problem that I've had for a month or two. I started getting Signal: network retry, Signal: restored (these were only a few seconds long between the retry and restore), and DeepStack errors like timeouts and restarts in the logs. Some alerts wouldn't play back. Then the memory would creep up and the server would become so unresponsive I would need to do a hard reset.

I finally discovered in the system event logs some bad block errors on the disk. Never had any write errors in the BI logs. I finally swapped the local drive where I keep the BI data and everything has been back in business again. Is there a reason BI couldn't detect write errors? It took a month before I figured it out and went through so many settings changes, reinstalls, and tweaks. What proved it was when I tried a full format and it never finished. I even ran a chkdsk and it didn't show any bad sectors. The health was good in the disk utility. I tried reseating the cables, etc. Then finally the system couldn't even boot unless I unplugged the drive.

Bad RAM

Update on this ticket.

I did some more digging and have determined that the “chirping” issue is due to hardware memory errors. (I was able to verify this with PassMark’s MemTest86) I’m hopeful the playback stuttering and other issues I’ve been having with Blue Iris are also related to bad RAM.

The chirp was coming from the buzzer on the motherboard.

I don’t think it had anything to do with BI other than when BI was running the system was having RAM issues (which were triggering a BIOS level fault that alerted by chirping the buzzer) . No errors in any windows logs. Process monitor did show lots of buffer overflows, which led me to test the ram.

Logs
Check the logs for errors.

Storage
Storage and RAM issues often go hand in hand. Check storage.

Isolate the issue
If not the above, you need to isolate the feature / functionality in order to discover the source of the leak. See if it only happens when camera X is enabled or a particular feature is used.

Submitting a ticket

The bottom line is if we are aware of a memory issue, we are the first to let you know. The above documents the usual culprits, many of which are due to hardware failures. Is it possible that your unique environment has introduced a memory issue? Possible. This is why the final steps are to check the logs for errors and isolate the issue. If you have additional information regarding log errors we can help. If you can isolate the issue, we are thankful and happy to investigate further.

Replying to a ticket after being forwarded to this article

If your response after reading above is "Yes, I checked all that and the issue still remains." or something to that extent, then you may want to keep running the last stable version or identify a local reseller that can provide support. The good tickets sub-forum shows the kind of user expertise that leads to resolution.

As you can see from the above customer anecdotals, there are some pretty tech saavy users that know Windows and how to diagnose hardware issues. I also listed the Tools they used to identify root cause below. If you do not have the hardware expertise, perhaps visit a local PC shop that can evaluate your hardware.

Tools

Tech saavy users have used the following tools to identify RAM issues.

passmark software: RAM tests
Process explorer: May provide hints based on errors.

Statistics: Posted by varghesesa — Mon Jan 24, 2022 4:43 pm

Frameworks • Creating video

2022-01-23T22:32:30+00:00

Introduction

Short helper article to explain how to create videos to better explain issues in a ticket.

Video

I strongly recommend creating a short desktop video with a narrative of the issue. I use Zoom to record the desktop. There are many choices provided by Windows. VLC is also an option.

FYI. You can point the phone camera at the desktop and create a video pertaining to a console or UI3 issue as well.

Some users provide a video without a narrative of what's going on which is more challenging because I have to keep going back and forth between the email and the video to understand. Alternatively, provide notes as to what you are demonstrating in the video. For example:

At 1:05, I doubled the playback speed to 2x.
At 1:49, the scrubber will not stick to the position where I wanted playback to begin.
etc.

Mobile device
If the issue lies with the Mobile device, phones have made it very easy to create a video with screen share.

In iOS, creating a video is simple now. https://support.apple.com/en-us/HT207935
Just as simple on an Android. Invoke Screen record.

If the video (mp4) is too large to send via email, use a file share service. Many use Google drive but there are lots of choices. DropBox, Box, iCloud etc.

Statistics: Posted by varghesesa — Sun Jan 23, 2022 10:32 pm

Frameworks • Log files

2022-01-18T02:14:37+00:00

Introduction

The log files are invaluable when trying to root out instability and crashes. With thousands of lines of events, how do you make sense of the logs? This article explains how to do so.

Activate log files

If you have not done so already, turn on logging. Check Save to file. Status button -> Log tab.

Understanding the log file

The log file is really good for providing clues when external functionality that BI depends on is NOT functioning.
The Status Log Errors article provides details on error events in the log file.
Each error also documents fixes. If you identify errors, go to the Status Log Errors article, find your error and try to self correct.

The major categories are:

DeepStack: 3rd party solution
Storage: Hardware. Hard drive
Database: Proxy for hardware issue. Hard drive
Recording: Hardware. Hard drive
GPU: Hardware. GPU
Memory: Hardware. Memory
ONVIF Trigger: Hardware. Camera
Registry: Windows
3rd Party software

The event code is the first column (digit) in every line.

0 = Informational event
1 = Warning
2 = Errors
3 = Motion events
4 = Check marks
10 = user access (login / logout)

I always look for Errors and try to resolve.
Address Warnings when time is available.

Excel
The log files are easy to import into excel. After doing so, it becomes very easy to process logs that are 1000s of lines long. One gotcha during the import is to set the date column to Text otherwise the column gets incorrectly translated to a number.

1. Once imported, the error code is 2. Filter the data by error code 2 to easily find all the errors events that occurred on your server.

2. For each error, go to the Status log errors file to resolve the issue. If you know excel pretty well, it's easy to determine the different types of errors seen by your server.

Crashes
Filter for the Restart after unexpected shutdown events. If the event exists, the server crashed. The time stamp of the event BEFORE the Restart line identifies when the crash occurred.

If the server crashed, good to see if any of the prior events to the crash provides hints as to the cause. Continue searching up to determine if common events occur before crashes. The restart lines always state which version of the software was restarted which is usually the version that crashed. Always good to know.

One user identified a camera was constantly rebooting at night prior to a BI crash. Disabling the camera restored stability.

A good best practice is to examine the log file on a weekly/monthly basis for errors and warnings.

Submitting a Ticket

The point of this guide is NOT to identify errors and email support. The point of this guide is to identify errors -> read through the details of your errors and apply the suggested fixes to self-correct.

If still an issue and your error is documented, provide details on what exactly you did to resolve the issue. Suggested fixes vary depending on the underlying issue.

Statistics: Posted by varghesesa — Tue Jan 18, 2022 2:14 am

Frameworks • Mobile apps troubleshooting - Connectivity

2022-01-15T18:15:28+00:00

Introduction

This article provides the framework for troubleshooting connectivity issues with your mobile app.

Troubleshooting

Sample Issues:

User states cannot connect to the WAN and/or the LAN.
User states the app crashes.
User states live view, playback etc is not working or intermittent.

Diagnostics

Needed steps to diagnose/understand the issue.

Server

Windows
Always good to run through the Windows Tuning article just to make sure Windows is not in conflict with BI. Pay attention to Anti-virus and Firewall sections.

Logs
Always check the logs for errors.

Connectivity issues
Connectivity issues are issues associated with app crashes or connectivity issues to the server.

Restart

Could be as simple as restarting the app.
Could be as simple as rebooting the phone.
Could be as simple as reinstalling the app.

Lookup (iOS) / Get IPS (Android) key

If the apps worked before, the fix could be as simple as updating IP addresses on the app. The Mobile devices article -> Lookup key section explains how to easily update those values on the app.

Mobile app server settings
Confirm your mobile app server settings.

License key
Http vs https protocol
LAN/WAN addresses
Username
Password

This section also has screenshots of the phone settings needed for the app to work correctly. Confirm your phone settings as well.

BI server settings

The Network connectivity section provides guidelines on how to trouble shoot mobile app connectivity issues.

Video streams
Issues with live view, playback of alerts.

Scope
Is the issue with a subset of cameras or all the cameras? If the issue is with one camera, you may need to troubleshoot that one camera. See troubleshooting articles in the Cameras section of the Troubleshooting toolkit.

Streaming Settings
If all the cameras are affected, maybe the stream is too complicated for the media player on your phone. Try simplifying the stream.

Submitting tickets

Support information

It's easy to share support info. It's important as well in order to understand your computer stats and the version of the software you are running.
Before submitting a new ticket, click the Help button (? button).
Copy Email support info to copy critical system info to the clipboard.
Then paste the clipboard into the top of your support email. Below is a snippet from my "Email support" button.

Email to: support@blueirissoftware.com

Support data:
Version: Release 5.3.8.12 x64 (1/22/2021)
Service: No
License: XXX
Name: John V
Email: @yahoo.com
Support: Basic

Steps to reproduce

What are the steps to reproduce the error? A video is always helpful.

Remote access wizard

Any errors? If so, did you go through help to self-correct. Screenshot of any errors.

Remote device tests

Test 1: Can you access the web interface from the local machine?
Test 2: From a browser on the phone, can you connect via UI3 to the server on the LAN?
Test 3: From a browser on the phone, can you connect via UI3 to the server on the WAN?

Screenshots

Global settings -> Web server tab
Global settings -> Web server -> Advanced button
Mobile app: Server settings

Gotchas

Learnings from past tickets.

Gotcha 1: Cameras are not visible

There are many reasons why a camera is not visible on the mobile app.

The camera could be marked as hidden. Camera settings -> General tab.
Webcasting could be disabled. Camera settings -> Webcast tab.
Permissions. The username logged into the app is a limited account that does not have permission to view all cameras.

Gotcha 2: Group/mosaic view is not visible

Similar to above.

Group webcasting is off. Gear icon (top right of console) -> Enable group webcast stream.
Permissions. The username logged into the app is a limited account that does not have permission to view all groups.

Statistics: Posted by varghesesa — Sat Jan 15, 2022 6:15 pm

Frameworks • Record troubleshooting

2022-01-09T02:34:18+00:00

Introduction

This article is about narrowing down record issues to root cause. Common ticket states I am missing recordings or recordings have skipped. This is not actionable.

If you are new to BI, best to review Record tab article / webinar.

The Next steps / Submitting a ticket section at the bottom of the article states needed information to move forward.

Troubleshooting

Record settings

Understand your record settings. The When triggered and When alerted selections will ONLY record when the camera triggers/alerts.

When triggered
Camera triggers (trigger tab settings) -> Record
Trigger: Camera triggered (motion event in logs). Easy to confirm whether camera triggered (besides checking the logs). In the Trigger tab, set Add to alerts list: = Hi-res JPEG files. If the camera triggers, a corresponding alert image is created in your Alerts folder! Easy way to use BI to tell you what is going on.

--> Record: Recording occurs until the trigger ends

When alerted
Camera triggers (trigger tab settings) -> AI (if enabled) -> Alert filters (if any. set in Alerts tab) -> Record
Trigger: Camera triggered (motion event in logs). Easy to confirm whether camera triggered (besides checking the logs). In the Trigger tab, set Add to alerts list: = Hi-res JPEG files. If the camera triggers, a corresponding alert image is created in your Alerts folder! Easy way to use BI to tell you what is going on.

-> AI: If using AI, DeepStack analysis will tell you if an object was found, i.e. confirmed alert. Fortunately a DeepStack event will also be registered in the logs.

-> Alerts tab: Filters in the AIerts tab can block an alert. Easy to check. Do you see the alert in the Alerts list. If not, then you know the Alerts tab is somehow blocking the Alert!

-> Record: Recording occurs until the trigger ends

BI does NOT record continuously with When triggered/alerted settings. In addition, Combine or cut video each (selected by default) places triggers/alerts into a combined recording file. Thus, playing back a recording (i.e. a clip) will result in the appearance of gaps when in actuality, BI is just jumping to the next alert.

So, the time lapse when playing back a file that was created with When triggered/alerted settings is due to BI playing alert 1 and then alert 2 and so on. The time between alerts are NOT recorded, thus the possible appearance of a gap in the recording.

See Recording article / webinar for details.

Cameras

Camera streams

Garbage in = Garbage recorded.
Are the cameras functioning?
Is a particular camera not recording? May have a camera hardware issue.
Is a subset of camera(s) not recording? May have a network hardware issue.
Do you have good healthy live video? Not choppy? Not lagging? Not glitching etc.

If the camera streams are poor, choppy etc then of course the recordings will have issues. The camera streams section has links to articles to resolve streaming issues if any.

Camera stats

Know your camera stats and how to fix. The key frames and pre-trigger buffer settings are so crucial for BI to perform well when saving D2D. See details in key frame section.

Below is an example from a past ticket.
Issue: I had a dark room and a door opened, someone moved a box and left and the recording never showed any of it. Not even a single idea the door had ever opened for 25 seconds.

Your camera stats tell you how the camera is performing.

The camera stats are terrible. The key frame ratio is 0.11, i.e. one key frame every 9.09 s. The person could have entered the dark room and cleaned out everything in 10s.

Fix: User also stated he was experimenting with H.265+/H.264+ encoding which is a major no, no. H265+ / H264+ are all proprietary encoding which seem to optimize network bandwidth (for 4k+ cameras) by minimizing key frames and thus the issue.

Network congestion

Surveillance equipment puts a heavy load on the network. Need to know your network stats. If the network is congested, BI can lose connection to a camera or frames may be lost on the network.

Storage

The biggest cause of record issues is when recording is set to D2D, but the key frame ratio is low or no pre-trigger buffer is created, as stated above in bold.

The second cause is BI does NOT have disk space to write the recording. Confirm your storage is functioning.

BI Server

Windows
Make sure BI and Windows are working well together.
See Windows Tuning article for details.
Focus on the Storage sections.
Always good to review Firewall and Anti-virus sections as well.

Windows power settings
Power savings: Make sure BI is running full throttle all the time.
If the disk drive or the network card is put to sleep, then obviously you will have poor recordings.
On a BI server, power saver settings should only put the monitor to sleep and nothing more.

BI Software - Check logs
The logs are a great place to start because it identifies errors associated with hardware BI is dependent on to function: Hard drives, NAS devices, DeepStack etc.
Look for Storage and Recording errors.

Server running
If you have a gap in your recordings, first make sure the server was running.
Search for Restarted after unexpected shutdown. Error code 2 (first column).
The timestamp of the preceding line tells you when the server went downs: 10:37:41.516.

Code:

0 9/16/2021 10:37:41.516 PMAI                  Delete: nothing to do [62/168 hrs, 2.00M/5.00G, 256.0G free]2 9/16/2021 10:38:03.393 PMApp                 Restarted after unexpected shutdown (5.5.0.11/1500/240)

Other software checks

Threads that can temporarily stop recordings.

Archive thread
The archive thread is responsible for moving files based on your Clips and archiving settings and has the ability to stop recordings temporarily. If you have Log clip move/delete details selected in Global settings -> Clips and archiving tab, then below log entries every 5 minutes indicates the thread is running properly.

Code:

0 1/1/2022 12:07:22.765 AMNew                 Delete: nothing to do [719/720 hrs, 11.0G/100.0G, 297.1G free] 1 locked0 1/1/2022 12:07:22.766 AMAlerts              Delete: nothing to do [167/168 hrs, 50.2M/20.0G, 297.1G free]

The actual details will be different based on your Clips and archiving setup. The 0 code event starting with Delete:, however, are the identifiers associated with archive thread events.

Compact / Repair thread
Compact / repair is a UI feature initiated by right clicking on the clip list -> Database -> Repair/regenerate. If there were issues with its functionality, it would be immediately visible in the UI feedback.

CPU Utilization
If the CPU is overloaded (>80%), then BI may be too busy to fulfill all the camera settings.
See link for details.

Hardware overloaded

You may have so many cameras connected to your server that you overloaded the hardware. Disable working cameras and see if the problematic cameras start working again. This is a telltale sign that you have overloaded hardware resources.

Hardware acceleration
To determine whether the hardware acceleration (HA) is overloaded, enable all the cameras and turn HA off for the non-working cameras. By doing so, you have moved the decoding load from HA to the CPU. See if the non-working cameras start working again.

Ethernet card

The Network congestion section above examines whether the server ethernet card is overloaded.

Hard drives

Customer Anecdotal:
Symptoms: The cameras keep freezing and stop recording after so long. Keeps getting network/signal retry but the network is fine and if I plug the old machine in it works fine.

The server suddenly ‘freezes’ along with most of the cameras. There’s doesn’t appear to be one ‘particular model’ of camera that is an issue. This can happen after half an hour or after 10 hours, it is intermittent. It can affect 2-3 cameras, or all of them. Once it freezes the iphone app will connect but says all the cameras are unreachable, the UI3 interface won’t connect/refresh at all, the blueiris cameras on screen are frozen and despite the Server saying it is at 4% RAM, the app on the iphone says it’s at 96%?!

Once Blue Iris was started it kept creeping up to 100% usage and then maxing the RAM out.

Issue/Fix: Faulty hard drive which was replaced.

Google search for many articles on "windows faulty hard drives" to determine whether you have a bad drive. Here is a link to one finding.
System File Checker (SFC) and Check Disk (chkdsk) are popular tools provided by Windows.

BI Software Settings

Below are accidental software settings that could have stopped recordings.

The following are ways recordings could have stopped based on BI settings.

Record settings

Camera settings -> Record tab

If the video checkbox is unselected, you turned off recordings for the camera.
If you choose one of the "When" settings, the cameras will NOT record unless the camera triggered.
Make sure your trigger settings are correct.

Camera Paused
Was the camera temporarily paused in BI?

Shield red
Was the shield red?

Profile
Was the camera not active for certain profiles?

The image above shows the camera is set to inactive for Profile 2. Camera settings -> General tab.
The current active profile is 2 (Work hours).
In the camera border, BI tells you the camera is running Profile 0 (Inactive) instead of Profile 2, which is a great visual reminder of what is going on.

Camera schedule

Similar to above, the camera schedule can also force a camera to the Inactive profile. Camera settings -> Schedule.

If you uncheck, Use global schedule when..., all the blank space in the schedule is filled with the Inactive profile.

Motion sensor
Is there a blue border around the camera? The blue border means the motion sensors are off for the camera.
If you chose one of the When+ record settings, then nothing will record because the camera will not trigger because the motion sensors are off.
The Alerts list / Timeline will always be empty because the camera is not triggering.

Either your Schedule is running the Inactive profile as seen in the image above. This would result in ALL the cameras not recording.
Or you paused the camera as stated above
Or somehow the motion sensor is turned off. Camera settings -> Trigger tab.

Gotchas
Learnings from past tickets.

Gotcha 1: Recordings stopped

You first need to understand the scope of the issue. Did recordings stop across ALL your cameras or do you have issues with one camera? Easy way to know is by opening the filter by date feature in the Clips list.

The red dots indicate days with recordings. Notice in example below, no red dots for Jan 4 - Jan 21. This implies the issue exists for all cameras.

The question to ask yourself is whether Recordings do not exist because BI is not recording or whether the Database is running into errors and not inserting clips / alerts into the database tables.

Check for database issues.
If not the database, you probably have a storage issue.

Submitting a ticket

Log files

Fix any server errors. The Log files framework explains how to use excel to dissect log files quickly.
The Understanding Log Files article/webinar is a case study on how I discovered NAS issues for a user.

If you really believe the issue is with BI software, provide the supporting evidence.

Describe the issue.
When you discover a gap, send the bvr to us for review.
We need a screenshot of the Alert in the Alerts list. This is how we confirm the camera was triggered/alerted and was supposed to continue recording. Does not apply to Continuous or Periodic record settings.
Specify the time when the gap occurred.
Share your camera settings. Camera settings -> General tab -> Export
Share the logs coinciding with bvr file. ** Very Important **
Support info

What we will do with the information.

Confirm recordings stopped in the BVR file.
Confirm your camera settings are not causing the issue
Check the logs to see what could have caused recordings to stop.
Were there any errors associated with writing to disk?

Before sending to us, you should first do the above to confirm for yourself the issue is NOT with your settings or your server hardware.

Statistics: Posted by varghesesa — Sun Jan 09, 2022 2:34 am

Frameworks • Isolate the issue

2021-12-08T06:16:33+00:00

Introduction

This article provides a framework on how to isolate the issue so BI support can replicate the problem and resolve the root cause. This is the final step in resolving instability as described in the Instability and Troubleshooting article.

Isolate the issue

Reinstall the version causing instability if you reverted back to an older version. Below is an overview of the key components to getting to resolution.

Verify all Security Software Exemptions.
Monitor RAM and CPU usage to verify these are stable leading to the issue.
Attempt to isolate any source of instability by temporarily disabling specific camera, functionality etc.

Windows or Third Party Software
Are updates to Windows or other third party software the issue.

The biggest question to ask yourself when observing instability is what has changed?

Windows update
Was there a Windows update?
While most users (including myself) never suspect a Windows update can cause issues, it is very possible. Common ticket:

I have a blue iris setup, exclusive windows host running for the last couple years without issue. In the last few weeks it has been totally unreliable. I have 3 times wiped the computer and started from scratch, once replacing the hard drive.

The fix was rolling back a Windows update.

Yesterday, I did an uninstall of the latest windows updates. The computer has since been running fine for the last 6 hours or so, versus crashing every15 minutes.

Another testimonial from another user.

This user had RAM issues after an update.

UH-Oooo what happened ? this was before the latest, Ver xxx? I’ve never ran over 4 gig RAM. Hey this all started as a CPU thing.. we should have never said RAM. (HAHA).

What I am seeing is something happens and the RAM grows a bit.. then it falls back but stabilizes at a higher amount. Then something happens again and it rises,, then it falls back to a higher amount again..

OK about my memory issue.. seems it was a windows update fault?
The KB5005539 (.net update) I think was the issue, and strangely KB5006670 seems to have fixed it.

If you suspect a Windows update is causing instability, use System Restore to remove the update.
See Windows Tools article for details.

Anti-virus & Firewall
Was there an update to your Antivirus or Firewall software?

Common framework

This section explains how to isolate BI functionality in order to identify root cause. A detailed description of your setup is not isolating the issue. The cool part is you can leverage BI functionality to isolate any source of instability by temporarily disabling specific camera(s), functionality etc.

Rule out hardware acceleration (HA) by turning it off
Rule out working cameras
Rule out camera features that are working fine.

Hardware acceleration
The usual culprit. Turn off hardware acceleration for all cameras.
Global Settings -> Cameras tab. Set Hardware accelerated decode = No.
Do the same for each camera. Camera settings -> Video tab.

Does stability return? If so, scroll down to Isolate Camera & HA section to determine whether there is HA incompatibility with all cameras or some cameras.

Isolate camera(s)
At this point, server is still crashing even though HA is disabled.
If not, skip to Isolate Camera & HA section below.

New camera
Did you add a new camera? Maybe the conflict is with the hardware and that one particular camera.
Simply disable the camera. Does stability return?

If the software crashes too soon after startup, you may need to leverage the registry to isolate the problematic camera.

Camera feeds
This section determines whether the issue is with the video feeds coming from the camera or features in BI.
Turn the shield to red. This turns off everything besides streaming. If the server still crashes it is one or more camera feeds.
While the server is down, use the registry to disable all the cameras.
Start the server and re-enable cameras one by one to identify the offending camera(s).

For the offending camera(s), go through all the settings on the camera and self-correct by simplifying the streams.
The Camera setup article has details.

BI Camera settings
If the server is stable with the shield red, the camera feeds are fine.
If the video feed is not causing the crash, then it must be the camera settings on a camera(s).
Time to identify the offending camera(s) and the camera setting(s) in BI causing instability.

Disable all the cameras.
Turn Shield to green.
Enable cameras one by one. Isolate the bad camera(s)

Restart BI if crashed.
Isolate the bad setting(s) on the camera(s) that causes the crash by turning features off, e.g. DeepStack.
The issue often lies with settings in the Trigger, Record or Alerts tab in Camera settings.
Report issue to support. See Submitting ticket.

Isolate Camera & HA
At this point, we know HA is causing crashes. However, we have not identified an offending camera.
See if incompatibility between HA and a camera(s) is causing the instability.
If you turned off HA for all cameras and stability returned, follow steps below.

Disable all your cameras.
Turn HA back on for all the cameras at the global level.
Global Settings -> Cameras tab -> Hardware accelerated decode. Choose appropriate setting.
Enable HA for each camera one by one and confirm stability.

The camera(s) that causes instability is the culprit. Turn off HA for those camera(s).

If all the cameras are unstable, HA is the problem. Turn HA off for all the cameras.

Crashes are frequently due to updated hardware drivers that introduced bugs. If HA is suddenly causing instability for all cameras, was there a Windows update recently? If a Windows update is the issue, then you may need to leave hardware acceleration (HA) off until there is a Windows resolution. Or you may have to revert back to the previous Windows version. See System Restore section in Windows Tools article.

Additional questions

The common framework helps identify common causes for instability.

Is hardware acceleration causing instability?
Which camera(s) are causing the issue?

Did you recently add a camera?
If you turn off cameras from a particular vendor, does stability return?

If a camera causes instability, is the issue with the video feed?
If not, is the issue with a particular camera setting? What is the root cause?

However, many issues are due to the combination of features plus your environment. Below are additional questions to ask yourself to get to the root cause.

Think of Blue Iris as blocks or modules that work together. The point of below questions is NOT to answer every question. The point of below is to help you think of relevant questions and investigate further to identify how to reproduce the error.

If stability does not return after turning the shield to red what else is on? This is a big clue.
Are you streaming to YouTube or website?
Are you streaming to your Home Automation system?
Are you posting to a website?
Are there inputs to BI besides cameras? Any DIO devices?
Is auto-cycle on? Group cycle or camera cycle? Turn it off. Does stability improve?
Are you uploading files to the cloud via FTP?
Are you constantly streaming video to a Web Interface, iPad, via mqtt or Webcasting? Does stability improve if you turn it off?
Do you notice crashes when someone connects remotely via the web interface or mobile apps?

In summary, we love customer support that leads to the narrowing down of the problem (usually with reproducible steps) so engineering can then determine where in the software to look and identify the problem.

Submitting ticket

Support info: Include support info in body of email. Global settings -> About tab -> Email support button. Copy/paste information in body of email.
Describe the issue.

Were you able to isolate the issue? Can you reproduce the crash? If so, what are the reproducible steps?

Server health

Confirm any and all errors in the logs were addressed and resolved prior to submitting a ticket.
Logs from previous month. Need to see a full month of logs.
Camera stats. Status button -> Cameras tab -> Export
Storage. Screenshot of Status button -> Storage tab.

Windows Information

Include the event viewer crash info. The Troubleshooting and FAQ section in Help provides details. Copy / paste information into body of email. Any other clues seen in Event viewer such as other frequent Errors.

Statistics: Posted by varghesesa — Wed Dec 08, 2021 6:16 am

Frameworks • Storage Troubleshooting - Clips and archiving tab

2021-11-03T17:32:34+00:00

Introduction

This article is focused on the Clips and archiving tab, i.e. Storage. If you are new and are trying to understand how camera recordings tie into storage on your file system (Clips and archiving tab), the Storage and Recordings section of the Checklist is the place to start. The webinar walks through the Help files and provides the foundation of how Recordings (from cameras) tie into Storage (Clips and archiving tab).

This article provides a framework for troubleshooting storage issues and documents learnings from past tickets (Gotchas section).

If you were directed here, see Next steps / Submitting a ticket section for needed information to move forward. Simply stating you have an issue will not lead to resolution.

Troubleshooting

Providing just a description of the issue will not resolve the issue.

Framework

Follow steps below to address storage issues.

Basics
Make sure you followed best practices. FYI, reviewing entire Windows tuning article is always beneficial.

Self-correct
Double check your Global settings -> Clips and archiving tab settings. Confirm your cameras are recording files to the New folder or other folders based on your settings.

Seek guidance from the Help button. Watch the Storage and recordings webinar.

BI stats and warnings
Self-correct BI warnings and errors. Status button -> Clip storage tab. Help button provides details.

Similar error when seen in the log files.

Code:

2 2/7/2022 11:59:19.456 PMClips               C:: 1427, 1.46T/1.95T (-26.3G)

Status -> Storage tab. Example of clip storage that should raise concerns. Red zones are an issue.

User went through best practices listed above (Basics section) to resolve the issue:

After above fixes:

Notice how both drives have remaining green (unused) space. This is a good practice to NOT allocate the ENTIRE drive to BI.

Logs

Archive thread
The archive thread is responsible for moving files based on your Clips and archiving settings. If you have Log clip move/delete details selected in Global settings -> Clips and archiving tab, then below log entries every 5 minutes indicates the thread is running properly.

Code:

0 1/1/2022 12:07:22.765 AMNew                 Delete: nothing to do [719/720 hrs, 11.0G/100.0G, 297.1G free] 1 locked0 1/1/2022 12:07:22.766 AMAlerts              Delete: nothing to do [167/168 hrs, 50.2M/20.0G, 297.1G free]

The actual details will be different based on your Clips and archiving setup. The 0 code event starting with Delete:, however, are the identifiers associated with archive thread events.

Log files
The logs will tell you if BI is having issues with your Storage. In order to know which errors to look for, we are documenting all the Status log errors. The ones concerning storage are in the Storage category and Recordings category. Each log error provides details and possible fixes to self-correct.

The Log files article explains how to easily deal with log files using excel.

The NAS Case Study article is a case study on how to leverage the logs to get to Storage issues.

Log tab
Might be easier to look for errors in the Logs tab if you check logs frequently. See Routine maintenance guidance for best practices. Keep in mind the Log tab is a snapshot in time of the recent events. Just because Storage and Recording errors do not pop up in the Log tab does NOT mean Storage and Recording errors do NOT exist.

Tools

Many tech saavy users have identified storage access errors or hard drive issues using the following tools. Understanding how to use each tool is up to you.

windirstat
CrystalDiskInfo
procmon
sfc: Windows tool. Scans and verifies the integrity of all protected system files.

Next steps / Submitting a ticket

See Storage ticket for details.

Gotchas
This section documents key learnings from past tickets.

Clips list / Alerts list

Issue: My clips list / alerts list is blank.

It's very unlikely BI deleted your recordings. Check the filesystem where you told BI to save your recordings. For example my New folder saves to G:\BlueIris\New folder. Global settings -> Clips and archiving tab.

And when I look at the content in the folder, I can confirm my recordings are there.

As you can see, recordings are saved based on the camera short name, e.g. Cam6. Since my short name Cam6 is assigned to an existing camera in BI, the recordings and alerts appear in the Clips list and Alerts list

If I change the short name to something else, e.g. Cam50, the Clips/Alerts list will be empty. Make note, the recordings and alerts are NOT deleted. The association between the camera short name and entries in the database, alerts and clips, are now broken. As soon as I change the camera short name back to Cam6, all my alerts and recordings reappear in the UI.

So, if your clips list and alerts list is suddenly empty, first confirm the recordings exist. Second, make sure the camera short name for one of your cameras did not change.

NAS

The category is NAS but this section can apply to RAID storage as well. Basically any non-internal drives.
USB drives may also apply.

Gotcha 1: Playback

Many permutations of the symptoms, but basically playback is NOT working and your storage includes non-internal drives.

New folder and database folder
If you are using the external storage for the New folder and/or database folder, move those folders back to your local hard drive, e.g. C: drive.
Create a small New folder on your local drive (c:) (via Clips and archiving) and delete files after 1 day so as not to fill up your c: drive.
If your database folder is NOT local, move the database folder as well.
See if playback starts working?
Then you know the issue is with your storage. This is how you identify the root cause by turning features off or simplifying your setup.

Windows
Confirm you setup storage correctly so Windows is NOT in conflict.
Double check you connected the Storage device correctly.
It's good to review the checks in the entire article.

Troubleshooting external storage
Instead of writing directly to external storage (New and Database folders on external storage), see if moving files to Stored from New works?
Can you now playback from the raid drive? Example of simplifying the setup and observing if functionality works.

Logs
Check the logs for Record and Storage errors. Log files are NOT the same as the Log tab.

Gotcha 2: Storage fills up.

Any time there are problems with NAS devices, it almost always has to do with permission issues or the NAS stopped communicating with the BI server. Unfortunately, the issue manifests itself with the drive containing the New folder (often the C: drive) filling up and crashing the server.

Common issues/fixes:

Database -> Repair / Regenerate may resolve the issue
Double check your Anti-virus/Firewall settings.

The logs will show the write errors well before filling up the C: drive, but this requires users paying attention to the Status button when it turns red or occasionally monitoring their logs for errors.

The tell tale errors are MoveFile Errors as seen in the below log snippet. Routinely check logs for Storage and/or Record errors.

Code:

0 7/19/2021 7:32:01.679 PMNew                 Move: 2 items 3.66G [602.1G/600.0G, 326.2G free] 126 locked0 7/19/2021 7:32:01.788 PMStored              Delete: 2 items 4.24G [2.93T/2.92T, 1.99T free]0 7/19/2021 7:32:01.791 PMAlerts              Delete: nothing to do [1439/4800 hrs, 895.5M/100.0G, 329.9G free]1 7/19/2021 7:32:01.860 PMClips               MoveFile Error 2: F:\BlueIris\New\FrontDoor.20210719_080000.bvr1 7/19/2021 7:32:01.925 PMClips               MoveFile Error 2: F:\BlueIris\New\FrontDoorStation.20210719_080000.bvr

The above errors are stating BI had errors moving files to the F: drive (NAS drive).

The Fix:
Use the exact same Windows username/password on your NAS device to provide the easiest chance for BI to access your NAS storage.
Check the Storage tab in the Status button to see whether BI recognizes your NAS device. There should be no RED regions.

Second, you need to make sure your NAS is working properly. Pay attention to log errors as mentioned above. The Log Files - Storage/Record Issues article documents how to approach the log files in order to understand the issue. In this case, the user's NAS would run correctly for 21 straight days before denying access to BI and causing BI to crash a few days later.

Below is a quote from another user that decided to replace his NAS storage.

I went to the Status page, which showed the overallocated storage, and clicked on Repair DB. Then slowly the data migrated to NAS storage. Maybe it lost contact with the NAS and needed a kick to restart it. All is well, now.
I'll download the NAS logs and give you more information shortly.
I tried to dump the logs on the NAS, but it hung up and now won't boot. Probably too much log. So it looks like the NAS just wasn't responding like it should.

Troubleshooting your NAS device is beyond BI support.
The forum may provide insight.

Statistics: Posted by varghesesa — Wed Nov 03, 2021 5:32 pm

Frameworks • Instability and Troubleshooting

2021-10-21T03:58:59+00:00

Introduction

Blue Iris is challenging because there are so many external dependencies that could cause issues with BI. Below are a few of the many external dependencies:

Windows
Server hardware
Networking
Hardware drivers from Intel or NVidia for decoding live streams
Camera vendors to provide reliable streams with supported encoding.
Gmail or other email servers for email alerts
Apple / Google platform for mobile app push notifications
Custom integrations via http request and mqtt interfaces

All these 3rd party dependencies can lead to instability.

Some users mistakenly believe sending us their Logs and/or Event viewer details will allow us to resolve their issue. The event viewer details are primarily used to confirm the crash was indeed due to Blue Iris.

Any known issues are documented. This article explains how to resolve unknown issues. If the issue is unknown, i.e. BI is crashing in your environment, isolating the issue is key to resolution.

The Blue Iris status log webinar walks through this article if you prefer to listen instead.

Routine maintenance

Check server health section for any known issues. See if resolving those issues restores stability.

Version control

If you believe a recent update introduced instability, fall back to the prior version and see if stability returns. Fortunately, BI provides a lot of functionality in regards to version control.

If the instability makes the system unusable (more than one crash per day), rolling back your BI version or Windows update may be the interim solution to get your system running again. When you have time to investigate, then reinstall the latest update and proceed through article.

Windows tuning article

The place to start is with the Windows tuning article to confirm there are first no conflicts between BI and Windows. In this article, walk through all the checks. Double check your Anti-virus and Firewall exemptions.

If you have not done so already, consider running BI as a service. By doing so, Windows will automatically restart BI when it runs into issues. This gives you time to further investigate instability while also keeping the server running. See Running as a service section in Help.

Startup

If the software does not start or crashes quickly, the only remedy is to adjust the registry until the software does start. The Software Startup Issues article will walk you through necessary steps.

Instability

Windows Event Viewer
The Windows Event Viewer confirms application crashes and the exception associated with the crash. In the BI Help file, follow instructions in the Troubleshooting and FAQ section to examine the Windows Event Viewer. If you confirm BI is crashing, review the Windows Events article to understand the issue.

From this article you should be able to determine:

Which version of BI first introduced instability.
The exception associated with the crash. Understanding of the exception

Stability
This section explains how to restore stability.

Third party dependencies
Removing known instability is the first step.

Hardware acceleration
Temporarily disable hardware acceleration, especially if you see GPU errors in the logs. This is often a source for instability.

Cameras
Disable / disconnect cameras that lose connectivity frequently.

After searching the logs over the last few days and paying attention to what preceeded the unexpect stops. I have discovered a camera that seems to be rebooting constantly when there is no light in the area. I have disabled that camera and the unexpected stops have not occured for the last 2 days.

Network
A reliable network is a huge factor in BI running correctly. We are not networking engineers. We outline some known pitfalls in the Network tests section.

CPU Utilization
If CPU utilization is >65%, this article explains ways to optimize your server.

RAM Utilization
This article explains how to deal with RAM issues.

Re-install windows

Identifying the root cause for crashes can be a mystery. We have known users that uninstalled/reinstalled BI, deleted and recreated cameras, reinstalled windows then BI, reinstalled phone apps, unplugged/replugged their hard drives and things miraculously started working again.

Customer anecdotal

So after concluding that BI was simply unuseable due to it's random, frequent crashes, in desperation I uninstalled BI. To be clear I did the Windows uninstall, then deleted the leftover BI folders from the C:\ drive and then used the Windows regedit app to delete the registry items for Perspective Software in HKEY_Current _User/software and HKEY_Local_Machine/software. Once this was done I then re-installed BI5 and re-configured the cameras, etc.
Now BI seems to be stable. It has not crashed randomly for two days now which is a huge improvement over it's previous behaviour.
I will monitor the situation going forward but the initial signs are good.

If you want to test your hardware before reinstalling Windows, you could run sfc /scannow or DISM to see if the hard drive or Windows has issues. You would need to search on YouTube for guidance. We are not Windows/hardware experts. Posting a question to the forum is a good idea as well.

A very viable next step at this point is to simply rollback Windows using System Restore or re-install Windows using the many Windows options. The Windows Tools article provides options and details.

Isolate the issue

At this point, it is safe to suspect something with your usage or environment is causing instability.

Root cause

At this point, you have to decide whether you want to help engineering identify the root cause. Not everyone has the technical expertise to help engineering get to the root cause.

The Isolate Issue article provides an example of providing the right information to quickly identify the root cause. If you have a good handle of BI functionality and IT, we appreciate any and all support.

The Server crashes ticket is another example of a good ticket that actually led to resolution.

If you do not have this skillset, no worries. BI provides some cool features to save the previous version of the software in case you need to roll back quickly and/or want to change update schedule to latest critical. You may want to consider changing your auto-update schedule at this time to avoid future instability. Version Control article has details.

If you are still reading, you are interested in helping identify the root cause. See Isolate the issue - Framework article.

Statistics: Posted by varghesesa — Thu Oct 21, 2021 3:58 am

Frameworks • Storage Troubleshooting - NAS Case Study

2021-08-03T23:08:01+00:00

Introduction

When there are issues with your BI Server, the log files are a good place to start to understand the problem. The purpose of this article is to provide an example of how we use the logs in order to get hints as to the underlying issue. This article provides guidance on how to examine the log files. There are many causes for a server crash or poor performance. This article should at least allow you to diagnose and self-correct or provide key information to help BI Support diagnose the issue.

There is a recorded webinar associated with this article if you prefer to listen rather than read. The Blue Iris status log.

Case Study

NAS Write Errors

I found this ticket interesting because the problem did not appear immediately. It took several days of running before the problem appears. This user had a small/medium sized server load with 14 cameras. The July log file shared for diagnosing the issue had 198,901 lines from July 1 - July 20.

So how do you approach the problem?

Start with the symptoms. The user reported:

Every few months (although not consistently) my primary drive gets full and goes past the limits (600G of 1T) and then BI stops moving files to the ‘stored’ network folder.

Analysis / Thought Process

From this information, I knew there is probably an issue with Global settings -> Clips and archiving. This means there should be Storage and/or Recording errors.

The logs happened to begin with camera connectivity errors. However, the ticket is in regards to storage, so I mentioned the issue in my analysis report (further below) but did not focus on this issue.

Code:

0 7/1/2021 12:00:00.449 AMApp                 Today's sunrise: 7:11 AM, sunset: 10:05 PM1 7/1/2021 12:00:03.467 AMHVAC                Signal: network retry1 7/1/2021 12:00:26.291 AMHVAC                Signal: Failed to connect1 7/1/2021 12:00:33.612 AMHVAC                Signal: network retry1 7/1/2021 12:00:56.176 AMHVAC                Signal: Failed to connect3 7/1/2021 12:01:01.581 AMWine                MOTION1 7/1/2021 12:01:03.756 AMHVAC                Signal: network retry1 7/1/2021 12:01:26.394 AMHVAC                Signal: Failed to connect1 7/1/2021 12:01:33.969 AMHVAC                Signal: network retry1 7/1/2021 12:01:56.624 AMHVAC                Signal: Failed to connect

The user stated the logs were created since the last crash. I went to the end of the logs and confirmed many, many Clip: Disk full (recording) errors at the end of the log. Thus, I knew these logs contained a point of time where the server was functional and crashed due to Storage & Recording issues as the user stated. The logs should tell me the problem.

Code:

2 7/20/2021 3:40:35.909 PMBasketball          Clip: Disk full2 7/20/2021 3:40:35.914 PMGarage              Clip: Disk full2 7/20/2021 3:40:36.237 PMMudRoom             Clip: Disk full2 7/20/2021 3:40:38.156 PMBasementEntrance    Clip: Disk full2 7/20/2021 3:40:38.342 PMDriveway            Clip: Disk full2 7/20/2021 3:40:39.445 PMMudRoomSide         Clip: Disk full2 7/20/2021 3:40:40.477 PMSide                Clip: Disk full2 7/20/2021 3:40:40.721 PMHVAC                Clip: Disk full2 7/20/2021 3:40:41.448 PMDrivewayRight       Clip: Disk full2 7/20/2021 3:40:41.589 PMFrontDoorStation    Clip: Disk full2 7/20/2021 3:40:41.702 PMPoolEq              Clip: Disk full2 7/20/2021 3:40:42.570 PMPool                Clip: Disk full2 7/20/2021 3:40:44.314 PMFrontDoor           Clip: Disk full2 7/20/2021 3:40:45.788 PMDrivewayLeft        Clip: Disk full2 7/20/2021 3:40:45.924 PMGarage              Clip: Disk full2 7/20/2021 3:40:45.974 PMBasketball          Clip: Disk full2 7/20/2021 3:40:46.316 PMMudRoom             Clip: Disk full2 7/20/2021 3:40:48.202 PMBasementEntrance    Clip: Disk full

So then, I went back to the top of the log files. I started scrolling down, scanning for Storage / Record errors. After scrolling through several, several pages (remember logs have 200k lines), I saw the below common pattern repeatedly.

Code:

0 7/1/2021 3:47:40.393 AMNew                 Move: 2 items 865.3M [600.8G/600.0G, 328.4G free] 80 locked0 7/1/2021 3:47:40.397 AMStored              Delete: nothing to do [2.92T/2.92T, 1.99T free]0 7/1/2021 3:47:40.412 AMAlerts              Delete: nothing to do [992/4800 hrs, 625.4M/100.0G, 329.2G free]

The Clips and archiving settings were fairly simple: New -> Stored.
I also starting observing that New after the (successful) Moves would hover around 328GB free, i.e. BI is managing storage correctly.

Since the log file was so large, I simply started searching for the next Move: command. With search, I quickly pounded through all the Move commands observing whether New remained around 328 GB (sometimes 330 GB, sometimes 327 GB).

Even with search, getting through all the Move commands in just ONE day took several, several clicks. I then would scroll down to the next day, then search for the next Move: command. By doing, I was fairly quickly able to narrow down the issue to the day. Eventually (not going to say quickly), I found the first Move: error.

On July 19, 2021 I found the first Move error (the needle in the haystack). This was line 147,021 in the log file. Notice how free storage for the first time dropped below 327 GB. Looking back, I probably should have started from the bottom of the file and searched up.

Code:

0 7/19/2021 7:32:01.679 PMNew                 Move: 2 items 3.66G [602.1G/600.0G, 326.2G free] 126 locked0 7/19/2021 7:32:01.788 PMStored              Delete: 2 items 4.24G [2.93T/2.92T, 1.99T free]0 7/19/2021 7:32:01.791 PMAlerts              Delete: nothing to do [1439/4800 hrs, 895.5M/100.0G, 329.9G free]1 7/19/2021 7:32:01.860 PMClips               MoveFile Error 2: F:\BlueIris\New\FrontDoor.20210719_080000.bvr1 7/19/2021 7:32:01.925 PMClips               MoveFile Error 2: F:\BlueIris\New\FrontDoorStation.20210719_080000.bvr

Analysis Report

From here, it was apparent BI could not move files to Stored and would eventually fill up the New drive and crash. Analysis report below.

Everything seems to be running ok. Move 1 files to Stored. Delete 1 file in Stored to make space.

0 7/8/2021 5:23:43.700 PMNew Move: 1 items 3.20G [602.6G/600.0G, 326.2G free] 100 locked
0 7/8/2021 5:23:43.919 PMStored Delete: 1 items 1.38G [2.93T/2.92T, 1.99T free]
0 7/8/2021 5:23:43.923 PMAlerts Delete: nothing to do [1173/4800 hrs, 712.4M/100.0G, 329.4G free]

Continues working until 7/19 at 7:27 PM

0 7/19/2021 7:27:01.363 PM New Move: nothing to do [599.8G/600.0G, 328.5G free] 126 locked
0 7/19/2021 7:27:01.366 PM Stored Delete: nothing to do [2.92T/2.92T, 2.00T free]
0 7/19/2021 7:27:01.373 PM Alerts Delete: nothing to do [1439/4800 hrs, 895.5M/100.0G, 328.5G free]

On 7/19 @7:32 PM, first time BI cannot move files to the NAS.

0 7/19/2021 7:32:01.679 PM New Move: 2 items 3.66G [602.1G/600.0G, 326.2G free] 126 locked
0 7/19/2021 7:32:01.788 PM Stored Delete: 2 items 4.24G [2.93T/2.92T, 1.99T free]
0 7/19/2021 7:32:01.791 PM Alerts Delete: nothing to do [1439/4800 hrs, 895.5M/100.0G, 329.9G free]
1 7/19/2021 7:32:01.860 PM Clips MoveFile Error 2: F:\BlueIris\New\FrontDoor.20210719_080000.bvr
1 7/19/2021 7:32:01.925 PM Clips MoveFile Error 2: F:\BlueIris\New\FrontDoorStation.20210719_080000.bvr

From there on, drive containing New starts getting full until BI crashes. Drive containing New folder was steady at 329 GB free for many days. Now it is slowly depleting.
Below snapshot shows, 10+ GB have been eaten (316.6 GB free).

0 7/19/2021 7:52:03.882 PM New Move: 6 items 14.5G [611.8G/600.0G, 316.6G free] 126 locked
0 7/19/2021 7:52:03.885 PM Stored Delete: nothing to do [2.92T/2.92T, 1.98T free]
0 7/19/2021 7:52:03.895 PM Alerts Delete: nothing to do [1440/4800 hrs, 896.0M/100.0G, 331.2G free]
1 7/19/2021 7:52:03.966 PM Clips MoveFile Error 2: F:\BlueIris\New\Garage.20210719_080000.bvr
1 7/19/2021 7:52:04.025 PM Clips MoveFile Error 2: F:\BlueIris\New\BasementEntrance.20210719_080000.bvr

Last move before BI crashes. Only 4k left on hard drive.

0 7/20/2021 10:04:52.773 AM New Move: 129 items 329.6G [929.3G/600.0G, 4K free]
0 7/20/2021 10:04:52.779 AM Alerts Delete: nothing to do [1454/4800 hrs, 897.5M/100.0G, 329.6G free]
1 7/20/2021 10:04:52.800 AM Clips MoveFile Error 0: F:\BlueIris\New\MudRoom.20210719_080000.bvr

Question is why did BI stop communicating with the NAS on July 19?

This ticket is still not resolved. Ken added more details to the log to gather more information. The user is reaching out to the forum and the NAS vendor as well.

I closed the analysis report by including the other issues I found in the log files.

Other recommendations:
Your HVAC camera is always disconnecting. You could try and resolve that issue. See the No signal article.

Statistics: Posted by varghesesa — Tue Aug 03, 2021 11:08 pm