Hi everyone, this is my first new thread since rejoining following the horrible loss of a year's forum content.

For most of the year I've been having an intermittent problem where BI5 (currently 5.8.something) goes loopy, for no obvious reason. I'm in touch with Ken on this but hoping to crowdsource ideas for both what the problem might be, and what I can monitor to pin it down. Basic system specs in my signature; I'm running 11 cams (Dahua/Hikvision). I've learned a lot recently; I must be pretty close to cracking the nut. I store direct to disk as BVR, no re-encode, no moving files around. Main streams only, using BI motion detection, no AI. VERY plain-Jane stuff here.

Here's what happens:

1. Everything working well, then:
2. I notice the BI use of memory starts to skyrocket (normally my BI uses under 2.5 Gb, but during the problem I've seen it eat its way to 11-12Gb)
3. Any interaction with the cameras (app, UI3, the actual BI software) gets sluggish then cameras stop showing, clips won't play
4. Gaps show up in the Timeline view for individual cameras (these gaps persist after a reboot, but may have alerts in them ??)
5. Assorted log entries about signal loss/restore and no images created for alerts
6. (note the cameras themselves are fine and streaming without issue during all this, I can see them via their web interfaces)
7. As it goes on, suddenly BI stops cutting the files to the set sizes and the file sizes will get really crazy; I've seen 20 GB files.

Some things I've learned along the way:

A. It got a lot less frequent after Ken reminded me earlier this year to exclude the storage and database folders from Windows Security/Defender. I've done this (had forgotten to, after a windows reinstall!). But the problem still can happen when Defender realtime scanning is all the way off. So AV scans can aggravate the problem but not actually be the root cause

B. Setting the overall system hardware acceleration off or Intel had no effect on the problem, nor did changing individual camera settings for hardware decode.

C. Because abnormal file sizes is one of the symptoms, I changed first a few, then eventually all, of my cameras to 2GB file size cutting, to make it easy to see which camera's files were impacted. What I noticed is that the more cameras were set to cut to 2 GB instead of larger sizes, the more the problem occurred!

D. In the early morning hours a few days ago, while the house slept, some weird BI error with one of the files resulted in the ENTIRE AUX1 directory on my 8Tb drive --14 days worth of recordings-- to be deleted from the drive. When I woke up and went to look over the drive to check file sizes, to my horror the drive was empty of all files from before the problem cropped up. My other storage drive (4Tb, Aux2 directory used for a subset of the cameras, was unaffected and has 14 days worth of files). My system is set to keep 14 days of recordings, and the amount of space on each drive allocated by BI for storage is safely below the actual drive capacity (6900 GBfor Aux1, 3500 GB for Aux2), . Neither drive is ever over 90% full.

E. After this incident, the "memory leak problem" hasn't happened again. Connecting dots from past tests, I now believe the problem only happens when BI has to actively prune back old files that are expiring. And, interestingly, it seems that having to prune just the 4 TB drive Aux2 folder isn't causing a problem, because I haven't seen the problem since just Aux1 was cleared. In 12 days perhaps the problem will come back (or, earlier, if I test by lowering the allowed # days or space for Aux1 to encourage the fault to occur for testing).

Here is the log entry from the weird thing that resulted in file deletions:
1 10/12/2023 4:51:30.009 AM Clips DeleteFile Error 32: D:\Blue Iris\AFewCams\Intersection.20231012_004428.bvr

This exact error happened 2670 times in rapid succession over the course of an hour (the same filename referenced in each instance).

1 10/12/2023 5:57:09.412 AM Aux 1 Delete: 1 items 1.95G [4/336 hrs, 7.88T/6.73T, 7.26T free] 7 locked 2760 errors (code 3)

I would appreciate any thoughts about what could be causing this and how to monitor the BI system, beyond the standard log, for what could be going on to lead me to the cause and definitive solution.

Thanks!

I'm having the exact same issue. Randomly BI will start saving 1 second 4 GB clips that will put me over the storage quota and nuke the entire database. Have you found a solution to the issue?

Haven't sorted it yet, in large part because after the night where all those files got deleted in Aux1, the problem hasn't recurred.

After all that bother, I reset some of the settings I thought might have been contributing, specifically changing the 2 GB file size back up to 6 GB. I just let it gradually repopulate with data for the past ~9 days. It's been working flawlessly.

Today, I decided to start some tests. I suspected that pruning back the many files when capacity/time limits were hit was contributing to the issue, so this morning I first changed the maximum amount of time files in Aux1 to be kept to 7 days (so, the system had to spend some time pruning older ones). Four hours later, system still operating smoothly, no RAM problems. I next changed the maximum space taken in Aux1 to be just a bit less than the current amount of storage there. System still going strong. I'm going to give it overnight then start messing with the file sizes.

Ken gave me an idea that there might be some issue at root relating to having the database on an SSD. Once I can recreate the problem with some reproducibility, I will try moving the DB to a spinning drive, and work through some things he sent.

For most of the year I've been having an intermittent problem where BI5 (currently 5.8.something) goes loopy, for no obvious reason.

5. Assorted log entries about signal loss/restore and no images created for alerts
6. (note the cameras themselves are fine and streaming without issue during all this, I can see them via their web interfaces)
3. Any interaction with the cameras (app, UI3, the actual BI software) gets sluggish then cameras stop showing, clips won't play

Same problem but I have not been paying attention to memory spikes so they may be happening too. I'm not "cutting" clips so I don't have those issues. The cameras seem to freeze (15 sec to 4 min) with red borders when viewed via the GUI or the mjpg stream. Some cams are wired some are not. My db is on a ssd but if ken thinks this is a ssd issue he needs to fix it, of course. I am currently thinking it might be a issue with watchdog. A bug possibly.

Since I've never had a "freeze" that lasted longer than 4 minutes, I'm also suspicious about the clip maintenace process that runs every five minutes per the BI documentation. When my cameras unfreeze the clip list in the BI GUI refreshes at the exact same time.