Massive Memory Usage

Rantech
Posts: 37
Joined: Thu Aug 24, 2023 4:07 am

Re: Massive Memory Usage

Post by Rantech »

Are you using Code Project with IPCAM Custom Models ? My issues only began a couple of weeks ago when I decided to try using the ipcam-custom models; prior to that all 6 of the systems I manage were stable even on the newer versions of BI. What I find is that with custom models on (default object detection off), I can force the memory issue on any versions after 5.8.8.12, including 5.9.2.2 which I just tried now. However, even with custom models on, I can not force the error on version 5.8.8.12 or earlier.

Version 5.9.2.2 started running up memory continuously after 2 cycles of editing camera settings ... on it's way to a crash; so unfortunately, it has not resolved the problem.

I have sent support details on the problem but unfortunately I don't think they have been able to recreate it on their end.
MikeBwca
Posts: 1115
Joined: Thu Jun 20, 2019 5:39 am

Re: Massive Memory Usage

Post by MikeBwca »

I was thinking of testing with custom models. I'll keep this in mind.
Rantech
Posts: 37
Joined: Thu Aug 24, 2023 4:07 am

Re: Massive Memory Usage

Post by Rantech »

Interestingly, ver 5.8.9.10 is the first version that had the new "pre trigger record time" and "pre trigger playback time" options so maybe there is something going on with those features and causing something to go awry when the stream starts ? don't know why there would be any interaction between those features and custom models though ...
BruceH
Posts: 89
Joined: Wed Apr 01, 2020 10:10 pm

Re: Massive Memory Usage

Post by BruceH »

My system is running CPAI 2.6.5 on BI 5.9.2.2 and is running "ipcam-combined" on three of my 4 cameras,

Memory has now crept up to 938MB, as it does appear to be creeping up very slowly, I have 32GB of RAM on this system, so if it does continue the slow creep, it might take some time before the system gets in trouble, I will check it again later and see what it is at.
BlueIris 5.9.3.4 CPAI 2.6.5
BruceH
Posts: 89
Joined: Wed Apr 01, 2020 10:10 pm

Re: Massive Memory Usage

Post by BruceH »

The memory creep that I thought I was seeing is no longer progressing, my memory usage has actually dropped back to 913MB again!
BlueIris 5.9.3.4 CPAI 2.6.5
veldthui
Posts: 16
Joined: Tue Jun 18, 2019 5:19 pm

Re: Massive Memory Usage

Post by veldthui »

I am running 5.9.2.2 and see the memory usage creep most times i edit something. Something simple as changing the short name for the camera caused the memory to go from 300 - 500MB to 5.5GB in about 2 hours. I restarted the service and it reverted back to the 300-500MB and has been steady the last 24 hours. I am using the AI but not using the custom models. The AI software seems to be okay as it the BI service that is chewing the memory up.
BruceH
Posts: 89
Joined: Wed Apr 01, 2020 10:10 pm

Re: Massive Memory Usage

Post by BruceH »

Do you have hardware acceleration turned on for any of your cameras, I had it turned (Nvidia) for two of my cameras, when I turned it off my memory usage dropped down, I am still not seeing any huge usage of memory on my system; at the moment it is sitting around 550MB after turning off all camera hardware acceleration. CPU 1/5 GPU 14%
BlueIris 5.9.3.4 CPAI 2.6.5
BruceH
Posts: 89
Joined: Wed Apr 01, 2020 10:10 pm

Re: Massive Memory Usage

Post by BruceH »

velthui

You mentioned changing the short name for the cameras, I hope that you realize that doing that can orphan all of the files that were recorded using the original short name, If you want to change the short name there is a whole lot of files that will have to be renamed once you are done, or your database is going to be a bit of a mess. There is a procedure that you would have to follow to ensure the DB integrity, or you could change the names, shut down BI and wipe out all the BVR and JPG files and the DB and start over fresh!
BlueIris 5.9.3.4 CPAI 2.6.5
Rantech
Posts: 37
Joined: Thu Aug 24, 2023 4:07 am

Re: Massive Memory Usage

Post by Rantech »

Much more testing ... I believe the issue is directly tied to the change in 5.8.9.10 where pre-trigger record and playback times were introduced. I changed all cameras to have both pre-trigger record and playback time = 5 sec to match the buffer time (which was forced in 5.8.8.12 and earlier since there were no options for pre-trigger record and playback) and then restarted BI with 5.9.2.2. I tried for a very long time (at least a dozen restarts on every camera) to get the problem to recur but I could not get it to recur.

I then changed every camera back to pre-trigger record of 3 sec and pre-trigger playback of 1 sec (my original settings) and as soon as the cameras restarted the memory use started to climb and within 20 min I went from 628 MB to 3.51 MB and still climbing.

It appears that there are now 2 workarounds to the problem:
1) restart BI service after changing camera settings (if not using default pre-trigger times)
2) use default buffer, pre-trigger record, and pre-trigger playback of 5 sec on each

Would appreciate hearing if someone was able to recreate the same behavior on a different system.
BruceH
Posts: 89
Joined: Wed Apr 01, 2020 10:10 pm

Re: Massive Memory Usage

Post by BruceH »

Rantech

I checked my times and they are as follows

Pre-Trigger record time = 5.0 sec
Pre-trigger playback time = 2.0 sec

These are the times that the system must have defaulted to as I have recently upgraded to 5.9.2.2, and the prior version 5.8.7.11 did not have these settings

I will try some different values to see if I can replicate your problem, if there is a specific set of numbers you want me to try, just let me know.
BlueIris 5.9.3.4 CPAI 2.6.5
Post Reply