Massive Memory Usage

veldthui
Posts: 16
Joined: Tue Jun 18, 2019 5:19 pm

Massive Memory Usage

Post by veldthui »

I have Blue Iris set up with 5 cameras and AI to detect people. Up until this morning the system was running fine. I have 32GB of memory and the specs say BI will run with 4GB.
This morning the App showed my memory usage at 96% and I remoted in and sure enough BI was using nearly all of the 32GB. It was using so much it was causing the remote to crash due to lack of memory available.
BI version 5.9.1.6
Rantech
Posts: 37
Joined: Thu Aug 24, 2023 4:07 am

Re: Massive Memory Usage

Post by Rantech »

Checkout my problem description on page 2 of the following thread ... I'd be curious if your issue is similiar to the description of my problems. The outcome seems the same but do the same actions cause the problem for you?

https://blueirissoftware.com/forum/viewtopic.php?t=3860
veldthui
Posts: 16
Joined: Tue Jun 18, 2019 5:19 pm

Re: Massive Memory Usage

Post by veldthui »

Not sure. I was changing stuff on a new camera. I watched mine for quite a while and it peaked at 96% of the memory used and actually dropped back to 62% for a short time before climbing up again to around 96% and then the system became unstable and I had to shut the Lenovo x3650M5 server down. I have it running windows 11 and BI as a service. After the reboot it appears to have settled down.

It certainly dows not like my new Reolink RLC-823A 16X camera. The PTZ lag is horrendous and takes up to 45 seconds to register the move. I used sub streams but had to knock the main stream down from 3840x2160 to 2560x1440. The sub is at 640x360.

I also couldn't get it to turn off the IR lights from BI while testing it. Also even though it is 2-way audio can't get that to work at all. Was hoping it would work with the app.

I will keep an eye out and see if I have any more issues.
Rantech
Posts: 37
Joined: Thu Aug 24, 2023 4:07 am

Re: Massive Memory Usage

Post by Rantech »

For me, restarting the BI service after changing camera settings resolves the problem and it runs fine. However, it's only guaranteed to work fine after a restart - if I change settings and don't restart, its a gamble. I am running 6 separate BI installations and 6 different facilities and this is the only one of the systems having this issue and I don't know why ... I have one system with 20 cameras and another with 16 cameras, and a system with 6 cameras is the one giving me grief - the others are working well.

Try changing a camera setting then note the ram use immediately after it restarts and watch the ram use for 3 to 5 min to see if it bounces around - up and down but always around that initial ram use; or alternatively, does it just keep climbing so that after 3 to 5 min it is well outside the original range. In my system, if I see this latter behavior, I know the memory use is just going to keep climbing til it gets over 90% and things start crashing. When you see this higher ram use after 5 min or so, if you restart the camera you had edited, does the ram use then drop back down to your initial target value? Does it then stabilize at that or does it start climbing again? If you restart the BI Service, does it start with RAM use around that initial target and stablize? If you can replicate these behaviors then I think you are seeing the same issue that I'm seeing - if not, then you likely have a different issue.
veldthui
Posts: 16
Joined: Tue Jun 18, 2019 5:19 pm

Re: Massive Memory Usage

Post by veldthui »

Okay I just made changes to the cameras short names which stopped and restarted 3 of the 5 cameras.
After an hour the memory usage has climbed from a steady 300 - 500MB for the last 24 hours to 5.5GB of usage. Definitely a big memory hole there somewhere so yes I have the same issue as you. Good after a restart until a camera is edited then huge memory loss.
Rantech
Posts: 37
Joined: Thu Aug 24, 2023 4:07 am

Re: Massive Memory Usage

Post by Rantech »

Good news is that we have a workaround (restart BI Service after every edit).

I think I have further narrowed it down to editing settings on some new Hikvision 180 deg Cameras (2CD2387G2P); not certain but in any case, it definitely is much more likely to happen if I edit settings on these cameras. I have tried changing video settings from Hikvision RTSP to ONVIF and back but that doesn't make a difference. Have changed to H264 video and turned off HA but again no difference. Upgraded CP to 2.6.5 but have not yet determined if that has resolved the issue.
Rantech
Posts: 37
Joined: Thu Aug 24, 2023 4:07 am

Re: Massive Memory Usage

Post by Rantech »

Hmmm ... unfortunately, I was wrong and it is nothing to do with the 180deg hikvision cameras.

Last night, I went to another BI system I manage that had been stable since installed with no ram issues at all. It only has 4 cameras and it is normally less than 500 MB RAM. I updated to BI 5.9.1.6 and CP 2.6.5 then I changed general AI settings to turn off default object detection and turned on custom models. I then set every camera to use only ipcam-general. Within 10 hours the memory use had jumped to 97% on a system with 32GB of RAM.
Rantech
Posts: 37
Joined: Thu Aug 24, 2023 4:07 am

Re: Massive Memory Usage

Post by Rantech »

This has been driving me nuts so I went back to BI 5.8.5.8 without changing any other settings in either BI or CP then repeatedly tried to trigger the issue by making a minor change to settings on every camera over and over again. I was not able to get the problem to occur.

Then I tried 5.8.7.11, and 5.8.8.12 and did the same. I was not able to get the problem to occur on either of those versions despite repeatedly editing and restarting every camera over and over again.

Then I went to 5.8.9.10 and was able to get the problem to occur on my second pass at editing camera settings. I was doing the same changes on every test with each version of BI that I tried.

It looks to me like something changed between 5.8.8.12 and 5.8.9.10 that initiated this issue. Those were the only old versions available from the update page so I was not able to narrow it down any further than this.
MikeBwca
Posts: 1115
Joined: Thu Jun 20, 2019 5:39 am

Re: Massive Memory Usage

Post by MikeBwca »

How does the latest version 5.9.2.2 work? If this resolves it, great.

If not, you two have a lot of specifics that should help BI Support.
BruceH
Posts: 89
Joined: Wed Apr 01, 2020 10:10 pm

Re: Massive Memory Usage

Post by BruceH »

I am running 5.9.2.2 and I am not seeing any memory issue, I have tried making random changes in the camera setting and my RAM usage is remaining stable, it goes up a bit then comes back down. With no activity on four cameras it is sitting between 900MB and 910MB, if there is something more specific I should be testing let me know and I will try it!
BlueIris 5.9.3.4 CPAI 2.6.5
Post Reply