Memory leak and 5.2.3.2 repeatedly crashing in ntdll.dll

Matts1984 · Post by **Matts1984** » Tue Jun 02, 2020 1:16 pm

If it doesn't work out for you, I'd be happy to take that server off your hands.

My installation is on a DL180 G6 with dual X5650's (no QS) and ESXi 6.5 as the base, Server 2019 as the BI guest OS. No complaints and working well, but I always like something new and shiny.

Is it too late to change your server to a E-2224G? Doesn't seem like that would be much of an upgrade (in $$$) but would get you the graphics. Or dual E-2288G's would be nice

miles267 · Post by **miles267** » Tue Jun 02, 2020 1:21 pm

Unfortunately the HPE Gen10 Plus doesn’t support QuickSync even with a Xeon that had integrated graphics.

Actually didn’t even get this new G10+ 32 GB RAM with the intent of using it for BI5, but now I fear I may need to repurpose it as such. How well does BI5 run inside a VM? My WS12R2 box is a Core i5 Ivy Bridge with 32 GB RAM.

Also, might you know which BI5 update versions introduce new MS Visual C++ library versions? I’m trying to update to any version past April 1, 2020 that doesn’t update VC++. It’s at that point that ntdll.dll issues begin.

Sent from my iPhone using Tapatalk

Matts1984 · Post by **Matts1984** » Tue Jun 02, 2020 4:27 pm

I can say that my VM runs very well and I've had no issues at all. The only limitation I've encountered is the artificial restriction to 8 cores per VM on the ESXi free license. My combined processors could provide 12 cores but I'll just have to use the remainder for other VMs. With the 8 cores currently allocated (and note my CPUs were released around 9 or 10 years ago), I cruise at 10% CPU with spikes to around 25 or 30%.

Unfortunately I have no idea on the C++ update issue you've run into.

miles267 · Post by **miles267** » Tue Jun 02, 2020 5:24 pm

The Xeon E-2224 CPU in the HpE G10+ is 4 cores and 4 threads. No hyper threading. So not sure how well this machine would fare with BI5 and the multiple cameras I’m currently using. I’m trying to unravel this ntdll.dll corruption issue on my dedicated WS2012R2 box so that it can be repurposed as a dedicated BI5 server that has a working server backup.

Also have Microsoft’s SBS Engineers involved in a ticket on this so we’ll see what they uncover.

At a minimum, I can reproduce that BI5 and the latest VC++ libraries 2015-2019 version in BI 5.2.9.6 have exposed an issue that corrupts ntdll.dll in WS2012R2.

Sent from my iPhone using Tapatalk

Shyabrick · Post by **Shyabrick** » Sun Jun 07, 2020 1:27 pm

SpeedCam wrote: ↑Wed Apr 01, 2020 7:06 pm I've had to get a PowerShell script monitoring and restarting the service as after it dies due to memory exhaustion it doesn't start back up again.

Pardon my ignorance. I've never played with PowerShell. Could you share the script or where would I find PowerShell info to learn how to create a script? My BI crashes 2-4 times a day... unreliable.

MikeBwca · Post by **MikeBwca** » Mon Jun 08, 2020 11:58 pm

Any update on these issues?

SpeedCam · Post by **SpeedCam** » Mon Nov 09, 2020 4:29 am

I upgraded my BI VM to Windows 10 (version 2005, build 19041.572) about two weeks ago. Currently running BI v5.2.6.5.
The regular leaky behavior I was seeing <b>has gone away with upgrading to Windows 10</b>; prior to the upgrade BI was still quickly leaking until it exhausted system memory and crashing about every hour as a result. In this new configuration BI holds pretty steady w.r.t. memory utilization. I can now drop my VM's memory allocation!
Shortly after boot in Win10:

: Shortly after startup, 3,631,660K private memory; Untitled.png (7.27 KiB) Viewed 8195 times

After about 6-7 hours later of running (with no crashes):

: About 6hours later, 3,771,700K private memory; Untitled3.png (17.76 KiB) Viewed 8195 times

However, I'm still seeing what appears to be regular crashes of BI, maybe one every 12 hours. They're spread out across modules though most appear to be in VCRUNTIME140.dll; having said that VCRUNTIME140.dll is probably a victim module assuming these crashes are due to memory corruption given the 0xc000005 Access Violation error code. Crashes seen in last 4 days:

Error 11/4/2020 4:00:07 PM Application Error 1000 (100)

Code: Select all

Faulting application name: BlueIris.exe, version: 5.2.6.5, time stamp: 0x5ea8376b
Faulting module name: VCRUNTIME140.dll, version: 14.21.27702.2, time stamp: 0x5ccadaa9
Exception code: 0xc0000005
Fault offset: 0x0000000000001495

Error 11/6/2020 11:30:01 AM Application Error 1000 (100)

Code: Select all

Faulting application name: BlueIris.exe, version: 5.2.6.5, time stamp: 0x5ea8376b
Faulting module name: VCRUNTIME140.dll, version: 14.21.27702.2, time stamp: 0x5ccadaa9
Exception code: 0xc0000005
Fault offset: 0x0000000000001478

Error 11/6/2020 12:16:06 PM Application Error 1000 (100)

Code: Select all

Faulting application name: BlueIris.exe, version: 5.2.6.5, time stamp: 0x5ea8376b
Faulting module name: ntdll.dll, version: 10.0.19041.546, time stamp: 0x5b56177b
Exception code: 0xc0000005
Fault offset: 0x000000000001a904

Error 11/6/2020 12:31:52 PM Application Error 1000 (100)

Code: Select all

Faulting application name: BlueIris.exe, version: 5.2.6.5, time stamp: 0x5ea8376b
Faulting module name: BlueIris.exe, version: 5.2.6.5, time stamp: 0x5ea8376b
Exception code: 0xc0000005
Fault offset: 0x0000000000da456d

Error 11/6/2020 6:45:01 PM Application Error 1000 (100)

Code: Select all

Faulting application name: BlueIris.exe, version: 5.2.6.5, time stamp: 0x5ea8376b
Faulting module name: VCRUNTIME140.dll, version: 14.21.27702.2, time stamp: 0x5ccadaa9
Exception code: 0xc0000005
Fault offset: 0x000000000000147c

Error 11/7/2020 1:10:01 AM Application Error 1000 (100)

Code: Select all

Faulting application name: BlueIris.exe, version: 5.2.6.5, time stamp: 0x5ea8376b
Faulting module name: VCRUNTIME140.dll, version: 14.21.27702.2, time stamp: 0x5ccadaa9
Exception code: 0xc0000005
Fault offset: 0x000000000000147c

Error 11/8/2020 12:00:01 AM Application Error 1000 (100)

Code: Select all

Faulting application name: BlueIris.exe, version: 5.2.6.5, time stamp: 0x5ea8376b
Faulting module name: VCRUNTIME140.dll, version: 14.21.27702.2, time stamp: 0x5ccadaa9
Exception code: 0xc0000005
Fault offset: 0x00000000000014b7

I acknowledge that this list format is completely useless for actually identifying the issue, it's just to demonstrate BI's still a bit crashy for me, for whatever that's worth. =)

SpeedCam · Post by **SpeedCam** » Mon Nov 09, 2020 5:11 am

Shyabrick wrote: ↑Sun Jun 07, 2020 1:27 pm
SpeedCam wrote: ↑Wed Apr 01, 2020 7:06 pm I've had to get a PowerShell script monitoring and restarting the service as after it dies due to memory exhaustion it doesn't start back up again.
Pardon my ignorance. I've never played with PowerShell. Could you share the script or where would I find PowerShell info to learn how to create a script? My BI crashes 2-4 times a day... unreliable.

Sure, the script runs through and (re)starts any services that aren't running but are configured to auto-start on system start, that way the script is good for any crashy service you might have.

The script is as follows, in my case saved as file "C:\bin\WindowsServices_StartAutostartServices.ps1":

Code: Select all

#Get services that should be started (StartMode = Auto) but are stopped (State = Stopped), then start these services 
$Logfile = "C:\bin\WindowsServices_StartAutostartServices.ps1.log"
#Add-content $Logfile ("{0} - Started" -f (Get-Date))
Get-WmiObject -class Win32_Service -computername . -namespace "root\CIMV2" | Where-Object {$_.StartMode -match "auto" -and $_.state -match "stopped" -and $_.PathName -notlike "C:\Windows*"} | Add-content $Logfile | Add-content $Logfile ("{0} - Done" -f (Get-Date))
Get-WmiObject -class Win32_Service -computername . -namespace "root\CIMV2" | Where-Object {$_.StartMode -match "auto" -and $_.state -match "stopped" -and $_.PathName -notlike "C:\Windows*"} | Start-Service

It is scheduled via taskschd.msc to run every minute. You can create a scheduled task like mine, assuming you save the file with the same folder\filename by running the following:

Code: Select all

schtasks /create /TN "Start any stopped auto-start services" /ru IRIS10\blueiris /rp * /SC DAILY /RI 1 /DU 24:00 /TR "powershell.exe -ExecutionPolicy unrestricted -File c:\bin\WindowsServices_StartAutostartServices.ps1" /RL HIGHEST

Note in my case I have the scheduled task to run in the same user context that runs my Blue Iris instance as it was granted permissions to access my NAS, change as appropriate for your setup.

TheQCC · Post by **TheQCC** » Sun Nov 22, 2020 4:11 am

I have a major issue with Blue Iris 5 and memory leaks..
Blue Iris 5 is filling up the non paged pool memory and crashing the computer (2)
The fault seems to be in netio.sys as shown by the Windows Pool Monitor.
If the Blue Iris service is stopped the non paged pool does not vary by more the 50MB. As soon as the Blue Iris service is restarts I can sit there and watch the non paged pool increase to fill the free memory.
I am running Blue Iris dedicated on a HP Z640 workstation with a backup on a HP Z240 workstation. Both with Win 10 Pro and 32GB of RAM
The Z640 has a Xeon 10 core (20 v cpu's) and five cameras.
I have a ticket in Blue Iris support, but have yet to get a response.

My jury-rigged solution is to have Task Scheduler restart the computer every two hours. Yes, two hours as that is how long Blue Iris takes to fill approx. 50% (13GB) of free memory.

If anyone has a better solution I would like to read about it.

TheQCC · Post by **TheQCC** » Tue Nov 24, 2020 6:45 pm

Update:
Another temporary resolution that does not require a reboot.
By running PoolmonX (https://github.com/zodiacon/PoolMonX) at startup and leaving it running minimized, the non paged pool memory does not exceed 512MB and Blue Iris does not die.
The above worked on two different computers.
Also, I do not believe it is a Blue Iris problem as iSpy Agent has the same problem and PoolMonX also fixed the problem.
This leads me to believe the problem lies with Windows 10 Version 10.0.18363 Build 18363

Memory leak and 5.2.3.2 repeatedly crashing in ntdll.dll

Re: Memory leak and 5.2.3.2 repeatedly crashing in ntdll.dll

Re: Memory leak and 5.2.3.2 repeatedly crashing in ntdll.dll

Re: Memory leak and 5.2.3.2 repeatedly crashing in ntdll.dll

Re: Memory leak and 5.2.3.2 repeatedly crashing in ntdll.dll

Re: Memory leak and 5.2.3.2 repeatedly crashing in ntdll.dll

Re: Memory leak and 5.2.3.2 repeatedly crashing in ntdll.dll

Re: Memory leak and 5.2.3.2 repeatedly crashing in ntdll.dll

Re: Memory leak and 5.2.3.2 repeatedly crashing in ntdll.dll

Re: Memory leak and 5.2.3.2 repeatedly crashing in ntdll.dll

Re: Memory leak and 5.2.3.2 repeatedly crashing in ntdll.dll