New cameras, old server, lots of hangs

Post Reply
Ferguson
Posts: 71
Joined: Sat Feb 22, 2020 1:32 am

New cameras, old server, lots of hangs

Post by Ferguson »

I have a problem that might be the cameras, but it is puzzling and hope someone can give me a start.

I had BI running at my prior home with 7 cameras, working perfectly (well, except the reolink for its own reasons). These were all 4k, a combination of hikvision, lorex and one reolink.

The server is more than capable, no performance issues.

I moved. I moved the hardware as-is, except that I found a GTX970 I had not remembered and added it to the server (it had an Intel HD530 built in). Otherwise no change but deleting old cameras and adding two new ones. I also upgraded to 5.9.0.7 (from whatever it had been early in 2024 before I started packing to move, not sure, but I stay fairly current with stable).

The new cameras are Empire Tech IPC-T58IR-ZE-S3 and IPC-T54IR-ZE-S3 (so 8mpx and 4mpx but otherwise pretty much identical).

I set them up on the same switch I've used for years, and configured and connected. I set them up as 4fps. The 8mpx hardly worked at all (hanging), the 4mpx seemed to be working, so I reached out to Empire Tech (so far with generic results and waiting for next iteration).

I reset the 8mpx to factory and left it unchanged other than password and IP address. No change (this puts it at 30fps).

But today in reviewing the recordings I see that the 4mpx is not working properly either. As recorded in BI it has hangs, from a second or two to maybe 15 seconds. These hangs are actually in the recordings (I exported some and played back).

On the 8mpx one I see the same hangs in a web view of the camera directly as BI sees.

On the 4mpx one the web view is pretty clean, no hangs. At least none to the eye. I've tried two different computers, I've swapped cables and ports, the issues follow the camera. There are no errors on the switch ports. And at 4fps there's almost no traffic, it's not a network loading issue. Wifi is not involved at any point.

BI is recording directly to disk, not transcoding.

Honestly I think the cameras are sick (though Empire Tech seems well regarded and I have two with similar issues). But I'm hoping someone has suggestions what else to check. I don't even know what other information to put here, so apologies if it is long on words and short on details, but I did not want it to be too very long.

Short list of things tried:
- Reset to factory
- switched cables and ports
- web browser on two different computers (for 8mpx which fails in web browser also)
- did a packet capture on BI server for 8mpx that hangs - no visible problems like timeouts or retransmissions
- restarted BI, restarted the camera many times, did a auto-set of the video parameters in BI several times
- Monitored server for performance issues -- it's barely awake (and it usually ran 7 8mpx cameras and even then was loafing)

Any thoughts what I can be missing? Is there a quick way to roll back just to see if it was the BI version that changed something? (I think that's highly unlikely, it was a minor release).

Linwood
User avatar
Pogo
Posts: 528
Joined: Tue Jul 18, 2023 7:21 pm
Location: Reportedly in the Area

Re: New cameras, old server, lots of hangs

Post by Pogo »

More details about the current hardware would be helpful as would additional detail about the camera configurations.

Try the cameras at their default settings with VLC viewer on a different machine if possible. If that isn't possible, shut down Blue Iris and try VLC on the same machine.

Direct to disk, sub streams, get all that jazz in place for whatever CPU headroom you may be able to muster for Blue Iris.

Check Task Manager for weird activity, resource allocation and hardware performance.

DON'T use H.265 encoding in the camera configurations. Use straight up H.264 with no proprietary "Smart" or "+" crap. Ensure your streams both main and sub are correct for the cameras..., etc, etc.

It's highly unlikely both cameras are 'bad'. It's much more likely that Blue Iris isn't playing nice for whatever reason(s) these days or there is simply a configuration issue between the two that isn't matching up well..., unless there is a very weird network bottleneck chocking everything down to a crawl.

But the first thing I would personally do is pull the GTX and roll back to whatever previous version of BI was likely on the machine when you last had it running at the performance level you recall as being acceptable. The installers are archived at IPCT by date. Download what was available toward the end of your previous subscription period and go from there.
Ferguson
Posts: 71
Joined: Sat Feb 22, 2020 1:32 am

Re: New cameras, old server, lots of hangs

Post by Ferguson »

Pogo wrote: Wed May 29, 2024 10:41 pm More details about the current hardware would be helpful as would additional detail about the camera configurations.
Server is a Z170-WS i7-6700 system running W10x64, 64gb memory, everything on SSD except target of BI's video saves. There's just no sign of any resource issues there, hard faults near zero and 49% memory allocated, CPU hovers in the single digits. Except when viewing stream GPU's are near zero, viewing I might see 20% on the NVidia. But again -- this hardware ran 7 8mpx cameras without any issue before I moved it, I just don't think it is the hardware unless the addition of the GPU hurt something.

The 4mpx camera is set for H.265 (more in a bit), General, full resolution frame rate 4, quality 6, bit type VBR, max 4096, iFrame=4, with substream H.265, 704x480 D1, 4 fps, VBR, quality 4, max rate 512k, iframe 4.

All the actual image are defaults, made no changes there.

The 8mpx camera that won't even work on a web browser is factory defaults, H.265, General, Full res, 30fbs, CBR at 5120kbs, iframe=60, substream D1, 30fps, CBR at 512kbs, iFrame 60. And image parameters defaulted.

The camera configuration in BI for the 4mpx (the 8mpx doesn't get far enough to bother now) is directly from the find/inspect:

Image
Pogo wrote: Wed May 29, 2024 10:41 pm Try the cameras at their default settings with VLC viewer on a different machine if possible. If that isn't possible, shut down Blue Iris and try VLC on the same machine.
So this is interesting. While running BI and VLC at the same time, VLC has a nice smooth display, no hangs; BI is still hanging same as when it runs alone. That seems to point to a BI issue:
Pogo wrote: Wed May 29, 2024 10:41 pm DON'T use H.265 encoding in the camera configurations. Use straight up H.264 with no proprietary "Smart" or "+" crap. Ensure your streams both main and sub are correct for the cameras..., etc, etc.

It's highly unlikely both cameras are 'bad'. It's much more likely that Blue Iris isn't playing nice for whatever reason(s) these days or there is simply a configuration issue between the two that isn't matching up well..., unless there is a very weird network bottleneck chocking everything down to a crawl.
So ... yes, i was using H.265 because that was the camera default.

I set it to H.264, and in a few minuets of watching it was smooth. Perhaps more surprisingly, the the web display from the 8mpx started working. I added the 8mpx to BI and in another few minutes it seems to be working.

So... maybe these camera's implementation of H.265 is flawed (I saw that because their own web display does not work)?

An overnight run and some browsing in the morning will tell me more.

Thank you.
User avatar
Pogo
Posts: 528
Joined: Tue Jul 18, 2023 7:21 pm
Location: Reportedly in the Area

Re: New cameras, old server, lots of hangs

Post by Pogo »

Ferguson wrote: Wed May 29, 2024 11:14 pm So... maybe these camera's implementation of H.265 is flawed (I saw that because their own web display does not work)?
The problem is Blue Iris and a horrible implementation of H.265 decoding with or without QuickSync HA support..., and even being able to handle main stream 4K in H.264 at any respectable frame rates and resolution/bit rates.

And regular browsers don't support H.265. If you are referring to the web UI of the cameras not working properly with H.265, that's a different issue altogether and usually a resource or bandwidth issue somewhere exclusive to the specific connection or how the camera webUI is attempting to be processed by whatever browser is accessing the camera UI. Camera web UIs will usually adapt to whatever restrictions they see at the display end to minimize such issues, but sometimes they just complain about needing a plug-in or simply don't adapt and politely convey that you're SOL.

I now recall your rig rundown in your other thread. The resources are certainly there. There is the possibility that the GPUs are banging into each other somehow in Blue Iris. You may want to designate the 6700 (Intel) as the default in the main BI 'Cameras' settings and ensure default is designated in each camera's decode configuration. This should isolate the two GPUs for testing. The Nvidea can still be swapped in and out on a camera by camera basis.

May also want to visit your BIOS to see how video support is set up there.
Ferguson
Posts: 71
Joined: Sat Feb 22, 2020 1:32 am

Re: New cameras, old server, lots of hangs

Post by Ferguson »

Pogo wrote: Thu May 30, 2024 12:43 am There is the possibility that the GPUs are banging into each other somehow in Blue Iris. You may want to designate the 6700 (Intel) as the default in the main BI 'Cameras' settings and ensure default is designated in each camera's decode configuration. This should isolate the two GPUs for testing. The Nvidea can still be swapped in and out on a camera by camera basis.
There's a big difference in processing speed, I assumed setting the default for BI to NVidia instead of Intel would force it over to the GTX?

There's another strange issue. The actual BI program interface so far seems to be running smoothly. The UI3 web display with both cameras up (and I assume that means either it is showing substream or is transcoding down) works fine, but if I click to show one camera (either one) the display just hangs and shows HTML5 viewer stopped, orange clock face, restarts, connects again. I used that frequently before, so something has changed there.

While it is hung and trying, task manager on the BI server show CPU at 5%, GPU on the GTX at about 15%.

I tried switching both cameras back to GPU 0 (the Intel) and saw no change -- I mean literally no change, I still show the GTX as the one in use, as though BI ignored the setting. Though to be fair I have no real reason to think the GPU is the issue (and I just updated the drivers, though the new ones were only a few weeks newer than I was running).

I've got a full gig wired network connection between the BI server and my desktop where I'm running UI3. I also tried UI3 on the BI server and got the same thing.

And again... the BI program itself runs fine now that the camera is in H.264.

So at the moment, my only problem appears to be UI3 not working properly.
User avatar
Pogo
Posts: 528
Joined: Tue Jul 18, 2023 7:21 pm
Location: Reportedly in the Area

Re: New cameras, old server, lots of hangs

Post by Pogo »

Someone else will need to hop in for the UI3 stuff. I only use it infrequently.
Ferguson
Posts: 71
Joined: Sat Feb 22, 2020 1:32 am

Re: New cameras, old server, lots of hangs

Post by Ferguson »

Pogo wrote: Thu May 30, 2024 1:37 am Someone else will need to hop in for the UI3 stuff. I only use it infrequently.
Thanks. I'm delighted to have the cameras working, but I use the UI3 stuff whenever I travel, just VPN in and run it. Since (unless something changed) you can't run the BI program itself on a different device, much less an android.
User avatar
Pogo
Posts: 528
Joined: Tue Jul 18, 2023 7:21 pm
Location: Reportedly in the Area

Re: New cameras, old server, lots of hangs

Post by Pogo »

I'll be curious to know how you get it sorted out.

Good luck with it.
Ferguson
Posts: 71
Joined: Sat Feb 22, 2020 1:32 am

Re: New cameras, old server, lots of hangs

Post by Ferguson »

Overnight the H.264 change fixed both cameras with regard to hangs in the BI interface live as well as playback.

I still have the UI3 problem but perhaps should open a new thread for that, as it appears different from the H.265 issues.
Post Reply