Blue Iris

Hi there everyone

Long time user, first time poster. In the time we have used blue iris software for our resorts camera software solution it has gone mostly well with only a few hickups due to myself and the other IT learning how it works.

We have gotten most of the kinks out of the system, but one issue remains, and it has been happening for almost since as long as we have been using blueiris.

the issue is the cameras appear to lose signal:
https://imgur.com/k8vt79g

but when looking at the camera settings we can see that here is still data comming in. In the image below I highlight this. Once I notice that it says there is data comming in, I know its an easy fix...
https://imgur.com/caURkV5

When doing a netstat -na on blue iris we can see that it does not appear to have an active connection to the camera at this point
<imagine a netstat -na here with without the cameras IP anywhere in it>

But, just for mitts and wiggles, lets see if the camera is online:
https://imgur.com/QUweALb
Yup, the camera is definitely working!


So, then we click OK (as circled in red in the screenshot above) we the test bar image on the camera:
https://imgur.com/NAFQcGf


And after a few seconds of waiting..
https://imgur.com/4SsvAcN

BAM! The image comes back! Without us even doing anything, other than opening is configuration page and click 'ok'.

However, this does not work EVERY time obviously. If there was a damaged ethernet cable, or if the camera as faulty or if the network was down, or if the IP changed.... or many other reasons can cause a camera to display no signal, however, I have noticed that if a camera is showing that it is still receiving data, ten by clicking 'OK' will bring the test bar image or a few seconds, and then the feed will be restored.

Here is a screenshot of the settings pages for that camera (sorry some of the IPs and images are censored in this posting as per our company policies.
https://imgur.com/1DUnQVw
https://imgur.com/6Xki8kL


Now, this might seam like a minor deal, and i would be, if the camera signal went down once and a while every few days..

but this issue happens to multiple of our cameras every single day. When we come in to work in the morning, the first thing we do is load up BI and 'refresh' each camera that had gone offline overnight, and we usually do it again at the end of the day, and sometimes in the middle of the day too.

It has come to the point that I might write a macro that will: check if there appears to be data coming into the cam, and if there is, click ok and check again in 3 minutes. Do this for all no signal cameras.

This macro would save us a lot of time, but if we could actually FIX the cause of the issue that would be ideal.

Im not sure if it has to do with our network, or if it might be a bug in the code of some kind. I notice that it does not ever happen to our single USB camera that we have connected directly to our BI server. All of the cameras are PoE cameras that this occurs to (we dont use any analog cams or any other feeds types.

Over the years of using BI we have kept it updated, and hardware has changed. we switched from using PoE injectors in some places to PoE switches, and occasionally also the other way around, and as far as I can tell, it is happening to both.

Blue iris is running as a service on a dedicated server. It is windows 10 pro x64 a rack mount supermicro x9dax with dual xeon e5 sandy bridge cpus at 3.1GHz and 128GB of dual channel DDR3@657MHz and a nvidia quadro k1200 w/4gb of ram for good measure, even tho we record everything straight to disk and do not re-endcode any of our feeds. The Main OS drive is on a kingston SSD, and ever since blue iris said that our database we getting big, we installed a high speed 480GB intel pci-e SSD for blue iris temp files and database to be stored on. All other storage drives are 18TB seagate iron wolf pro, or 16gb seagate purple security drives.

AV scans are turned off, and the firewall is turned off completely (i know, not the greatest security settings)

while no screenshot of this is shown, we have our power settings optimized. No other programs are running on this server except for docker and the LPR software, but those are currently not even running at the moment as we are tryingn to determine if we need them. Either way, the system performs well, even with the extra AI and docker running.

On average, when all the cameras are running the average CPU is around 26% and around 11GB of ram is being used. With the AI and docker it can go up to 70% cpu and something like 30GB of ram or something, but its not running and we had these issues for a much longer time than since we we installed that software a few weeks ago. We also always do straight to disk recordings in the default blue iris standard.

We have a mixture of cameras, most of them are axis, but some are hikvision, some are d-link, some are trendnet, and a few are reolink and a couple of them are random cheapy ones from amazon. We have 35 cameras, about half of them are outside but none of this appears to matter. These signal losses happen to the ones on the inside and outside, the axis cameras and the 30 dollar poe outdoor camera from amazon that appears to have survived just fine over 1.5 cold winters thus far btw. So its not their location or type of camera either (except for the usb cam, that cam stays on and does not time out)

If you like, you can view our public cam here: (link removed, form detected this as spam) or directly from: https://mysterypeak.mtseymour.ca/jpg/image.jpg
and our snow level cam:(link also removed) .. both of these are axis cameras. Funny enough, these two go down for that reason rarer than most of the other cameras, but i have seen both of those cameras also go no signal and a 'refresh'' of the feed was required to get it going again.

Thank you for any suggestions or help on this. The speccy specs are attached to this posting if you want to know more about our set up (had to zip it up as .txt files are not aloud :/ )

hoping to hear some suggestions. anything that might help is appreciated!

A lot to unpack. And a lot to go sideways with a system that involved.

The obvious things to check are what everything has in common aside from Blue Iris.

I always say start simple. It will get complicated soon enough all by itself. LOL

You've determined that signal loss doesn't seem to actually be occurring on the network, but is somehow being affected to the extent that Blue Iris is confused about what to do with it -- if streams are indeed getting to it. Without going back through your entire post for a reference to how you are directly accessing the cameras to confirm their online state, you may want to do a simple test at the same switch as the Blue Iris machine is using and try some VLC RTSP sessions of a 'no signal' camera from a laptop to make sure there's an active stream there on port 554 vs. a port 80 web interface session. If there's a clean RTSP stream with no errors, then Blue Iris should be receiving the same input and that would at least settle that.

Another very simple test for possible network issues coming into play is pick a trouble camera and just ping the snot out of it for a few hours with large packets, say 2048 or 4096 bytes. They should still be in the ~10ms range if things are really clean. There should be zero loss even if there is some lag or congestion. Again, plug into the same switch as Blue Iris with a laptop and go for it.

Do as many general network performance tests as may be applicable to how a clean and efficient IP video network should perform with or without Blue Iris or any other VMS platform. If everything comes up roses there, on to the rig, its components and their configuration compatibility, and any overkill features and functions not necessary for basic Blue Iris functionality including CPAI and any multi-functional GPU configurations that may just be banging into each other. Break it down to only the requirements needed by Blue Iris to efficiently and effectively run a handful of cameras as would normally be expected. Re-introduce the other stuff once a trouble free solid foundation has been established. You'll likely find the problem source sooner than later.

Review configurations. Certain little things matter, others don't. Again, keep it as simple as possible for the troubleshooting. Ensure the correct CPU/GPU references are in place, properly recognized and functioning appropriately. Make sure camera webUI based configuration settings match the corresponding Blue Iris configurations.

All basic stuff. Can take time with lots of cameras, though this particular issue affecting all cameras is more than likely a server issue. Still, start with one camera at a time. If you find a fix for one, you've probably found it for others as well.

A couple things out of the gate. I don't see sub streams in the configuration examples. I see limit decoding checked. I see 'default' for processor type. You list any number of possibilities there in your system specs. Who's on first?

Another item I noticed is a defrag schedule. Not something you want to do with the Blue Iris database and storage structure. Hopefully it is limited to non-Blue Iris storage devices but could really disrupt things if it's trying to trim and defrag BI drives. And speaking of storage, depending on how things are set up management-wise, non-alignment with how BI likes to see things and dynamically manage them usually creates issues and can waste resources as well.

A.C. power stability. All good? Maybe some fluctuation up there on the hill? POE all good? No excessive wiring lengths and solid copper ethernet cabling used everywhere? No CCA aluminum bargain junk CatX wiring anywhere mission critical?

Lastly, time sync. I noticed Windows Time running. Are we all on the same page there? Time sync is very important.

Simplify what you're expecting from the rig. Break it down. Get rid of anything that Blue Iris doesn't need to effectively run your handful of cameras reliably in a basic surveillance configuration 24x7x365..., or at least maybe for a whole day for starters?

My drastic suggestion would be to get Blue Iris off the existing box and all the other clutter and into a basic Win10 Pro Intel iCore rig with a modest i7 8500, 32GB RAM, SSD of your choice and a big-ass spinner for initial storage with a couple more on the side if you need the archives. It would be much happier -- and so would you!

Anyway, just my $.02 and some things to chew on.

Just a brief follow up after a re-read of both posts above...

I neglected to suggest the obvious which is to check the log and activity entries on the Blue Iris Status page for clues. (If you haven't done so already, you can check the 'Save to file' box at the bottom left to create monthly text files for easier reference in the future.)

You may also want to inspect Task Manager and any other relevant Windows log areas of weirdness going on when these events occur.

I seem to recall Send RTSP keep alives as being an option that you could try disabling.

There’s also an option to treat b/w video as loss of signal so you might want to see if disabling that helps.

I'm getting this too. I cannot find the issue. All of my IP assignments are static and non-conflicting. I have reset and default camera setting and software several times. I keep getting lost connection time out for about a minute randomly throughout the day. Then the cams come back up and are fine for a while.

I'm also getting a random "login timed out" errors randomly too. Those also just seem to fix themselves.

...

Attached

It would help if you could condense the columns to expose all of them, particularly the No Signal column. If the cameras are dropping but the numbers don't show it, that points to other possibilities aside from lack of actual signal for the drops.

There are a lot of mega pixels going on there. What kind of horsepower is running everything and what brand of cameras are involved?

The sub-streams are curious as are the I-frames.

And what is your method of viewing and managing the server?

The main possibility that comes to mind is the additional H.265 processing requirements could be taking the CPU/GPU to the limit and simply not getting everything done on a consistent basis..., especially if also being used for the sub-streams. The cameras are still online, just not getting their streams adequately processed. The big test there is to simply switch to H.264 in the camera software (if the option exists) and see if that helps things stabilize. If so, revisiting your storage configuration requirements (and options) may be in order to give the processing resources enough breathing room to keep things running smoothly -- and looking better than H.265 in the opinion of many as well.

If the cameras happen to be Reolink, that's a whole other bag o' snakes.

Not to hijack the thread, I'm having a similar issue with BI. My static cameras keep dropping off and on. Just my static ones. Not the pan and tilt ones. They are stable except the one that is going back on an RMA. Sometimes they give the "No Signal" with the brown frame. Those will generally come back up after a random time. Other times they just stop with the image still showing. I have to look and see if the clock is working. These are all on Wi-Fi. They were working fine for months, then this started about 2 weeks ago. I'll try any ideas.

Captain Quack.

Are you able to temporarily run an ethernet cable to any of them to rule out wifi as the problem ?