Blue Iris

Re: 5.5.5.11 DeepStack: timeout

2022-03-08T23:38:22+00:00

UPDATED:
I just remembered this:

reverted to 5.5.4.5 and all such errors went away, implying there is a regression of some sort in the integration between 5.5.5.11 and DeepStack.

There have been reports of erratic behavior with every update after 5.5.4.5 and I am not going to update until the next stable release. Unless there's a compelling reason, I advise folks do the same. In your case, I'd roll back to the previous version and prove that it goes away. Then submit a customer support request with the same subject you did here, let them know that it works in 5.5.4.5 but not in 5.5.5.11. Then include a link to your forum topic for background.

That's the best way to make sure it gets passed to engineering.

PSA:
When writing for support, you must include basic software and license details. For your convenience you will ﬁnd an Email support button on this page. This button will copy important information to the Windows clipboard. In order to make use of this information, you must them open an email to send to support@blueirissoftware.com. Please make the subject descriptive in order to prevent our mail server from combing it with others of the same subject. In the BODY of the message (not the subject), use the Windows PASTE command (control-V) to insert your support information at the beginning of your message.

Statistics: Posted by YrbkMgr — Tue Mar 08, 2022 11:38 pm

Re: 5.5.5.11 DeepStack: timeout

2022-03-08T20:30:49+00:00

For whatever it's worth, I'm still using DeepStack: Version 2021.02.1. I have no issues worth risking stability so until I do, "if it ain't broke".

Statistics: Posted by YrbkMgr — Tue Mar 08, 2022 8:30 pm

Re: 5.5.5.11 DeepStack: timeout

2022-03-08T04:55:58+00:00

Bummer. I've had another 20s timeout after which BlueIris restarted DeepStack. I'm worried this may truly be DeepStack instability which I found mentioned in other places, and for which the "Gotchas" articles says, "Unfortunately we do not know the root cause."

Statistics: Posted by carteriii — Tue Mar 08, 2022 4:55 am

Re: 5.5.5.11 DeepStack: timeout

2022-03-08T04:42:06+00:00

THANK YOU! I swear I searched this forum (& others) but I clearly hadn't found that "Gotchas" article right at the top of the DeepStack subforum which mentioned the timeouts, which is exactly what I have been seeing. Silly me for only using search and not doing some top-down browsing!

I also now realize that even after installing the 2022.01.01, I still had the older "server.exe" so BlueIris was in fact running the older version, despite my attempt to use the latest. I've now completely removed the old DeepStack install (including server.exe) and re-installed from scratch the 2022 version (copying deepstack.exe to server.exe) so we'll see what happens next. In only the past few minutes, no timeouts so far.

I do wish there was a way to enable even more detailed logging that included the actual DeepStack API calls to see exactly which were failing with an easy way to re-play those calls. I suspect there might be a more verbose debugging option in DeepStack but I haven't put the time into finding that.

Thank you again, @YrbkMgr!

Statistics: Posted by carteriii — Tue Mar 08, 2022 4:42 am

Re: 5.5.5.11 DeepStack: timeout

2022-03-07T23:26:13+00:00

On logs... I've spent a lot of time with them and am creating a BI Log File Analyzer (decombobulator) to make logs more useful. That said, know this: it ain't for the faint of heart. I recommend you spend some time in the Self Help section of the forum as there is an article on Status log errors

Statistics: Posted by YrbkMgr — Mon Mar 07, 2022 11:26 pm

Re: 5.5.5.11 DeepStack: timeout

2022-03-07T17:22:41+00:00

Some google searching turned up at least one thread suggesting it could be the Windows (CPU?) version of DeepStack that is unstable:

https://www.reddit.com/r/BlueIris/comme ... ck_issues/

This implies the Windows native version is deprecated (in favor of Docker), even though a 2022 build was released, implying that the statement was or is not entirely true:

https://github.com/VorlonCD/bi-aidetection/issues/19

Statistics: Posted by carteriii — Mon Mar 07, 2022 5:22 pm

Re: 5.5.5.11 DeepStack: timeout

2022-03-07T17:00:39+00:00

Things are working sometimes, but I still have big timeouts, sometimes followed by the message that, "DeepStack has been restarted". If I wasn't paying close attention and looking at the BlueIris logs, I might not even know, instead having just missed what should have been an alert. I'm pasting a screenshot of a small section of my log from this morning.

You'll note a few instances of a timeout that include the ms duration (with my 20-second timeout), and in a few places there is a timeout without any time given. It seems that timeouts initiated from an ONVIF/EXTERNAL event show the time (e.g. 20016ms), but timeouts initiated by MOTION_A do not give the time, presumably implying the logging is just a bit inconsistent.

Is there a more detailed log that I can get from BlueIris, showing the specific commands & payload? I'm curious to see if there is a pattern of what generates a timeout. When BlueIris starts DeepStack, is there any way to see more detailed logs on the DeepStack side of things, to see if the call is even making it to DeepStack?

Would those of you reading this also please do a check of your logs to see if the timeout or restarting of DeepStack happens for you? Just use the filter and look for "timeout" or "restart".

One other question about the logs. What is the buffer size or duration shown in the UI, which in turn determines the count shown in the 2nd column? For example, in my screenshot below the count of restarts is 342 but the number of times it appears in the text log is far greater. So if/when any of you check, just realize the log in the UI isn't showing everything you might have (if you are saving the text file of the log).

Statistics: Posted by carteriii — Mon Mar 07, 2022 5:00 pm

Re: 5.5.5.11 DeepStack: timeout

2022-03-06T18:38:14+00:00

Will do (going back to 1 instance). If it matters, I have 8 cores on this Radeon 5900 and have been running with more than one instance for a while and thought it was ok until this recent issue with 100% timeouts. I have also explicitly tested the ports (e.g. 82 & 83) to make sure they are responding. Is there a known issue with more than 1 instance, or just a matter of caution & testing to see if that might be part of the problem?

@MikeLud, I also want to offer a big public thank you for all you've done with the combined model!

Statistics: Posted by carteriii — Sun Mar 06, 2022 6:38 pm

Re: 5.5.5.11 DeepStack: timeout

2022-03-06T16:59:42+00:00

Try only using one instance.

Agreed.

Statistics: Posted by YrbkMgr — Sun Mar 06, 2022 4:59 pm

Re: 5.5.5.11 DeepStack: timeout

2022-03-06T16:58:37+00:00

Cool. Thanks for the follow up. I'm still using v5.5.4.5 - updates have been buggy so unless there's a compelling reason to update, Ima hang back on that.

Statistics: Posted by YrbkMgr — Sun Mar 06, 2022 4:58 pm

Re: 5.5.5.11 DeepStack: timeout

2022-03-06T16:56:09+00:00

Try only using one instance.

Statistics: Posted by MikeLud — Sun Mar 06, 2022 4:56 pm

Re: 5.5.5.11 DeepStack: timeout

2022-03-06T16:34:51+00:00

DeepStack settings listed below. I have been able to stop & start deepstack, and I have been able to see the default page with no problem. Last night I became convinced that the timeout was not due to DeepStack taking too long, but because it wasn't getting called at all. As I mentioned, I could see the status page just fine, so I thought perhaps that the BlueIris update (& rollback and then update again) combined with possibly a Windows 10 update that I also ran could have broken the Windows firewall settings (I never touched DeepStack). I finally tried re-installing DeepStack, and only because I saw a newer version available (2022.01.01) vs. what I had previously installed (2021-09.01), I went with the newer version. The installation did prompt me with a Windows alert to give permission (which may or may not be relevant) but after that, DeepStack appears to be working again.

I realize (& feel bad) that I claimed the DeepStack version was likely the issue, forgetting that I also ran Windows updates that could also have been a contributing factor.

For what it's worth, I am running the combined.pt and dark.pt models (mentioned in other threads here and on ipcamtalk.com), but until the updates I ran this weekend, everything has been running fine with no timeouts. I'll be watching closely today since there wasn't much activity last night.

Statistics: Posted by carteriii — Sun Mar 06, 2022 4:34 pm

Re: 5.5.5.11 DeepStack: timeout

2022-03-06T05:01:40+00:00

first things first. Gear icon -> AI -> DeepStack section. Post that dialog. If you see "Stop Now" button, click test in browser. Can you start it from here? Let us know how that goes.

Statistics: Posted by YrbkMgr — Sun Mar 06, 2022 5:01 am

Re: 5.5.5.11 DeepStack: timeout

2022-03-06T04:36:29+00:00

Can you browse to the DS web server instance (default is http://127.0.0.1:82 ) ?

If not, open task manager > details > right click the labels at the top (eg Name PID Status CPU etc) > Select Columns > near the bottom select Command Line > OK

For the "server.exe" process, you should see something like C:\DeepStack\server\server.exe -VISION-SCENE=False -VISION-DETECTION=True -ADMIN-KEY= -API-KEY= -PORT=82

If that's all there and you still can't browse to it, try changing the port number or updating DeepStack - a new version was released a couple of months ago.

Statistics: Posted by pqRec — Sun Mar 06, 2022 4:36 am

Re: 5.5.5.11 DeepStack: timeout

2022-03-06T02:28:38+00:00

Update: Now I'm getting DeepStack timeout messages even with 5.5.4.5, so my original assertion that 5.5.5.11 had a problem must not be correct. Something else has changed for me after the upgrade & reverting back. I'm running the CPU version of DeepStack and don't see my CPU pegging, and I haven't touched the models in a long time (I am running the dark.pt and license-plate.pt models).

I am seeing a lot of "DeepStack has been restarted" messages after a timeout. Any suggestions on how to get more detailed logging from DeepStack to know what's going on?

DeepStack: Alert cancelled [DeepStack: timeout] 15011ms
DeepStack has been restarted

Statistics: Posted by carteriii — Sun Mar 06, 2022 2:28 am