The DeepStack integration has been widely popular. The value of AI in surveillance is it improves accuracy to detect objects and reduces false alerts. AI will draw attention to activity that matters by identifying objects that could be missed by humans because of lack of attention or hard to see at night. AI also suppresses events that do not matter such as moving shadows from the wind, changes in sunlight, flies, birds, squirrels, rain etc. AI is becoming a very effective tool for outdoor cameras.

The above image shows the user experience across all the BI modalities: console, web interface and mobile interface. Consistent across all modalities are annotations on the alert image identifying objects found and icons in the alert list to easily identify alerts / cancelled alerts.

Annotation colors

• Orange: High level of confidence (>67%).
• Yellow: Confidence level <67%, but meets your threshold, e.g. 50%.
• Blue: Static object/Occupied state. See above for details.
• Red: Cancelled alert based on "To cancel" field in Camera settings.
• Green: Objects of no interest.
  For example a boat appeared in the scene but boat is not listed as an object of interest.

Alerts list Icons:

Static object

A static object is an object that is NOT moving. The most common example is not to keep triggering on a mailbox that the AI keeps misinterpreting as a person. BI comes with advanced alerting by overlapping motion with objects before sending an alert. This reduces the majority of false alerts due to static objects.

Static objects will only have a blue annotation. Occupied state will include the time symbol.


Occupied state
There still exists some stubborn scenes where Occupied State provides even more value. For example, a parked car in the driveway underneath a tree for shade could be challenging. The shadows from the tree could trigger motion and the motion boxes could very well overlap with the car causing alerts. Occupied state should prevent such alerts based on logic described below.

Furthermore a parked car is NOT a permanently static object like a mailbox. When the car leaves and comes back, BI should alert on the departure/return. The occupied state algorithm is trying to account for all these cases.

Occupied state has the timer icon AND uses the blue annotation.

Logic behind Occupied State
Same object type: +/- 10% confidence level
Same location: +/- 5%
Same object color (removed because many cameras go black&white, i.e. IR, at night. v5.5.1.17)

Occupied state list: BI maintains two lists during an alert.

• Current alert list: objects identified in current alert
• Static object list: A list of objects that it perceives to be static, e.g. a parked car, based off of past alerts.

Identifying static objects
When an alert occurs, each object(s) in the current alert list is first compared against all the objects in the static object list. If an object in the current alert matches any object in the static object list , then the object is considered static, e.g. parked car, and BI will NOT fire an alert based on this object. The static object icon will be displayed if all objects are determined to be static.

Identifying new objects
If an object on the current alert list does NOT match any object in the static object list an alert is fired. The new object will be placed in the static object list. This is how the software distinguishes between parked cars and cars that drove by. This is also how the static object list gets updated.

Removing static objects
Static objects continue to remain in the static object list if the object is matched. With every match, the timestamp on the object is updated.

If an alert arrives and the static object list has objects that were NOT matched by any of the current alert objects AND those static objects are older than 2 min, those objects are purged from the static object list. We know AI is not perfect which is why we created the 2 min window. A static object detected in one frame may not be detected in another frame due to lighting changes etc. In order to remove these inaccuracies, we assume as the camera alerts, at least one frame will identify the static object correctly within 2 min or earlier. FYI, engineering keeps tweaking the window. It started at 30s and moved to 2 min based on false alarms.



Static Objects is a setting that can be turned off if necessary (you want more sensitivity). This may be a good idea in high traffic areas like a camera facing a door or a car entry gate.

Camera settings -> Trigger tab -> Artificial Intelligence (default settings)


Click on the Help button for details on each setting. Below is some insight based on tickets.

Custom modelsList of Objects Available for detection

Motion triggers vs Alerts

With Artificial Intelligence there is now a wider gap between Motion Triggers, e.g. the camera triggered because a deer walked across the lawn, and an Alert, i.e. the AI correctly did not recognize the deer as a person, and therefore cancelled the alert.

The AI setting "Hide cancelled alerts on timeline and 'all alerts'" is on by default, i.e. most users after setting up their Motion and AI, do not want to be bothered by the "nothing found" alerts, because that is why they installed AI in the first place.

However, I do see users in the country side that like to record wildlife such as bears, coyotes, deer etc when they approach their property. BI AI will cancel the Alert, however motion triggers are still always recorded. This implementation was chosen just in case the AI missed something, you still had the motion trigger recording. It turns out, this implementation provides added benefit for the users in the country side that still like to capture recordings of wildlife. The "Hide cancelled alerts on timeline and 'all alerts'" can be unselected if you want "nothing found" alerts still listed in the Alerts folder. Furthermore, cancelled alerts are always listed in the cancelled alerts folder regardless of the "Hide cancelled alerts on timeline and 'all alerts'" setting.

Intel from a user. 20220-Jan-28 The DeepStack website needs some cleanup. There are pages on the site that point to a legacy installer that does not work! Stick to the URL above and you will be fine. In particular, the incorrect installer name is DeepStack.Installer.exe. My hunch is this version was created around 2019 before the company decided to open source their software. Another clue you have the wrong version is when you bring up the DeepStack splash page, Activation Key information is part of the page.



If you choose the wrong version, you will find BI cannot start DeepStack automatically. It cannot communicate with the server either with below errors in the Status -> Log.


As a final check, below is a screenshot of my install directory. I installed the Windows GPU version so your directory may be different based on the version installed. I have seen tickets where the install directory did not even have a README.md file which made me suspicious as to whether the installation was valid.

deepstack --VISION-DETECTION True --PORT 82

docker run -d \        --name=deepstack \        -p 80:5000 \        -v /opt/deepstack-storage:/datastore:rw \        -e "VISION-DETECTION=True" \        -e "VISION-FACE=True" \        deepquestai/deepstack

-VISION-SCENE=False -VISION-DETECTION=True -VISION-FACE=True -ADMIN-KEY= -API-KEY= -PORT=82

BI leverages AI with your motion settings in order to provide very intelligent alerts. This is why a camera facing a parking lot will alert on a moving car but not alert on a parked car. This article is a deep dive into the DeepStack Analysis functionality.

The webinar associated with this article is The DeepStack Analysis Feature. The webinar provides a demo of the BI software while going through the article content.

Playback window article/webinar
This article deep dives into the playback window. The playback window has powerful functionality to troubleshoot motion and DeepStack settings. Setting the motion sensor correctly is a big if not bigger part in getting smart alerts from your cameras.

Clone the camera

Leverage BI functionality as much as possible to make your life easier. Fine tuning your Motion and DeepStack settings to deliver smarter alerts is as much an art as a science. Therefore, use camera clones to compare before and after results so you know for yourself whether you made alerts better or introduced more issues.

Cloning cameras is easy in BI. When adding a camera, select Copy from another camera.


Unselect Clone master when creating the duplicate. Camera settings -> General tab.



DeepStack Settings


Instead of walking through all the settings (see DeepStack article), this article highlights the key settings that may need to be changed for Fine tuning. The other settings are either on/off by default or changed based on user preference. They should have no bearing on the Fine Tuning process.

• Save DeepStack analysis details: This is THE KEY SETTING. BI now makes understanding what is happening in the software so easy.
  Easy to check if DeepStack analysis is active. Simply go to your Alerts folder and see if *.dat files are starting to populate after camera alerts.
  Definitely unselect after fine tuning is completed. This feature takes CPU resources and storage resources.
• Hide cancelled alerts on timeline and 'all alerts': I prefer this unselected while fine tuning because I like to leverage the Alerts list to see all alerts that are processed in BI.
  If this feature were active, the "nothing found" alerts would be listed in the "Cancelled alerts" folder.
  Many users select this feature once fine tuning is completed.
  
  

  

• Use mainstream if available: Unselected during fine tuning.
  This setting is important because the motion sensor is applied to the sub stream if you have connected two streams to your camera.
  Motion and object overlap analysis is a huge value add that makes BI AI alerts accurate. So I want DeepStack to analyze the exact same frame that was used to detect motion so the overlap analysis is as accurate as possible.
  User preference whether to leave unselected after fine tuning. I leave it unselected. However, if you have really good synchronization between your main and sub stream and you feel you are getting more accurate AI object detection from the high res main frame, then selecting the mainstream may make sense. Keep in mind, analyzing high resolution images also takes more CPU/GPU load.
• Other optional settings I find useful at times
  • Recording tab: Set recording to When triggered. Uncheck Combine or cut video each. This way a new BVR is created for each trigger. Because you have a BVR for each trigger event, it's easy to replay a missed alert/false alert and tweak your motion settings and observe if the tweaks improve your alerts. If you cannot figure out an issue, it is also easy to send the short bvr of the motion trigger to support for review!
  • Trigger tab: Leave Motion overlays off. Camera settings -> Trigger tab -> Motion Sensor. Highlight: Do not highlight motion.
    The overlays may interfere with DeepStack resulting in more missed objects. Best to turn overlays off when using DeepStack.
    Engineering will be addressing this issue.
    FYI, with D2D recording, turning motion overlays on/off is less important since they are saved as meta data regardless. Therefore, even if you do not see highlights in Live view, you can still do so during playback.
    If you selected Re-encode when recording, then unfortunately, no motion overlays are available if they are turned off for Live View.
  • Trigger tab: Set Add to alerts list = Hi-res JPEG files.
    With the *.dat files now created with Save DeepStack analysis details option, having the JPEG images makes finding the corresponding *.dat file easy, especially when you are stuck and need to consult with support.
    With the JPEG images, when you right click on an alert -> Open containing folder, BI will find the JPEG for you.
    
    

    

    
    After doing so, the preceding *.dat file in Window Explorer is the *.dat file associated with the selected Alert. Makes identifying *.dat files for alerts of interest easier in Windows Explorer.
    
    

    

Motion sensor
This section explains default settings that work well with DeepStack.

Understanding why DeepStack did not alert has just become a lot easier with the "Save DeepStack analysis" feature. Now BI can show you exactly which images(i.e. frames/samples) were processed by DeepStack when a motion trigger fired so you can understand why an alert was or was not sent.

In order to get the feature to work:

• First check "Save DeepStack analysis" in Camera settings -> Trigger tab -> Artificial Intelligence. This setting will start creating *.dat files in the Alerts folder pertaining to the meta data for each motion trigger.
• Open the Status -> DeepStack window.
• Double click any motion trigger in the Alerts List and the DeepStack Status window will populate with the DeepStack meta data making it much easier to understand what is going on.

This example highlights an alert that was cancelled by DeepStack.

Now you see exactly what BI does when making alert decisions based on your settings. Tweaking settings based on missed alerts has become much easier.

The logs also provide data consistent with the DeepStack analysis data.


From the logs, motion was detected at 11:00:51.907. DeepStack cancelled the alert at 11:01:01.504 roughly 10s later.

Status button -> DeepStack tab

DeepStack analysis is great because it will tell you why alerts were missed. At the end of the day, you basically have 3 levers to play with:

• The sub stream is where motion detection occurs (if you setup dual streams). Running DeepStack analysis on the sub stream may provide better accuracy in regards to object / motion overlap.
  Running DeepStack on the sub stream will provide faster inferences (less CPU resources). If you have a CPU based DeepStack deployment, this may be of value since BI and DeepStack are both competing for the CPU.
• Alternatively run more samples per trigger. My break time is 10s, so I sample every second to improve my chances of finding objects.
  

  

• A final trick is to play with motion zones so the object is in an ideal position for accurate DeepStack analysis. (Pro Tip 1 below)
  
  Motion fine tuning and leveraging the playback window is also good to know when tweaking alerts.

Intel from a user. 20220-Jan-28

A new version of DeepStack was just released. There is an issue with the way Blue Iris starts up the new version of DeepStack. The issue is DS version 2021.09.01 had two of the same files just named differently in the folder "C:\DeepStack\server". One was named deepstack.exe and the other was named server.exe. Blue Iris is trying to start up server.exe.

DeepStack version 2022.01.1 only has one file in folder "C:\DeepStack\server" named deepstack.exe. I did a temporary fix by copying deepstack.exe in the same folder with the file name server.exe.

Link: https://github.com/johnolafenwa/DeepSta ... /2022.01.1

Engineering states 5.5.5.1 will have the fix.

Issue: DeepStack keeps timing out. Customer comments
I have been successfully using Deep Stack for some time. However on <date> it ceased working. The system would no longer flag clips for recognized items.

It appears to identify objects but times out.


Fix:
Unfortunately we do not know the root cause. Users would re-install DeepStack. User comments below.

See Microsoft Process Explorer section to download/install Process Explorer.

Process Explorer is a good way to see if DeepStack is running correctly on your machine.
Below is an example from an actual ticket.

Issue: For some reason, DeepStack stopped working with one camera.

I have the latest software and am using the 1st CPU version of DeepStack. Everything has been working but recently one of the two Front Cameras which are in the same Group, stopped working using DeepStack. A car just went by so I am showing you that Side-Front shows the Snapshot (size) and then jpg Group and car:63% but the camera below called Front shows none of that. It was working until the last few updates. Does any of this ring a bell on what caused my main Hikvision to stop working with DeepStack or with the other group.

Update: As I was typing this, I decided to hit the Start Now of deepstack on the AI tab and it started working. Strange that one camera would work with the DeepStack settings and the other wouldn’t until I hit Start Now.

Support Response:
Check the logs for DeepStack errors.
Run Windows Process Explorer to make sure you have just have the one set of DS executables showing beneath BlueIris.exe.
IF others exist, they may be abandoned by a crash, causing issues with CPU and RAM over time.

Process Explorer Screenshot


Resolution:
When Blue Iris starts/stops DS the conhost, server and python processes will be listed as children of BlueIris.exe.
It's safe to kill those other ones (or restart Windows).

Issue: Defined an area of interest (Zone B).


Told BI to only trigger based on Zone B.


Received an Alert due to motion outside Zone B.


The problem is DeepStack will observe the entire visible image.

Fix:
Two options.

1. Add more intelligence to Object Crossing logic.


By improving the logic, BI object tracking will only trigger if there is movement from A-B or vice versa.
The camera would never trigger if there was motion just in B, for example a piece of paper was blowing in the wind.
Thus BI would never have triggered and DeepStack would never have the opportunity to analyze the frame where the car was driving by on the road.

2. Obscure the road completely by redefining Zone A.
By obscuring the road, it is completely out of the picture. DeepStack will never analyze the street.
Probably leads to less CPU/GPU resources for DeepStack as well.

For some reason, certain deployments (CPU) result in DeepStack filling up the temp directory over a few days as the server is running.
c:\…AppData\Local\Temp\DeepStack



We still do not know the root cause. Please share with Support if you know the issue.
Below is a work around shared by one user.

Unfortunately what I have isn’t so much a fix as it is a workaround. It’s just using Windows Task Scheduler to call a batch file that runs ‘del C:\DeepStack\redis\temp-*.rdb’ every hour.

See DeepStack: Server Error 100 section in Status Log Errors article for details.

DeepStack is on (or believed to be on). However, no AI alerts appear in clip list. And no errors in the Status->Log!
We are getting reports that users have DeepStack running based on the DeepStack splash page popping up. However, NO Deepstack annotations appear when they should. It's as if DeepStack is not turned on. Clips list looks like below. Only motion annotations


Once DeepStack is activated, you should receive AI annotations. Even with no objects, the "nothing found" AI annotation appears!


If DeepStack were turned on, the clip list would look like below when objects were found.



First confirm the BI server and the DeepStack server are communicating. Global settings -> AI tab.
Also confirm you have DeepStack turned ON for the camera you are debugging! Camera settings -> Trigger tab -> Artificial Intelligence. Remember, if you have multiple Profiles, DeepStack needs to be set correctly (i.e. on or off) for each Profile.

Big gotchas:

• If you uncheck Object detection from the Advanced section of Camera settings -> Trigger tab -> Motion sensor Configure, you have unbeknownst to yourself, turned off DeepStack because object detection is needed for DeepStack to work.

  

• If all the trigger sources are inactive in the Alerts tab, you have unbeknownst to yourself, turned off DeepStack.
  If you want to turn off all you alerts, best do so by deleting all your entries in the action map or disabling all your entries in the action map.
  The default setting is to have all your trigger sources active. At a minimum keep the Motion Zones trigger sources enabled so DeepStack is not affected.

  

Assuming your settings are correct, run DeepStack manually to see if requests are being sent to DeepStack from BI.

• Uncheck Auto start/stop in Global settings -> AI.
• Restart Windows to make sure no DeepStack processes are still running. DeepStack could be unstable if trying to do so manually with all the additional processes that DeepStack spawns (server.exe, python.exe, redis-server.exe).
  
  

  

• Start DeepStack manually from the command line. "deepstack --VISION-DETECTION True --PORT 82". Port 82 is used by default but if you chose a different port (Global settings -> AI tab), specify the same number in command line.

From the console, every time you trigger the camera or camera triggers on its own, observe whether a new request appears in the command window.

Deepstack is running but sometimes BI responds server not running

The below logs show a successful Deepstack response at 1:09:15. Then at 1:09:46 BI states Deepstack is unreachable, however user confirmed Deepstack is still running.
The fix:
First confirm all Software Security Exemptions (e.g. Windows Defender) have been applied. The Windows Tuning Article provides details.
https://blueirissoftware.com/forum/view ... =12&t=2080

The unreachable message occurs if DS does not respond on the IP:port that you specify. If this is not due to issues with security software, perhaps DS is having other issues. I do see an occasional DS state where it is not responding.

BI is now detecting Deepstack not running or unreachable states and restarts DeepStack behind the scenes (assuming BI is set to start/stop DeepStack). This should fix the issue without any user interaction needed.

If BI cannot communicate with Deepstack, BI will let you know.
Assuming you have Auto start/stop with BI enabled, simple fix is stop Deepstack if it's running and restart BI.

AI / Deepstack is very computationally heavy. If your CPU usage jumps and you don't have a GPU the best you can do is throttle the calls to Deepstack.

• Turn on Deepstack only for outdoor cameras. BI motion settings are usually fine for indoor cameras.
• Reduce number of images sampled per alert.
• Tighten your BI motion sensor settings so cameras do not trigger an alert so frequently.

Shout out to our users for sharing issues and solutions!

For some reason, the BlueIris software is ignoring what the Deepstack server says in terms of confidence and the label. This seems to happen during the second, third or sometimes fourth image that Deepstack analyzes. I have confirmed with a packet capture that despite these additional images being sent back to the BlueIris software with a successful response, BlueIris ignores it and goes to cancel the alert. I confirm these images meet the criteria of the minimum confidence percentage and "to confirm". I would like to also mention that if DeepStack responds back with a prediction that is within the criteria on the first image, it works OK.

Fix: I just needed to change the keyframe interval to be the same value as the FPS so it is equal to 1.00 (as displayed from Blue Iris). The detection is working much better now. See Camera stream optimization article for details.

Alternative fix: If for some reason you cannot adjust the key frame interval on the camera, identify the current key frame interval (Status -> Cameras tab). If the value is 0.25, i.e. 1/4, i.e. 1 key frame every 4s, then set the pre-trigger buffer (Camera settings -> Record tab) greater than or equal to 4s. This will guarantee BI at least one key frame to process.

This could imply the Deepstack server is returning server errors and BI is just reporting back "nothing found".
BI will eventually report the Deepstack errors in the Status->log to make users aware of the issue easier.

Until done so, one user creatively used Postman to identify the issue. If you are a developer, you are probably familiar with Postman. It's a very popular tool that helps test and debug APIs. If new to Postman, lots of content on the internet how to use it. By using Postman, the user discovered object detection requests to Postman were returning 500 error codes. 500 errors mean the server had a problem processing the request.

FYI, 400 errors mean the request to Deepstack is not correct. For those new to Postman, a 400 error means the request you created in Postman to send to Deepstack is not correct. You need to first make sure you are sending correct requests to Deepstack.

Once the user realized their Deepstack installation was not working correctly, they reinstalled Deepstack. Deepstack does provide error logs, "C:\Users\[Username]\AppData\Local\DeepStack\logs" (%LOCALAPPDATA%\DeepStack\logs). Understanding the logs were challenging so best course of action was to try reinstalling Deepstack.

User also noted uninstalling Deepstack may not flush out all the files. The user went back into the installation directory (C:\DeepStack) and manually deleted any remaining files.

Furthermore, user noted, it's good to go into task manager and make sure any old Deepstack processes are not running. You can go into Task Manager and kill any of the following processes: server.exe, python.exe, redis-server-exe if they exist.

FYI, installing / using AI Tool for debugging may be worthwhile as BI continues to build out the debugging capabilities. In AI Tool, error messages will pop up right away if old Deepstack versions are still running.

Another clue that Deepstack is not running properly is by running Deepstack manually and observing the response time in the console. If Deepstack is taking 1m to respond, there is probably something wrong.

Front door view has the road and therefore a lot of unnecessary alerts. All you really want are people coming to the house or property.


So you create a zone in front of your house to reduce the noise from the road.


No alerts! The problem is the camera will trigger when a person's feet crosses Zone B. The image being processed by Deepstack is just the feet! So Deepstack will often return nothing found.

The fix! Make sure to always define a Zone A which is the overall area of interest on the camera. The image processed by Deepstack is the overlap of all the zones, thus the entire person is sent to Deepstack for processing and the appropriate alert is sent.



One important point that I forgot to mention in the Webinar but did come up in the Q&A is how to tell BI to only be concerned with motion in Zone B??? Simple, see below. By setting "Object crosses zones" to B, I am telling BI to only worry about motion in Zone B. Notice with AI, I uncheck "Object travels" and "Object size exceeds" settings. With AI, the motion sensor settings can usually be simplified!