Blue Iris

Re: CUDA Errors with YOLOv5 Object Detection on CPAI 2.5.x – Seeking Insights

2024-07-21T18:36:18+00:00

Looking good !

With "Reliability History" I had used a program called "WhoCrashed" and went down that rabbit hole. People at the time said they would have thrown it out of a window due to those multiple failure modes

well, just goes to show hey, how persistence and perseverance pays off.

Statistics: Posted by Sardon — Sun Jul 21, 2024 6:36 pm

Re: CUDA Errors with YOLOv5 Object Detection on CPAI 2.5.x – Seeking Insights

2024-07-17T04:46:47+00:00

Looking good !

With "Reliability History" I had used a program called "WhoCrashed" and went down that rabbit hole. People at the time said they would have thrown it out of a window due to those multiple failure modes

Statistics: Posted by TimG — Wed Jul 17, 2024 4:46 am

Re: CUDA Errors with YOLOv5 Object Detection on CPAI 2.5.x – Seeking Insights

2024-07-16T18:53:15+00:00

I previously found the .NET version faster with the GTX1650, but I haven't had time to test it yet with this version of CPAI. Your results will be interesting, but I need to get some days of data and use Excel pivot tables to tease out the truth.

As for the stability of this BI5 pc, it was a long hard fight. The entire story was lost in the great forum crash, but basically I found:

1. A satellite receiver card that periodically blocked USB ports.
2. A power supply that glitched occassionally,
3. A RAM stick that failed when the psu was OK,
4. A driver for my second NIC that was so old it prevented Win11 installing.

My biggest ally was "Reliability History" in Win10 and onwards. Well worth a look if you haven't seen it before:

Screenshot 2024-02-22 182915.png

I lock down Windows updates for the maximum period (until I have time to fix whatever breaks), but it looks like MS are trying to push some through. I'll check it this weekend; it's now the main reason I reboot that pc !

After so long, I've finally managed to resolve things! Seems I've identified issues that are now sorted out. We're headed to the topside!

Statistics: Posted by Sardon — Tue Jul 16, 2024 6:53 pm

Re: CUDA Errors with YOLOv5 Object Detection on CPAI 2.5.x – Seeking Insights

2024-03-03T12:33:36+00:00

Just wanted to drop a quick update; fingers crossed I'm not jinxing it by saying this too soon, but it looks like the CPAI 2.5.6 update has sorted out those CUDA memory access issues and bombing out after 10-12 hours.

After giving it some thought, I’ve realised CUDA is actually a bit snappier than DirectML. It's not a huge difference, just about 10ms (maybe even a little faster), but every little helps.

So far, so good – the system has been running smoothly without a hitch for the last 24 hours.

Module 'Object Detection (YOLOv5 6.2)' 1.9.1 (ID: ObjectDetectionYOLOv5-6.2)
Valid: True
Module Path: \modules\ObjectDetectionYOLOv5-6.2
AutoStart: True
Queue: objectdetection_queue
Runtime: python3.7
Runtime Loc: Shared
FilePath: detect_adapter.py
Pre installed: False
Start pause: 1 sec
Parallelism: 0
LogVerbosity:
Platforms: all,!raspberrypi,!jetson
GPU Libraries: installed if available
GPU Enabled: enabled
Accelerator:
Half Precis.: enable
Environment Variables
APPDIR = \modules\ObjectDetectionYOLOv5-6.2
CUSTOM_MODELS_DIR = \modules\ObjectDetectionYOLOv5-6.2\custom-models
MODELS_DIR = \modules\ObjectDetectionYOLOv5-6.2\assets
MODEL_SIZE = Medium
USE_CUDA = True
YOLOv5_AUTOINSTALL = false
YOLOv5_VERBOSE = false
Status Data: {
"inferenceDevice": "GPU",
"inferenceLibrary": "CUDA",
"canUseGPU": "true",
"successfulInferences": 51402,
"failedInferences": 9,
"numInferences": 51411,
"averageInferenceMs": 45.582642698727675
}
Started: 02 Mar 2024 11:51:32 AM GMT Standard Time
LastSeen: 03 Mar 2024 12:25:46 PM GMT Standard Time
Status: Started
Requests: 51412 (includes status calls)

Statistics: Posted by Sardon — Sun Mar 03, 2024 12:33 pm

Re: CUDA Errors with YOLOv5 Object Detection on CPAI 2.5.x – Seeking Insights

2024-02-25T11:35:15+00:00

I wish to convey my admiration for the state of your reliability monitor; it truly stands out for its impressive performance.

My setup, however, has encountered challenges, particularly following a Blue Iris crash yesterday, which I believe was triggered by my balcony camera freezing, necessitating a restart of the application.

Upon reviewing the reliability monitor, I've noted several Blue Iris crashes, which underline persistent instability issues tied to CPAI and CUDA errors, as illustrated below.

Adopting Yolo.NET (DirectML) has, fortunately, led to a crash-free experience in relation to the GPU, puzzling me about the underlying causes of CUDA's instability. DirectML has not crashed since:

This experience has solidified my decision to continue using Yolo.NET moving forward.

The switch to .NET has not only stabilised the system but also seems to have slightly improved inference times. When comparing performance, the medium model under .NET appears to either match or slightly outperform CUDA's speed with a small model, which is quite fascinating considering CUDA's hardware specificity. However, faced with CUDA's consistent failure after 11 hours of operation, I find myself with no choice but to adopt DirectML.

An intriguing point I've observed, albeit not definitively linked to .NET, is a detectable inconsistency in CPAI's object detection capabilities, as highlighted below:

Despite being in the frame under strict AI settings (Night Profile: Min Confidence 60%, Pre-Trigger images: 3, Post-Trigger images: 30, Analyse one image every 100ms), the system failed to detect me in that instance. Strangely, it had successfully detected me when I returned home yesterday evening in the same lighting conditions etc. This inconsistency is bewildering, especially given the meticulously tuned settings aimed at ensuring maximum detection reliability.

The transition from CUDA to DirectML may have influenced the frequency of detections, a shift from the previously uninterrupted detection performance. This change could either be coincidental or a minor glitch that happened to coincide with the move from CUDA to DirectML.

Sharing this concern seemed necessary as it continues to merit further attention.

In my quest for solutions, I stumbled upon a recent topic on here suggesting that "unicorn" is now obsolete. Also, after further digging I found elsewhere that it was suggested adjusting the global AI tab in the camera settings. By entering "Nothing found:0" in the "To cancel" box, it effectively removes "Nothing found" alerts (marked green) from the Confirmed alerts list, compelling the AI to review all images within an alert to identify the best one. Sounds very similar to the Unicorn trick?

It does make me wonder if replacing "unicorn" with "Nothing Found:0" in the cancel field might compel the AI to re-examine the images and successfully identify the object, or is this wrongly assumed ?

I'll conclude my observations here, as I've shared a substantial amount and have other concerns to address in separate posts.

To recap, I am determined to continue utilising the reliability monitor to address any persisting errors and enhance the score as much as possible for system stability purposes and I really need to figure out why CPAI fails to detect sometimes as per the example above.

Regards
SN

Statistics: Posted by Sardon — Sun Feb 25, 2024 11:35 am

Re: CUDA Errors with YOLOv5 Object Detection on CPAI 2.5.x – Seeking Insights

2024-02-22T18:35:06+00:00

I previously found the .NET version faster with the GTX1650, but I haven't had time to test it yet with this version of CPAI. Your results will be interesting, but I need to get some days of data and use Excel pivot tables to tease out the truth.

As for the stability of this BI5 pc, it was a long hard fight. The entire story was lost in the great forum crash, but basically I found:

1. A satellite receiver card that periodically blocked USB ports.
2. A power supply that glitched occassionally,
3. A RAM stick that failed when the psu was OK,
4. A driver for my second NIC that was so old it prevented Win11 installing.

My biggest ally was "Reliability History" in Win10 and onwards. Well worth a look if you haven't seen it before:

I lock down Windows updates for the maximum period (until I have time to fix whatever breaks), but it looks like MS are trying to push some through. I'll check it this weekend; it's now the main reason I reboot that pc !

Statistics: Posted by TimG — Thu Feb 22, 2024 6:35 pm

Re: CUDA Errors with YOLOv5 Object Detection on CPAI 2.5.x – Seeking Insights

2024-02-22T10:31:09+00:00

Given the ongoing issues, I've decided to experiment with the .NET version of YOLO, which uses DirectML. I'm hoping this change might alleviate the crashes that have occur ever 10/12 hours with the CUDA variant.

It is baffling though, that considering my setup mirrors Tim's, identical GPU, identical drivers etc, yet, mine buckles every 10-12 hours. Anyway I've fired off a response over on the CPAI forum, we'll see...

Switching to the .NET variant is primarily for testing purposes. I'm a bit apprehensive about potential impacts on inference times, but we'll see. Here's to hoping it can make it through a day without faltering. Fingers crossed.

Catch you later with updates!

Statistics: Posted by Sardon — Thu Feb 22, 2024 10:31 am

Re: CUDA Errors with YOLOv5 Object Detection on CPAI 2.5.x – Seeking Insights

2024-02-22T09:05:32+00:00

Your system has been running smooth for nearly a month.

Started: 31 Jan 2024 8:35:56 AM GMT Standard Time
LastSeen: 21 Feb 2024 5:52:45 PM GMT Standard Time

My system is now showing a pattern of when it crashes. It's crashing after approximately 10-12 hours of continual use. I knew the small model weren't the issue nor the memory size of the GPU as indicated by Chris M and this troubles me.

For Chris M to put it down to a model size is not the kind of support response I'd expect especially coming from someone who is the co-founder of the project.

I'm going to write another response with my pattern findings and advise despite setting the model to small it's still bombing out with the same errors.

The only work-around I can think of at present is for me to restart the service twice per day which is pants in all fairness, it's just not sustainable having to do so but there is nothing else I can do at this stage.

I'm well gutted to be honest... oh well, maybe I'll have to go back to to original setup NON CPAI mode with false alerts LOL

Statistics: Posted by Sardon — Thu Feb 22, 2024 9:05 am

Re: CUDA Errors with YOLOv5 Object Detection on CPAI 2.5.x – Seeking Insights

2024-02-21T18:06:13+00:00

I have CPAI working on 5 2MP cameras and a 5MP camera. Clearly it's on the sub-streams ( BI5 AI settings/ "Use main stream if available" is un-ticked). Three of the 2MP cameras and the 5MP can trigger at similar times as they are all on the front of the house (Left High ptz, Reolink doorbell, Drive camera, Right High ptz) and I have had no visible problems with CPAI or CUDA.

Statistics: Posted by TimG — Wed Feb 21, 2024 6:06 pm

Re: CUDA Errors with YOLOv5 Object Detection on CPAI 2.5.x – Seeking Insights

2024-02-21T17:56:42+00:00

The Info button on YOLOv5 6.2 shows this, so Model size=Medium:

Module 'Object Detection (YOLOv5 6.2)' 1.9.0 (ID: ObjectDetectionYOLOv5-6.2)
Valid: True
Module Path: \modules\ObjectDetectionYOLOv5-6.2
AutoStart: True
Queue: objectdetection_queue
Runtime: python3.7
Runtime Loc: Shared
FilePath: detect_adapter.py
Pre installed: False
Start pause: 1 sec
LogVerbosity:
Platforms: all,!raspberrypi,!jetson
GPU Libraries: installed if available
GPU Enabled: enabled
Parallelism: 0
Accelerator:
Half Precis.: enable
Environment Variables
APPDIR = \modules\ObjectDetectionYOLOv5-6.2
CUSTOM_MODELS_DIR = \modules\ObjectDetectionYOLOv5-6.2\custom-models
MODELS_DIR = \modules\ObjectDetectionYOLOv5-6.2\assets
MODEL_SIZE = Medium
USE_CUDA = True
YOLOv5_AUTOINSTALL = false
YOLOv5_VERBOSE = false
Status Data: {
"successfulInferences": 1237095,
"failedInferences": 554,
"numInferences": 1237649,
"numItemsFound": 1403967,
"averageInferenceMs": 50.89386587125483,
"histogram": {
"car": 1291233,
"person": 51705,
"truck": 36987,
"bird": 11131,
"DayPlate": 7780,
"bus": 3621,
"bicycle": 308,
"horse": 11,
"dog": 973,
"cow": 28,
"deer": 49,
"cat": 41,
"motorcycle": 73,
"NightPlate": 5,
"pig": 2,
"bear": 3,
"sheep": 17
}
}
Started: 31 Jan 2024 8:35:56 AM GMT Standard Time
LastSeen: 21 Feb 2024 5:52:45 PM GMT Standard Time
Status: Started
Requests: 1237660 (includes status calls)
Provider: CUDA
CanUseGPU: True
HardwareType: GPU

Installation Log
2024-01-28 13:18:01: Installing CodeProject.AI Analysis Module
2024-01-28 13:18:01: ======================================================================
2024-01-28 13:18:01: CodeProject.AI Installer
2024-01-28 13:18:01: ======================================================================
2024-01-28 13:18:01: 143.8Gb of 237Gb available on
2024-01-28 13:18:01: General CodeProject.AI setup
2024-01-28 13:18:01: Creating Directories...Done
2024-01-28 13:18:01: GPU support
2024-01-28 13:18:02: CUDA Present...Yes (CUDA 11.8, cuDNN 8.5)
2024-01-28 13:18:02: ROCm Present...No
2024-01-28 13:18:04: Reading ObjectDetectionYOLOv5-6.2 settings.......Done
2024-01-28 13:18:04: Installing module Object Detection (YOLOv5 6.2) 1.9.0
2024-01-28 13:18:04: Installing Python 3.7
2024-01-28 13:18:04: Python 3.7 is already installed
2024-01-28 13:18:04: Creating Virtual Environment (Shared)...Virtual Environment already present
2024-01-28 13:18:04: Confirming we have Python 3.7 in our virtual environment...present
2024-01-28 13:18:20: Downloading Standard YOLO models...Expanding...Done.
2024-01-28 13:18:20: Copying contents of models-yolo5-pt.zip to assets...done
2024-01-28 13:18:37: Downloading Custom YOLO models...Expanding...Done.
2024-01-28 13:18:37: Copying contents of custom-models-yolo5-pt.zip to custom-models...done
2024-01-28 13:18:37: Installing Python packages for Object Detection (YOLOv5 6.2)
2024-01-28 13:18:37: [0;Installing GPU-enabled libraries: If available
2024-01-28 13:18:38: Ensuring Python package manager (pip) is installed...Done
2024-01-28 13:18:40: Ensuring Python package manager (pip) is up to date...Done
2024-01-28 13:18:40: Python packages specified by requirements.windows.cuda.txt
2024-01-28 13:18:41: - Installing Pandas, a data analysis / data manipulation tool...Already installed
2024-01-28 13:18:43: - Installing CoreMLTools, for working with .mlmodel format models...Already installed
2024-01-28 13:18:44: - Installing OpenCV, the Open source Computer Vision library...Already installed
2024-01-28 13:18:45: - Installing Pillow, a Python Image Library...Already installed
2024-01-28 13:18:46: - Installing SciPy, a library for mathematics, science, and engineering...Already installed
2024-01-28 13:18:47: - Installing PyYAML, a library for reading configuration files...Already installed
2024-01-28 13:18:48: - Installing PyTorch, an open source machine learning framework...Already installed
2024-01-28 13:18:49: - Installing TorchVision, for working with computer vision models...Already installed
2024-01-28 13:19:39: - Installing Ultralytics YoloV5 package for object detection in images...(checked) Done
2024-01-28 13:19:40: - Installing Seaborn, a data visualization library based on matplotlib...Already installed
2024-01-28 13:19:40: Installing Python packages for the CodeProject.AI Server SDK
2024-01-28 13:19:42: Ensuring Python package manager (pip) is installed...Done
2024-01-28 13:19:44: Ensuring Python package manager (pip) is up to date...Done
2024-01-28 13:19:44: Python packages specified by requirements.txt
2024-01-28 13:19:45: - Installing Pillow, a Python Image Library...Already installed
2024-01-28 13:19:47: - Installing Charset normalizer...Already installed
2024-01-28 13:19:48: - Installing aiohttp, the Async IO HTTP library...Already installed
2024-01-28 13:19:49: - Installing aiofiles, the Async IO Files library...Already installed
2024-01-28 13:19:50: - Installing py-cpuinfo to allow us to query CPU info...Already installed
2024-01-28 13:19:52: - Installing Requests, the HTTP library...Already installed
2024-01-28 13:19:57: Fusing layers...
2024-01-28 13:19:57: YOLOv5.1m summary: 391 layers, 21805053 parameters, 0 gradients
2024-01-28 13:19:57: Adding AutoShape...
2024-01-28 13:19:58: Self test: Self-test passed
2024-01-28 13:19:58: Module setup time 00:01:56.48
2024-01-28 13:19:58: Setup complete
2024-01-28 13:19:58: Total setup time 00:01:57.35
Installer exited with code 0

Statistics: Posted by TimG — Wed Feb 21, 2024 5:56 pm

Re: CUDA Errors with YOLOv5 Object Detection on CPAI 2.5.x – Seeking Insights

2024-02-21T16:34:12+00:00

Thank you for sharing that info Tim.

I'm still facing errors with my setup, which are frankly beginning to frustrate me. Out of curiosity, may I ask what size model you are utilising?

Chris M recommended opting for a smaller model size to strike a balance, in fact here is his quote

"I have a NVIDIA 1030 and it's pretty basic but still does OK. The trick is definitely to balance accuracy and model size, so I'd definitely recommend trying a smaller model size (the dashboard allows you to do this via the gear icons on each module). Also try to minimise the number of modules you have that are using the GPU.

I'm sceptical that the model size is not the root of the issue. Despite altering my configuration to not use the IGPU solely for display purposes (it's also headless as I utilise RDP), I encountered errors again this morning, indicating that something is amiss.

The plan is to now follow Chris M's advice and test with a smaller model size to see if the error persists, which prompted me to ask about your preferred model size. I've noticed it takes around 10/12 hours for this bug to occur in CPAI. So if all goes well, since I set this small model and no errors by tomorrow, it may be the model size. God knows ! LOL

Anyway , If I continue to face issues with a smaller model, it will necessitate a thorough reassessment of my approach. It's particularly disappointing as I've made significant progress with CPAI, which has been performing admirably until now, only to be hindered by this vexing software issue, rather typical, in all fairness.

At this juncture, I'm contemplating a complete uninstallation and fresh installation as a potential solution. I'll observe how this unfolds tonight. Additionally, I've noticed that CPAI's inference times are exceedingly high upon initial start-up but seem to stabilise and improve to lower millisecond durations with continued use. This pattern, starting in the thousands and gradually reducing might be normal, although it's a consistent observation.

Could you elucidate the differences between these model sizes and their respective impacts? On a positive note, CPAI successfully just detected someone at my front door, so it's still operational using a small model to some extent.

Would be good to know what the different size models contain or how they are built. I looked on CPAI and didn't find anything

Now, I shall turn my attention to the other discussion regarding daylight saving time. Hopefully I can get that sorted too!

Thanks again man for all the help
SN

Statistics: Posted by Sardon — Wed Feb 21, 2024 4:34 pm

Re: CUDA Errors with YOLOv5 Object Detection on CPAI 2.5.x – Seeking Insights

2024-02-20T22:18:15+00:00

I am a version or two behind with CPAI v2.5x but am up to date with BI5.

Regular back ups is the only reason I don't fall down too many rabbit holes, as I like to try things to help other forum members.

Statistics: Posted by TimG — Tue Feb 20, 2024 10:18 pm

Re: CUDA Errors with YOLOv5 Object Detection on CPAI 2.5.x – Seeking Insights

2024-02-20T19:58:45+00:00

I'm eager to learn about your setup, do you employ an iGPU alongside your GTX 1650, or is the GTX 1650 your sole GPU for both display and processing?

Your insights could shed light on the root cause of my CUDA errors and the impact of my dual-GPU configuration.

I have to say that the rationale behind my dual-setup choice was to maximise resource efficiency, offloading display functions to the iGPU to free up the GTX 1650 for AI processing. I'm beginning to wonder if this ambitious setup might be the culprit behind the errors.

Please let me know and P.S. How much memory does your 1650 have ? Chris M thinks it may be a limitation due to the card only having 4GB ram. Surely can't be the case.

I run BI5 on a headless pc, so there is no monitor connected, just a dummy HDMI plug in the GTX1650 so that I can log in to it with "Splashtop" which is like Teamviewer. The cpu is an i7-8700k and the pc has 16GB of RAM. The iGPU was used for BI5 acceleration, but I stopped using that when sub-streams came along. My GTX1650 is a 4GB RAM version. Task Manager screen looks like this:

Statistics: Posted by TimG — Tue Feb 20, 2024 7:58 pm

Re: CUDA Errors with YOLOv5 Object Detection on CPAI 2.5.x – Seeking Insights

2024-02-20T17:14:15+00:00

Hi Tim,

I'm currently grappling with persistent CUDA errors and am exploring every potential remedy. My system incorporates a dual GPU setup, using the integrated GPU for display (i9-10850k) and the GTX 1650 exclusively for AI computational tasks. Yet, the issue remains unresolved, prompting a revaluation of this strategy.

Additionally, I've initiated a discussion on the Code Project forum, where Chris M has requested further details. This effort aims to widen the quest for a solution. I'm eager to learn about your setup, do you employ an iGPU alongside your GTX 1650, or is the GTX 1650 your sole GPU for both display and processing?

Your insights could shed light on the root cause of my CUDA errors and the impact of my dual-GPU configuration.

I have to say that the rationale behind my dual-setup choice was to maximise resource efficiency, offloading display functions to the iGPU to free up the GTX 1650 for AI processing. I'm beginning to wonder if this ambitious setup might be the culprit behind the errors.

Please let me know and P.S. How much memory does your 1650 have ? Chris M thinks it may be a limitation due to the card only having 4GB ram. Surely can't be the case.

Regards
SN

Statistics: Posted by Sardon — Tue Feb 20, 2024 5:14 pm

Re: CUDA Errors with YOLOv5 Object Detection on CPAI 2.5.x – Seeking Insights

2024-02-18T21:19:44+00:00

I was going to suggest posting it over there as it does appear to be a CPAI issue. I've had no issues at all here with my GTX1650/ CUDA/ YOLO5, and we probably have the same drivers.

Statistics: Posted by TimG — Sun Feb 18, 2024 9:19 pm