Blue Iris

I did this a couple of years ago to show the difference in speed between DeepStack and CPAI and the different object detection modules including custom models. MikeLuds custom models were shown to be MUCH faster than the default OBJECTS model, but that was expected as he had removed items that we are not interested in (Broccoli comes to mind) for CCTV. That analysis all got lost in the big forum crash
The idea was to use the full text logs from BI5 (One month of data) which contain the identification module name (Thanks Ken), the object and the timing data. The logs look like this:

0 18/05/2024 09:00:30.294 Drive AI: [ipcam-combined] person:85% [808,44 855,128] 115ms
0 18/05/2024 09:00:30.294 Drive AI: [ipcam-combined] car:79% [892,65 944,108] 115ms
0 18/05/2024 09:00:30.294 Drive AI: person:85%,car:79%

So the model, identification type and timing can all be derived from the Logs.

Since this is for Excel, you can then take many thousands of data points and Pivot the data to see what happens before and after a trigger time. This morning at 10:30 I changed CPAI from using YOLOv5 6.2 to YOLOv5 .NET as it should be faster with my GTX1650 and I7-8700K. Here I have taken 2.5 hours before and 2.5 hours after that, which is 3,325 rows in the Log, and Excel has averaged the times in msec:


You can do this after making changes to see what effect it has had. Here it is is clear that .NET reduces the identification time by 25% on my rig.

So here I am a week later. I have now done the same analysis with 7 days of data before and after the change. This is the Excel analysis of 107,974 rows of data from the Log that all starts with "AI[" to weed out other Log entries. You don't need a calculator to see that YOLOv5 6.2 was having problems with my BI5/CPAI installation, and that YOLOv5 .NET is the model for me:


Note: This doesn't mean model x is better than model y. It simply proves that model y works better on my BI5 system than model x does. Data analysis tells you what is happening so that you can make informed decisions. This isn't a controlled lab test either; just my home cctv data Log being analysed.