I THINK my A.I. YOLOv4 training is underway … I will know in 12 hours :-O – no make that 16 now

Posted on by

I have 13 classes and I followed Ritesh and his gang at Augmented startups and bought his course which won’t be ready for 2 more weeks. He is a super nice guy and he and a couple others will work with me to get it running – but I think I just succeeded (there is an 8 hour time difference)! I also followed this guy on medium.com’s tutorial but am NOT using Colab – I have set things up per the tutorials online for that course! But I think it is working. I am getting a blur of text scrolling at insane speeds (if you pull up the handle on the scroll window it stops enough for you to read) in my cmd window and it says 12 hours left. I think I got carried away classifying things – I had only intended on 3.

I tried to follow AlexeyAB/darknet’s instructions but at one point – he starts talking about something – I think it was dragged out of place or he didn’t finish his … SQUIRREL! Not sure. I will document what I did and my performance

Anyway, to document some of this stuff for myself – I have adjusted the yolo-custom.cfg file in the following places (as recommended by the medium.com’s link above):

Performance on 1st go . I am not sure if its using the GPU (or the correct GPU) – add – -gpus 0 (that is a zero) and see very bottom of this article- I think it is indeed working.

batch=16
width=416 # not sure why i need to change this but I don't know enough yet
height=416
max_batches = 26000 # (classes * 2000)
steps=20800,23400 #80% and 90% of 26000
filters=54 #(classes + 5) * 3  ... in 3 [yolo] layers
classes=13 # number of classes ... in 3 [yolo] layers
filters=54
classes=13
filters=54
classes=13

I think the above is self-explanatory esp. in light of the medium.com article. THese are the standard things to change except the batch size.

Also I had to reduce the batch to 8 for it to not complain about running out of CPU ram. I expected it to complain about GPU ram – but it was CPU ram – there IS a message about GPU ram but that didn’t appear.

My memory is 3GB over normal, my GPU 1 is only at 4% (the NVIDIA CARD) and GPU 0 is at 11-18% (this is the one running the screen I am typing on). I wonder if its using the GPU – Darknet was compiled to use the GPU so I am a little confused but being a Friday night – 12 hours is not going to kill me or the CPU – so lets crack on and see what happens.

Here is some future reading – I need t0 figure this out.

https://cloudxlab.com/blog/video-processing-with-yolov4-and-tensorflow/
https://pjreddie.com/darknet/install/
https://github.com/pjreddie/darknet

GOT IT! …

YES! Now look at the Processes which was blank before - darknet.ext is indeed using up resources and its pulling 77W so its doing something. Sucks though cause now it threatens 16 hours ... argggg. Anyway now that is running I can get back to surfing and figuring out other ways to reduce speed (eg - do I need more CPU ram? I can do that - that's cheap).

C:\>nvidia-smi
Fri Jun 26 18:38:04 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 442.23       Driver Version: 442.23       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 2060   WDDM  | 00000000:01:00.0 Off |                  N/A |
| N/A   78C    P2    82W /  N/A |   2316MiB /  6144MiB |     63%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2620      C   ...4\darknet\build\darknet\x64\darknet.exe N/A      |
+-----------------------------------------------------------------------------+