So...  it works! sweet! Here's the proof!

 

 

I've integrated Deeplearning4j into MyRobotLab, so now we can take advantage of neural networks to help us recognize objects in video streams in real time.  Deeplearning4j is an open source project and you do not need to be connected to the internet to use it, except for the first time it downloads the models.

We have integrated an image classifier based on the VGG16 model that was built using the ImageNet dataset.  It does a pretty good job as recognizing about 9000 different objects.  This model is about 500MB so when it first downloads, it's pretty slow, so be patient!

So, the way it works now, if you add the "dl4j" filter to the OpenCV service, it will start a deeplearning4j service, and load the VGG16 model.  As the video stream comes in from OpenCV, the frames are passed to the deeplearning4j service and it returns back a list of items that it has classified and their associated confidence levels of the classification.

One other item of note is that the classification happens as fast as it can on a separate thread, so adding this filter to opencv doesn't slow down the frame rate at all!!!  I was about to get 30FPS from OpenCV (even though, it was only updating the classification about once per second.) 

I am so amazed with how you have been able to get this all integrated. I will have to dig through the site for a how to on getting this all setup when I get home today and try to lock myself in the lab for the weekend! Thanks for sharing and for all your hard work!!! I really appreciate your dedication to continue to make this pproject cutting edge!!!

Kyle

Doki

7 years 6 months ago

do you know the Yolo Network ? 

If this works also with Dl4J it would be able to detect multiple objects and not just one.

https://pjreddie.com/darknet/yolo/

There are plenty of videos on youtube about the yolo neural network. and the wights are avalible for download also i think.

Great work Kwatters

My observations so far

1.. It's awesome

Here's some Python to get it worky

mrl development build 2354

from org.myrobotlab.service import Runtime
from org.myrobotlab.service import OpenCV
from time import sleep
 
# create and start the opencv service 
opencv = Runtime.createAndStart("opencv","OpenCV")
 
# start the camera, change the index dependent on which camera in your setup that you want to use
opencv.setCameraIndex(1)
opencv.capture()
 
#sleep for a bit to let opencv warm up :)
sleep(3)
 
#add the deep learning filter
opencv.addFilter("dl4j", "DL4J")
 
# If this is the first time you've invoked the dl4j service then   
# wait until the traning data zip file is downloaded to your home folder
# it is just over 500Mb
# you will see some dl4j diagnostic messages in the log output
#
#[python.interpreter.3] INFO  org.deeplearning4j.zoo.ZooModel - Checksum local is 3501732770, expecting 3501732770
#[python.interpreter.3] INFO  org.nd4j.linalg.factory.Nd4jBackend - Loaded [CpuBackend] backend
#[python.interpreter.3] INFO  org.nd4j.nativeblas.NativeOpsHolder - Number of threads used for NativeOps: 2
#[python.interpreter.3] INFO  org.reflections.Reflections - Reflections took 168 ms to scan 8 urls, producing 29 keys and 189 values
 
# Wait for around 10 minutes after you see the dl4j diagnostic messages. 
# In practice, I've found that 5 minutes can be enough of a wait 
# Something apears to be blocking so do a Control C in the command shell from where you launched mrl. 
# Wait a minute or two more
# You'll then see the dl4j filter appear in the target filter list 
# Place objects in the field of view of the camera

 

Kakadu31

7 years 6 months ago

It seems I have Problems downloading the zip folder, i tried severall times but it wont work. Send a noworky. Im trying on Windows 10 wiht opencv 3.3 build 2394.

Dear Acapulco Rolf 

Thanks for sharing these step , it's very helpful

I'm stuck on where the zip file should be downloaded !

and i did these step in my MRL version 1.0.2693.16 but it did not recognize anything..

Any advice ?

Thanks a lot  

Acapulco Rolf

7 years 5 months ago

Additional dependancies for the Raspebrry Pi are here

There is a readme file in the same location with instructions

There's an "out of memory" condition on the Pi when the dl4j service starts up

One way to overcome that will be to use a smaller model file than the 500Mb VGG16 model