Gael recently pointed us at the an open source project that had trained a few different neural networks in Keras to detect emotions on detected faces.  More info here.

https://github.com/omar178/Emotion-recognition

We were able to take the model from that project that was trained with Keras and safed in HDF5 file format and load that into Deeplearing4j.  The model that was trained in the emotion-recognition project was called the "mini XCEPTION" network.  There seem to be a few different approaches in that open source project, but this is the one that was posted that was trained. 

So, it takes a  recognized face in and passes it to the emotion classifier.  This new stage requires that a face detector filter is in the image pipeline.  

If a face was recognized in the pipe the image is then cropped.  The cropped image is a square and there is an optional border in number of pixels (this is the "box slop" factor.)

That cropped image is cut to gray scale, and resized to 64x64 resolution and all the color values are normalized between 0-1   (divide by 255) ..   That preprocessed image is then passed into the mini XCEPTION network.. where the output is the probabilities of the following categories.

  • angry
  • disgust
  • scared
  • happy
  • sad
  • surprised
  • neutral

A confidence threshold can be applied to trim down the results from the classifier..

Here's a few examples.

Happy..

 

Neutral

Angry...

hairygael

5 years 1 month ago

Wow impressive integration!

I am always amazed to see how quickly you can understand and integrate so much code into a service or a filter. I had spent almost two days trying to make a terrible adaptation via python and didn't succeed to even import the models... So I am very impressed!

Looking at the exemples, I notice the FaceDetect bounding box isn't showing and the result is put on the camera frame, what happens when the camera sees 2 or three faces? How do we determine who is happy or angry?

Could we have the bounding box display the state of each persons? Like below:

Or does the size of the bounding box determines the closest personne which is then taken in account for the single result?

Thanks for the encouragement.  This stuff is pretty tricky to get right.  When it comes to loading a custom model from keras (or any other deep learning framework)  it's important to make sure that you format the input to the network in the way that the network is expecting it.

In this case, the input to the neural network needed to be a 64 x 64 pixel image in gray scale 8 bit color.  

So, the work was to take the output of the face detector filter, and crop the image to just the detected face.

Anyway,, once you got the input image in the right format/size/shape... then you can feed it through the network and you get an array of numbers on the output.  

The last part is to get the labels for what each of those numbers means...  

Anyway, each neural network has slightly different requirements as to the format of the input, which makes it a bit difficult to integrate new ones arbitrairly...

Anyway.. yes. we can add the bounding boxes to the faces..  I was being lazy and just put the label in the upper corner of the screen.  There's no reason it can't track multiple faces.. but for now, it's only setup to track the current face detected by the face detect filter...  

I'm sure there will be a few itterations of polish on the emotion detection...  The more I think about it, the more I think that face detection , face recognition and emotion detection all really should be grouped together .. perhaps some sort of an uber filter that only processes faces that are recognized in images... ultimately, we'd want to have a published callback with detailed information about, not only that a face was detected, but who it was and how they are feeling...

 

Woo ooo testing the filter for emotion!

It's fun and I am sure we can do many stuff with it!

Although  I am facing an issue, does the FaceDetect have a problem with 214 version? My CPU i7 3.60ghz rise up to 85/90% of use with the single FaceDetect filter. (nothing else running on the PC)

Adding the Emotion filter and my CPU is at 100% getting over temp in a few seconds.

I am wondering if you encounter the same effect.