Attempts to find humans through OpenCV facedetect. Utilizes a pan / tilt kit and LKOptical track points to track human after detection.



So far you have to use Eclipse and change the file in the service directory to set your Arduino pin and com port settings, etc. Then start it like you would any other MRL Service.

This java code sets everything up:

  1.  // create services ==============================================
  2. Runtime.createAndStart("runtime""Runtime");
  3. Arduino arduino = (Arduino) Runtime.createAndStart("arduino""Arduino");
  4. Speech speech = (Speech) Runtime.createAndStart("speech""Speech");
  5. Servopan = (Servo) Runtime.createAndStart("pan""Servo");
  6. Servo tilt = (Servo) Runtime.createAndStart("tilt""Servo");
  7. Twitter witter = (Twitter) Runtime.createAndStart("twitter""
  8. OpenCV opencv = (OpenCV) Runtime.create("opencv""OpenCV");
  9. PID xpid = (PID) Runtime.createAndStart("xpid""PID");
  10. PID ypid = (PID) Runtime.createAndStart("ypid""PID");
  11. Java java = (Java) Runtime.createAndStart("java""Java");
  12. Sphinx ear = (Sphinx) Runtime.createAndStart("sphinx""Sphinx");
  14. //xpid ==============================================
  15. xpid.setMode(1);
  16. xpid.setOutputRange(-11);
  17. xpid.setPID(;
  18. xpid.setControllerDirection(0);
  19. xpid.setSetpoint(80);// #setpoint now is 80 instead of 160 because of 2
  20.                                                       // PD filters
  22. // ypid ==============================================
  23. ypid.setMode(1);
  24. ypid.setOutputRange(-11);
  25. ypid.setPID(;
  26. ypid.setControllerDirection(0);
  27. ypid.setSetpoint(60); // set point is now 60 instead of 120 because of 2
  28.                                                           // PD filters
  29. xpid.invert();
  31. //twitter ==============================================
  32. // twitter.setSecurity("","","","");
  33. twitter.configure();
  34. // twitter.tweet("#myrobotlab is Awesome!")
  36. // arduino ==============================================
  37. arduino.setSerialDevice("/dev/ttyACM0"57600810);
  39. // opencv ==============================================
  40. opencv.startService();
  41. opencv.addFilter("Gray""Gray");
  42. // add enough pyramid down filters to get 160x120 resolution
  43. // opencv.addFilter("PyramidDown1", "PyramidDown");
  44. // opencv.addFilter("PyramidDown2", "PyramidDown");
  45. opencv.addFilter("lk""LKOpticalTrack");
  46. opencv.addListener("publishOpenCVData"this.getName()"input",
  47. OpenCVData.class);
  48. opencv.setCameraIndex(1);
  50. // ear ============
  51. ear.addCommand("find human", opencv.getName()"capture");
  52. ear.addCommand("kill all humans"this.getName()"stopEverything");
  53. ear.addCommand("happy fun"this.getName()"tweetOn");
  54.  ear.addCommand("two two four"this.getName()"tweetOff");
  55. ear.addCommand("talk to me"this.getName()"speechOn");
  56. ear.addCommand("no talking"this.getName()"speechOff");
  57. ear.addComfirmations("yes");
  58. ear.addNegations("no");
  59. ear.attach(speech);
  60. ear.startListening();
  62. // now get the servos working ============================
  63. arduino.attach(pan.getName()14);
  64. arduino.attach(tilt.getName()15);
  65. pan.moveTo(90);
  66. tilt.moveTo(90);
  67. lights(010);

And then this method is where all the magic happens. The FaceDetect and LKOptical filters pump an OpenCVData object to this method for processing:

  1.   public void input(OpenCVData opencvData) {...}



GroG's picture


Thanks for making the service..
Thanks for making the video...
Thanks for adding the video to the Service page !

I'm wondering ... 
0:13 - it sets an LK Point on your shirt?  

The red light is lock & kill - LEDs are a nice touch :D  - it's cool to see the Sentry do mood swings :)

The biggest question is why the tight spiral when a LKOptical point is set?  Is this intentional or is it from hysteresis ?

This is turning into a WICKED COOL SENTRY !!!!

raver1975's picture


The tight spiral is from hysteresis, the servo's keep compensating, maybe if I change the PID settings it will be better.

There is still a problem where the camera is in motion, and I'm not sure if there is a problem with the commands in the input method "overlapping",, for example if the last time the code ran through the input method and it changed the servo postion, if the servo is in the right position for the next time through the input method. So I think there are some delay effects with setting the LKOptical filter'ss tracking point.

GroG's picture

Hmmm... I have

Hmmm... I have seen hysteresis a lot...  never as a spiral...

are you sure the spiral code isn't running at the same time LKTrack is trying to track :D ?

To me the motion makes sense - both are pumping info to the servo - LK / PID system is trying to keep it on the point and the servo search is trying to spiral ... add them together and you get tiny spiral :)

GroG's picture

Was "no talking" supposed to

Was "no talking" supposed to have shut it up ? :)

I love the spiral search - I noticed that when you got up - you left the field of view and the lkpoint was lost.. 

an interesting algorithm might be to follow the trajectory of the last LKPoint and start a spiral (small to big) from guestimated point location..