Color Tracking In MRL using OpenCV, Jython, and Arduino Services

Hi guys! :)

I have a project that uses arduino and pan/tilt servos to track a desired color. With the use of a camera and myrobotlab, this project slowly come to life. I and with the help of GroG, are working on it to make it possible. However, it's not yet done and the project is still under progress. Here's some of the highlights of the project. Comments and Suggestions will be greatly appreciated :)

First, you need to download MRL:

-you can download MRL at this link. Be sure to pick the latest build ( at this time of writing, the latest is ver.770) Here's the link:


Next is you have to extract and open the folder and open myrobotlab.bat file:


After a while it should run like this:

Notice the BORG tab selected :)


Next is you should download the arduino service. Just right click the arduino service and click install. The same procedure applies to the OpenCV service

NOTE: Some anti-virus programs don't permit downloading the services. For me, I turned of my firewall and it worked.

Here's a pic (disregard the start under the info. It should be install because I already downloaded the service)


After downloading the Arduino and OpenCV service, Run OpenCV service by right clicking and choosing start. Then, name your service. Here I called it OpenCV. Click the OpenCV tab and scroll down the filters and select the PyramidDown filter to avoid the video booming up in your screen XD. and select the camera button then press capture. Normally the stop button is capture. :


Next is select the InRange filter to select the color you want to be tracked. Click the Inrange filter. Play with the values until the desired color will become white. 


Color tracking is a bit sensitive. Sometimes you should be consistent with the color ( due to poor lighting and etc. that affects the color consistency) so you'll have to update those values sometimes to get an accurate output.

Now, if you are all set, Remember the values in InRange filter. Like this from the picture above:

Parameters         Min          Max

   HUE                      3              33

SATURATION        87            256

VALUE                    151             256

If you are done taking down the values select the jython tab and copy these codes. You NEED to replace the values in "change values of the InRange filter with your own values that you took down down a while ago.

Now if you are done copying. Hit execute button( the button with a gear :D) . Notice in the pic below the jython console is printing x,y coordinates that later will be sent to arduino.


Now, start arduino service and follow these steps:

1- click tools and choose board then select the arduino model you have

2- click again tools and choose Serial Device then select the proper COM port the arduino uses(at this time your arduino should               be connected in your computer)

3- click the compile (check) button. Wait for it to finish compiling (when you see a "done" in the console)

4- click the upload button and again wait for it to finish

5- click the connect button. Wait for the "good times"


Test the connection by clicking the pins tab and selecting some analog pins to active. Put some wires or sensors (potentiometers, etc.) to the analog pins. Then, select oscope tab to view some values sent to the myrobotlab.


Stay tuned :)






GroG's picture

Yay !

Umm..  I don't understand what you mean?

Please change the servo script cause it's still in the loop <--   To what, do you mean to track ? 

We need 2 PIDs to map...  I'll start that script...

So when you run the whole script you should see the following behavior :
OpenCV -> starting, creating filters, setting values -> then publishing the center coordinates of an object
Arduino -> starting, connecting to the serial port, starting a analog pin (Oscope readout)
2 x Servos -> starting, attaching themselves to the Arduino on the correct pins - moving in a loop for a while

Are we good on all of these?

Did the upload to the Arduino work from MRL ?

Next will be to start 2 PIDs to do remapping off coordinates - they will listen to the OpenCV  X coordinate and translate to commands on the pan servo...
The other will listen to Y coordinates and translate to the Tilt servo..

michael96_27's picture

Nevermind this line: please

Nevermind this line: please change the servo script...... XD

Anyways, I uploaded successfully in the arduino in MRL eventhough it takes too long :) Oscope is running a trace too.

Yes, we can now start the final script :D

GroG's picture

Yes,getting closer to the


getting closer to the "final",  I've added 2 PID services (one X and one Y) - havent checked it in yet...

Do you have a usb joystick of some sort?  I'd like to test the tracking & PID with a joystick before feeding it OpenCV data...

It looks like the original joystick service isn't 64 bit compatible, but I have already found another one which is...  got to re-write the Joytick service too ...

michael96_27's picture

Yes I do have but it's a usb

Yes I do have but it's a usb steering wheel. Will it work?

GroG's picture

Pan and Tilt need 2 axis of

Pan and Tilt need 2 axis of input ...  don't know how that could be done with a steering wheel ;)

GroG's picture

The new gamepad library

The new gamepad library interface is completely different from the one I was previously using... 
This will take a little while..

michael96_27's picture

the usb gamepad will only be

the usb gamepad will only be used for testing right? 

BTW, If the gamepad lib/service is accomplished,  can we use that to control the robot for wireless applications?

GroG's picture

"the usb gamepad will only be

"the usb gamepad will only be used for testing right?"   <- correct, however, it would be easy enough to make it so you can "switch" to manual control from OpenCV tracking, with a push of a button.

"BTW, If the gamepad lib/service is accomplished,  can we use that to control the robot for wireless applications?"  <- yes, you have the right idea - Once a service is added to a system, its messages can be used to affect any other service...  So it can control Servo's, Motors, etc...  You could even have it control OpenCV filters .. for example pressing different buttons could change the filter configuration, and switch from Color Tracking to Motion tracking...

michael96_27's picture

Sounds exciting!  So, it

Sounds exciting! 

So, it looks like I have a next project after this color tracking isn't? XDD

michael96_27's picture

Maybe we have issues on rxtx

Maybe we have issues on rxtx that hinders us to communicate with the arduino isn't it? Anyways, I've sent you another logfile. And I'm already patched up with ver.786 bleeding edge.

Is there any micro servos like the ones that I use (Tower pro SG90) laying around in your house? 

Maybe we can do this together by making a clone of the servos shield. You just need a 4017 decade counter (I'm using CD4017 here in my shield) connect CLK to pin 6 and RST to pin7. 

Here's a pinout for cd4017:,r:5,s:0,i:85&tx=87&ty=13


And here's the link of the servoshield(not mine) download the eagle files for reference. Be Sure to just use the cd4017 that connects to pins 6-7 on the eagle pcb.


GroG's picture

Thanks for the info, Funny, I

Thanks for the info,

Funny, I have already started to put together a pan / tilt kit with 2 servos.
It will take me too long to make a shield, but thanks for the info.
I'll be working on a Adafruit Motor Shield - but the interface to the Servos is different.  It uses the Servo libraries which come from Arduino.

michael96_27's picture

Is the pan/tilt kit of yours

Is the pan/tilt kit of yours now working? :)

GroG's picture

"For me, the best step for

"For me, the best step for now is to verify Aceduino Shield service runs successfully. In order to that, can we try to use the scrollbar/trackbar in the service to position the desired servo?"  <- Can't do this yet, the Aceduino uses its own library and own API which is different from a typical Arduino Servo - We are trying to verify at a lower level (Programming level) that the Aceduino Shield can be controlled correctly by the Arduino.  After we do that we can standardize the Aceduino Service to use a "standard" Servo API and then you should be able to attach Servo's to it.

GroG's picture

Just so you know: The "I

Just so you know:

  • The "I feel lucky, give me the bleeding edge" - updates the myrobotlab.jar - which is the library which changes the most - it does not update any of the dependencies - e.g. other libraries, files or data which exist in that folder...   only the myrobotlab.jar
  • When you download the whole zip manually - you get the latest myrobotlab.jar + files, scripts, etc - however the only services that are initially in the zip is the Runtime & Jython

Did you download the zip (which is good), unzp, but not "install" the Arduino service, before you ran the script ?

GroG's picture

Joystick service done....

Or at least "done enough".  And the RXTX libraries have been updated with CreateLab's versions.  Unfortunately, create lab's version does not take out the large pause when searching for serial ports on Windows 64 bit os's.   The Arduino IDE suffers from this too, as you'll possibly notice when starting the Arduino IDE (the pause is not for graphics, but it's searching for COM ports at the time) - And the Arduino IDE searches for COM ports every time you press the "tools" menu.

Back to the Joystick, service :

I replaced old libraries with new Sun Dev libraries...

I created a Pan / Tilt kit script - it's under Jython->examples->basic->

the values are catered to my system...

I have also found out that Servos are increadibly noisy (electronically) .. so much so they mess with communication and other electronic values on the board...

I'll have a vid & pictures soon...

GroG's picture

I'm going to start the

I'm going to start the tracking script now.

Your tracking is only as good as the weakest link...   And my weakest link is between the arduino & the Servos (because of servo noise)

michael96_27's picture

Do I have to download the

Do I have to download the latest bleeding edge (zip) to get the new Joystick service or I'll just have to update MRL?

BTW: Nice stop and execute buttons on jython console :))

GroG's picture

Hi Michael, When the

Hi Michael,

When the repository changes, it's typically safer to download the zip and re-install the services.  In the future this might not be necessary..  but at the moment it is.

I need to finish out some form of Joystick GUI - one which allows you to select the input device - (Joystick, Gamepad, Steering Wheel) - and allows the output to be remapped to value ranges which you find useful.

For example : 
0 - 180 for typcial Arduino Servo
-1.0 - 1.0   for raw values
0.0 - 1.0 for positive real range

I'd be curious if your steering wheel is read ad a gamepad - I suspect that it is.

Thanks for the feedback on the buttons, I appreciate any ideas to make MRL  more "usable"

michael96_27's picture

Nevermind the steering wheel

Nevermind the steering wheel XD

I'll buy a gamepad tomorrow to suit the pan/tilt kit (steering wheel looks pretty awkward XD)

GroG's picture

I was thinking we might not

I was thinking we might not need PID to complete this if you want to map the values with a constant amount.

Let's try to out...

A servo has a range of 0 - 180, roughly 1 increment = 1 degree...
If you use a different servo library you may have a different range e.g. the ACEduino has 500 - 2500, but you can figure out what the timing value corresponds to a degree.

We want a point in the display to be mapped to a servo  position location - I think you were saying this all along :D

So you just need to figure out the degree value of a pixel location.  This depends on you specific camer, it's the "field of view" and is typically reported as - FOV - 60 degrees...

Since I could not find this spec for my webcam, I calculated it this way  (

So I have the range
OpenCV point x = 0  to 320 pixels with a camera which has 57 degrees field of view
320/57 = 5.6 pixels per degree

So for every 5.6 pixels the target is off center we want to move 1 increment of the servo..

Make sense ?

michael96_27's picture

Yes, it does make sense. The

Yes, it does make sense.

The idea somehow or almost alike with my Processing script. I tried some tutorials in the net and I found a face tracking tutorial. The idea is pretty much the same as what you said. It tracks a face by adding (if the face is on the right side) 1 or 2 degrees to the right. I've tried it on my aceduino shield and it successfully ran.

I'll post the Processing code and Arduino code. It might help :))



* Pan/Tilt Face Tracking Sketch
* Written by Ryan Owens for SparkFun Electronics
* Uses the OpenCV real-time computer vision  framework from Intel
* Based on the OpenCV Processing Examples from
* This example is released under the Beerware License.
* (Use the code however you'd like, but mention us and by me a beer if we ever meet!)
* The Pan/Tilt Face Tracking Sketch interfaces with an Arduino Main board to control
* two servos, pan and tilt, which are connected to a webcam. The OpenCV library
* looks for a face in the image from the webcam. If a face is detected the sketch
* uses the coordinates of the face to manipulate the pan and tilt servos to move the webcam
* in order to keep the face in the center of the frame.
* Setup-
* A webcam must be connected to the computer.
* An Arduino must be connected to the computer. Note the port which the Arduino is connected on.
* The Arduino must be loaded with the SerialServoControl Sketch.
* Two servos mounted on a pan/tilt backet must be connected to the Arduino pins 2 and 3.
* The Arduino must be powered by a 9V external power supply.
* Read this tutorial for more information:
import*;  //Include the video library to capture images from the webcam
import java.awt.Rectangle;  //A rectangle class which keeps track of the face coordinates.
import processing.serial.*; //The serial library is needed to communicate with the Arduino.
OpenCV opencv;  //Create an instance of the OpenCV library.
//Screen Size Parameters
int width = 320;
int height = 240;
// contrast/brightness values
int contrast_value    = 0;
int brightness_value  = 0;
Serial port; // The serial port
//Variables for keeping track of the current servo positions.
char servoTiltPosition = 90;
char servoPanPosition = 90;
//The pan/tilt servo ids for the Arduino serial command interface.
char tiltChannel = 0;
char panChannel = 1;
//These variables hold the x and y location for the middle of the detected face.
int midFaceY=0;
int midFaceX=0;
//The variables correspond to the middle of the screen, and will be compared to the midFace values
int midScreenY = (height/2);
int midScreenX = (width/2);
int midScreenWindow = 10;  //This is the acceptable 'error' for the center of the screen. 
//The degree of change that will be applied to the servo each time we update the position.
int stepSize=1;
void setup() {
  //Create a window for the sketch.
  size( width, height );
  opencv = new OpenCV( this );
  opencv.capture( width, height );                   // open video stream
  opencv.cascade( OpenCV.CASCADE_FRONTALFACE_ALT );  // load detection description, here-> front face detection : "haarcascade_frontalface_alt.xml"
  println(Serial.list()); // List COM-ports (Use this to figure out which port the Arduino is connected to)
  //select first com-port from the list (change the number in the [] if your sketch fails to connect to the Arduino)
  port = new Serial(this, Serial.list()[0], 115200);   //Baud rate is set to 57600 to match the Arduino baud rate.
  // print usage
  println( "Drag mouse on X-axis inside this sketch window to change contrast" );
  println( "Drag mouse on Y-axis inside this sketch window to change brightness" );
  //Send the initial pan/tilt angles to the Arduino to set the device up to look straight forward.
  port.write(tiltChannel);    //Send the Tilt Servo ID
  port.write(servoTiltPosition);  //Send the Tilt Position (currently 90 degrees)
  port.write(panChannel);         //Send the Pan Servo ID
  port.write(servoPanPosition);   //Send the Pan Position (currently 90 degrees)
public void stop() {
void draw() {
  // grab a new frame
  // and convert to gray;
  opencv.convert( GRAY );
  opencv.contrast( contrast_value );
  opencv.brightness( brightness_value );
  // proceed detection
  Rectangle[] faces = opencv.detect( 1.2, 2, OpenCV.HAAR_DO_CANNY_PRUNING, 40, 40 );
  // display the image
  image( opencv.image(), 0, 0 );
  // draw face area(s)
  for( int i=0; i<faces.length; i++ ) {
    rect( faces[i].x, faces[i].y, faces[i].width, faces[i].height );
  //Find out if any faces were detected.
  if(faces.length > 0){
    //If a face was found, find the midpoint of the first face in the frame.
    //NOTE: The .x and .y of the face rectangle corresponds to the upper left corner of the rectangle,
    //      so we manipulate these values to find the midpoint of the rectangle.
    midFaceY = faces[0].y + (faces[0].height/2);
    midFaceX = faces[0].x + (faces[0].width/2);
    //Find out if the Y component of the face is below the middle of the screen.
    if(midFaceY < (midScreenY - midScreenWindow)){
      if(servoTiltPosition >= 5)servoTiltPosition -= stepSize; //If it is below the middle of the screen, update the tilt position variable to lower the tilt servo.
    //Find out if the Y component of the face is above the middle of the screen.
    else if(midFaceY > (midScreenY + midScreenWindow)){
      if(servoTiltPosition <= 175)servoTiltPosition +=stepSize; //Update the tilt position variable to raise the tilt servo.
    //Find out if the X component of the face is to the left of the middle of the screen.
    if(midFaceX < (midScreenX - midScreenWindow)){
      if(servoPanPosition >= 5)servoPanPosition -= stepSize; //Update the pan position variable to move the servo to the left.
    //Find out if the X component of the face is to the right of the middle of the screen.
    else if(midFaceX > (midScreenX + midScreenWindow)){
      if(servoPanPosition <= 175)servoPanPosition +=stepSize; //Update the pan position variable to move the servo to the right.
  //Update the servo positions by sending the serial command to the Arduino.
  port.write(tiltChannel);      //Send the tilt servo ID
  port.write(servoTiltPosition); //Send the updated tilt position.
  port.write(panChannel);        //Send the Pan servo ID
  port.write(servoPanPosition);  //Send the updated pan position.
 * Changes contrast/brigthness values
void mouseDragged() {
  contrast_value   = (int) map( mouseX, 0, width, -128, 128 );
  brightness_value = (int) map( mouseY, 0, width, -128, 128 );
And this is the Arduino Code:
* SerialServoControl Sketch
* Written by Ryan Owens for SparkFun Electronics
* 7/15/11
* This sketch listens to serial commands and uses the data
* to set the position of two servos.
* Serial Command Structure: 2 bytes - [ID Byte][Servo Position byte]
* ID byte should be 0 or 1.
* Servo position should be a value between 0 and 180.
* Invalid commands are ignored
* The servo position is not error checked.
* Hardware Setup
* Servos should be connected to pins 2 and 3 of the Arduino.
* 9V DC Power supply is recommended as USB can't always handle powering two servos
#include <ServoControl.h>  //Used to control the Pan/Tilt Servos
//These are variables that hold the servo IDs.
char tiltChannel=0, panChannel=1;
#define servo_tilt 6
#define servo_pan 7
//These are the objects for each servo.
ServoShield servos;
int valtilt = 0;
int valpan = 0;
int tilt=0;
int pan=0;
//This is a character that will hold data from the Serial port.
char serialChar=0;
void setup(){
  servos.setbounds(servo_tilt, 500, 2400);
  servos.setbounds(servo_pan, 500, 2400);
  servos.setposition(servo_tilt, 1500);
  servos.setposition(servo_pan, 1500);
  Serial.begin(115200);  //Set up a serial connection for 57600 bps.
void loop(){
  valtilt = map(tilt, 180, 0, 500, 2400);
  valpan = map(pan, 180, 0, 500, 2400);
  while(Serial.available() <=0);  //Wait for a character on the serial port.
  serialChar =;     //Copy the character from the serial port to the variable
  if(serialChar == tiltChannel){  //Check to see if the character is the servo ID for the tilt servo
    while(Serial.available() <=0);  //Wait for the second command byte from the serial port.
    tilt =;
    servos.setposition(servo_tilt, valtilt);  //Set the tilt servo position to the value of the second command byte received on the serial port
  else if(serialChar == panChannel){ //Check to see if the initial serial character was the servo ID for the pan servo.
    while(Serial.available() <= 0);
    pan =;
      //Wait for the second command byte from the serial port.
    servos.setposition(servo_pan, valpan);   //Set the pan servo position to the value of the second command byte received from the serial port.
  //If the character is not the pan or tilt servo ID, it is ignored.
Hope it helps :))


GroG's picture

Thanks for that... It's good

Thanks for that...
It's good to have other input..
I have not tried it, but the code there does have something I was thinking of avoiding.

When it finds a target it increments but some stepSize (in this case stepSize=1)  .. so it moves + or - 1 depending on direction.  This does not account for the distance the target is from the center.  If the target is very close or near the edge of the field of view, the servos are given directions to move + or - 1.   I think it would be nice to estimate the amount - and jump to it in one command..

The code I've done so far is accessable through the bleeding edge update - right now its only the X axis - and its trying to move the servo in absolute location (versus incrementally) .  It needs more work, but its a start..

michael96_27's picture

Okay, I'll download now the

Okay, I'll download now the latest bleeding edge build.

Is the code your'e referring is the color tracking script?

GroG's picture

Yes,I checked it in with my


I checked it in with my config set - you'll have to switch it to yours...

this line to 
configType = 'GroG'  
changed to
configType = 'michael'



michael96_27's picture


The script's finally working GroG! 

I think we can now move on to adding the tilt servo in tracking the object :))

I'll post a vid or maybe a photo soon :)


GroG's picture

WhooHoo ! Awesome.. I'm

WhooHoo !

Awesome.. I'm painting stuff today, but tonight I should be able to update it ..

Right On ! (Post a video - I wanna see if your Servos are as noisy as mine)

GroG's picture

I added tilt, although didn't

I added tilt, although didn't test much...  I'll do more testing later..

Your servos may be inverted to mine so you may need to change the 

tilt.moveTo(90 + yAvg/sampleCount)  to
tilt.moveTo(90 - yAvg/sampleCount)

you might wan to experiment with the 

 if (sampleCount > 10):  and change 10 to a lower value if your servos & arduino can handle it....

Post some pictures or video when you get a chance..

I will probably get more of a controlled lighting place and see what colors my camera isolates best then do a video - after doing that I'll change to stepper motors & a stepper shield ... considering the problems I've had with servos :P

I think a stepper driven pan / tilt kit might be pretty smooth in comparison..  we'll see..

GroG's picture

Fixed a long time bug.. where

Fixed a long time bug.. where filters set programmatically would not populate the GUI - now when a filter is added or removed programmatically, the event is published, and in this case the GUI displays the filter's name.  Still have an outstanding bug where filter's data does not display, when it is set programmatically.. And the camera / capture info is not correct - when set in a program

GroG's picture

Evil Servos !

Experimenting with the new script...  I can see the OpenCV data overwelming my servos.  If the noisy servos get too many commands, and too many changes of direction, too quickly, the kill the communication between the computer and the Arduino...

Evil Servos !!!

michael96_27's picture

HAHAHA XD I hope the servos


I hope the servos will cooperate later :D

michael96_27's picture

Do you have now a script on

Do you have now a script on contolling servos with a joystick via MRL.

BTW, I'll try now the pan color tracking. Been busy this past days :))

GroG's picture

Here's some info on the

Here's some info on the Joystick Michael -