Ro-Bot-X and I are working together even though we are 2500 miles away !
Hey Look... I can see Ro-Bot-X's living room through the eye of the bot he's currently working on ! (Hopefully he won't mind me posting it :) )
It's a Chumby 1 and were getting MyRobotLab to run on it.
It was a challenge and learning experience getting the OpenCV running on the Chumby. It required cross compiling binaries on a Linux X86 PC --to--> ARM 9 with a tool chain.
So far, we have MRL running on it with OpenCV and Serial which means potentially it can us Ro-Bot-X's uBotuino to control Motors, Servos, read Sensors, etc.
I'm making two new Services in MRL. A Chumby service which will interface with all of the hardware a Chumby has to offer and simplify/standardize it so other Services can use it. And a ChumbyBot service which will contain the behaviors and personality of the little bot.
His board handles Motors a little differently than the 2 way's I am familiar with, so well figure that out and put it in the Motor service with configuration.
I'm beginning to like the little Chumby. It runs Linux and has support for USB, I2C, Video, Microphone, Stereo Speakers, Wifi, Touch-Screen, FM Radio, even an Accelerometer.. and for just $90 on Amazon or lower on ebay
We are just starting to connect the wires and pipes together, but I think the hardest part was the OpenCV support...
What current works
- Remote communication from 2500 miles away attached to my MRL instance
- OpenCV webcam and remote control of vision filters (facedetection, color isolation, LKOptical tracking, etc)
- Web control through an MRL applet
- Remote GamePad control
- TextToSpeech with sexy Audrey voice
Things to test
- MRL/Sphinx4 Speech recognition
- Audio Recording and PlayBack through MRL
- HTTP getting info on the intertubes - jokes, daily news, shoutbox data etc.
- Fancy Sound generation (Drums, Sax, Guitar etc through MRL & JFugue)
- Play Chess? (Thanks Patrick :D)
- Servos
Things to develop
- Encapsulate Chumby's hardware in a easy to use Service
- Behaviors and Personality for ChumbyBot
- Augment Motor to handle uBotuino's method of control
We will make sure all the lower level stuff works well (Motors, Servos, Sensors, etc) then I believe Ro-Bot-X was interested in grabbing colored objects around the house (cleaning? dunno)
Wheeeee !
The frame rate of the camera is pretty lousy, but he said he got it for $5.. To reduce the lag as its shipped 2500 miles you can drop the color.
Here I tried isolating the red square out of his carpet... Unfortunately, he has not mounted the thing on a base or camera on a servo.. but that will come shortly...
RoboScope - monitors IR Sharp sensor with 2500 mile test leads !
I was able to log the data coming from the IR Sensor in the SensorMonitor Service .. Here are a few snapshots. It works as a great diagnostic tool, but the real nice part will be the ability to set "Alerts" where a sensor reaches a certain pre-defined (or even dynamic) level and fires off an event. For example, 50 cm = Stop and look for open area.
The IR Sensor was on a servo. The servo swings back and forth and in turn the range changes. As the range changes I can see the value of the IR sensor coming back as a graph. The dip is the change of range since part of the time, the IR sensor is aimed at a nearby wall. The SensorMonitor can graph (and soon Alert) on multiple sensors at once.
The Detailed Nerdy Part:
The first step in this process is to get Java running on Chumby.
A great wiki is provided here http://wiki.chumby.com/index.php/Java
However, It contains references to older versions of JamVM and GNU Classpath. I do not believe that these versions support all the JNI necessary to communicate successfully to the hardware.
Our objective then becomes install the "latest" JamVM and GNU Classpath
Some other lessons I have learned through this process :
- Sun embedded JVM for Arm (headless) - worked great and it was an incredibly easy install, but it is not open source and THE BIG strike against it - is there was no javac (compiler) which would make the compilation of rxtxSerial difficult !
- OpenJDK - seems like a great project - it was originally just targeting PCs - it is huge and cross compiling it for the Chumby seemed like a exercise in extreme torture (please prove me wrong!)
- Don't compile the wrong versions of JamVM and GNU Classpath - they are different projects, so only certain versions of each are compatible - the target we are going for is JamVM 1.5.3 and GNU Classpath 0.98
- Don't compile install JamVM first ! - this only causes problems in the long run - JamVM will not run without a valid GNU Classpath already in place - yes I burned time with this.
- Don't cross compile - another exercise in torture (please prove me wrong!)
- Use JamVM 1.5.3 - at this time JamVM 1.5.4 is available, however, it will not compile on the Chumby - because it needs 40 megs of memory at least to do it - if your an expert in cross compiling you can get around this. I am not an expert in cross compiling - so I'm sticking to 1.5.3
- Don't attempt to bootstrap or compile GNU Classpath on the Chumby itself
The basic strategy will to get an ARM tool chain installed. This will allow us to compile code which runs on the ARM from a regular PC. Thankfully, the very helpful Chumby wiki has such information in step by step instructions (http://wiki.chumby.com/index.php/GNU_Toolchain)
RXTXComm
A very popular way to send serial information through Java uses the RXTXComm.jar from http://rxtx.qbang.org . MyRobotLab's Arduino Service uses this code to talk to Arduinos, and theoretically any microcontroller attached via USB-FTDI or serial port. Since Java must interface directly with the hardware in this situation, it will be necessary to build Native code with the cross compiler toolchain we just installed.
Unfortunately, there is a bug in the configure/build at the moment for rxtxcomm.
cd rxtx
wget http://rxtx.qbang.org/pub/rxtx/rxtx-2.1-7r2.zip
unzip rxtx-2.1-7r2.zip
./configure --target=arm-linux --host=arm-linux --build=i386-linux
make
After all that labor I have successfully built a librxtxSerial.so for Chumby - which means Java programs can now use it to communicate over Chumby's serial port or USB-FDTI to a microcontroller. There is also a librxtx - Parallel I2C Raw and RS485 libraries. I have attached them to this node.
OpenCV + JavaCV + JavaCPP + MRL
This was rather "educational", and successful in the end.
First a toolchain was necessary to compile all the parts on a Linux machine running a X86 processor (my home box). Chumby has a convenient download for this here. It's "real" name is arm2008q3.
OpenCV was challenging to compile, in that the cmake build wants to get stuff from /usr/include (not a good idea). I have had mixed feeling regarding cmake... but at this point I am beginning to like the gui manager. After symbolically linking a new /usr/include and taking care of all the other dependencies and compiler specifics, I built it with very few options. No GTK or GSTREAMER as I don't believe those are on the target Chumby.
JavaCV is a great interface and MRL uses it to get to OpenCV. JavaCPP is the method which Samuel Audet uses to access the natively compiled OpenCV. Samuel Audet manages both JavaCV & JavaCPP. Although JavaCV supports a wide range of platforms, it does not support Chumby hardware. With some help from Mr. Audet I compiled the necessary interface binaries. I gave him a copy so that JavaCV will have another platform it can successfully talk too. Isn't open source great?
The OpenCV binaries and JavaCV/JavaCPP are attached to this post as well.
JamVM 1.5.4
Ro-Bot-X Writes
Cool man! Thanks for your efforts!!!
I guess my daughter moved the uBotino board out of the camera range... But the carpet looks fine. BTW, what time was this picture taken? I'm just trying to figure if it was before or after Roomba cleand the place... :D
Anyway, I'll install the camera on a pan servo (no tilt at the moment) and a Sharp sensor on top of it, mounted on the test robot. Chumby is powered by a wall wart, if I connect it to a battery, it will deplete it before I'll be back from work. Chumby powers the USB hub and the hub powers the uBotino (and the Sharp sensor). But for the servo, I need to switch the battery on... So, let me know when you want to do tests and I'll turn it on for you.
You're talking about my different method of controlling motors. This does not affect MRL or Chumby. Only the Arduino firmware. There we need to make the custom config, perhaps selectable by a variable that MRL can flip True. Also, the motors speed will be a byte instead of a int, but that can be also dealt with inside the SetSpeedLeft and SetSpeedRight functions. Going further, after we work with encoders, the motor functions can accelerate to the SetSpeed, then the Stop command will decelerate until a full stop is achieved. PID control can be used for this. But let's go small steps first.
Cheers!
Webcams used
Hello Ro-Bot-X,
Could you say which webcams you used? Some Chumbarian were asking....
If I remember correctly you had one which worked and had very slow frame rate ...
And one which did not work and had very fast frame rate ...
I would think that the OpenCV list of supported cameras would work... I have used Logitec webcams in the past (although with PCs not Chumbys)
Logitec C600 and they have worked great...
The webcam with the low
The webcam with the low refresha rate is similar with this one:
http://www.dealextreme.com/p/compact-usb-pc-webcam-300k-pixel-25948
The webcam that needed a driver is similar with this one, but it is 1.3MP:
http://www.dealextreme.com/p/usb-pc-webcam-300k-pixel-with-6-led-13
I would get a camera that has over 1MP real, not software. Heck, I could get this one and replace the mic with a ultrasonic sensor, looks cool, like Wall-e's eyes:
http://www.dealextreme.com/p/compact-1-3mp-pc-usb-webcam-with-built-in-microphone-black-51874
I'm done with the move, things get back rolling, I'll try again to revive my Chumby. As soon as I'm done I'll let you know!
Cheers!
How to Cross-Compile OpenCV for Chumby on Linux x86
This is a breif descriptio of how to compile the OpenCV sources for a Chumby on a Linux x86 box.
The specific versions use for the description:
Chumby : Chumby 1
OpenCV : OpenCV-2.3.0
Linux box : Fedora 13 running on an x86
CMake : cmake & cmake-gui
On the Linux box
# yum -y install cmake cmake-gui
here are the cmake options I chose
Some additional configuration will be needed. The Chumby does not have jpeg, tiff, png or python support so I removed these. Strangely, the build will not complete because of some linker options where are not supplied.
These extra linker flags must be added to a variable CMAKE_SHARED_LINKER_FLAGS -lrt -lpthread . The list of useful cmake variables is here http://cmake.org/Wiki/CMake_Useful_VariablesThe variable CMAKE_SHARED_LINKER_FLAGS is already defined but only viewable in "Advanced" mode. The flag entries -lrt -lpthread need to be added.
Here is a complete list of changes
Commandline options:
-DBUILD_EXAMPLES:BOOL="1" -DCMAKE_VERBOSE:BOOL="0" -DOPENCV_BUILD_3RDPARTY_LIBS:BOOL="1" -DWITH_TIFF:BOOL="0" -Dlrt:STRING="-lrt" -DBUILD_NEW_PYTHON_SUPPORT:BOOL="0" -Dpthread:STRING="-lpthread" -DBUILD_DOCS:BOOL="0" -DBUILD_TESTS:BOOL="0" -DWITH_XINE:BOOL="0" -DWITH_JPEG:BOOL="0" -DWITH_PNG:BOOL="0"
Cache file:
BUILD_EXAMPLES:BOOL=1
CMAKE_VERBOSE:BOOL=0
OPENCV_BUILD_3RDPARTY_LIBS:BOOL=1
WITH_TIFF:BOOL=0
lrt:STRING=-lrt
BUILD_NEW_PYTHON_SUPPORT:BOOL=0
pthread:STRING=-lpthread
BUILD_DOCS:BOOL=0
BUILD_TESTS:BOOL=0
WITH_XINE:BOOL=0
WITH_JPEG:BOOL=0
WITH_PNG:BOOL=0
"cc1plus: warning: include location "/usr/include" is unsafe for cross-compilation"
To compile binaries with improper headers at best would create compiling havoc, at worse nasty runtime errors. So, I moved the include and linked the appropriate include so that the ARM includes are now under /usr/include
# mv include include_gcc
$ make clean
$ make
[100%] Building CXX object samples/gpu/CMakeFiles/example_gpu_surf_keypoint_matcher.dir/surf_keypoint_matcher.objLinking CXX executable ../../bin/surf_keypoint_matcher_gpuYay! But wait there's more !
which means opencv started using floating point in its flann library, and the current gcc 4.3.2 does not have this function.
It's defined as a template but not used, I know how to solve that - comment it out, and pray to the linker :)
//inline long double abs<long double>(long double x) { return fabsl(x); }
opencv.2.3.1.chumby.tgz
libopencv_contrib.so
libopencv_core.so
libopencv_features2d.so
libopencv_flann.so
libopencv_gpu.so
libopencv_highgui.so
libopencv_imgproc.so
libopencv_legacy.so
libopencv_ml.so
libopencv_objdetect.so
libopencv_video.so
The G++ command from hell ! Location of your dependencies will vary :P
Don't forget to move include_gcc back into place !
On the Chumby
Try to keep things "clean" by looking for old versions of your new libraries.
# find / -name libopencv*
Get the latest release of MRL
# wget http://myrobotlab.googlecode.com/files/myrobotlab-0014.zip
go into the myrobotlab-0014/bin directory and delete everything ! - yep all of it is for windows or Linux x86 - it won't work on a Chumby
# cd myrobotlab-0004/bin
# rm *
Now get the Chumby OpenCV 2.3.1 binaries
# wget http://myrobotlab.googlecode.com/files/opencv.2.3.1.chumby.tgz
# tar zxvf opencv.2.3.1.chumby.tgz
Get the interface so's
# wget http://myrobotlab.googlecode.com/files/libjniopencv.chumby.3.2.1.tgz
# tar zxvf libjniopencv.chumby.3.2.1.tgz
Got this error after exporting LD_LIBRARY_PATH and PATH with everything
libjniopencv_core.so: ../../lib/libopencv_core.so: cannot open shared object file: No such file or directory
if you look at it carefully , it's looking for it in a specific ../../lib location
# cd myrobotlab-0014/bin
# mkdir arm7
# mv libjni* arm7
# mv libopencv_* to myrobotlab-0014/lib
have nearly everything except the kitchen sink exported
export LD_LIBRARY_PATH='/mnt/usb-E0FD-1813/myrobotlab-0014/lib:/mnt/usb-E0FD-1813/myrobotlab-0014/bin/arm7:/mnt/usb/myrobotlab-0014/bin:/lib:/mnt/storage/lib:/mnt/storage/local/lib:/lib:/usr/lib:/usr/local/lib:/mnt/storage/lib:/mnt/storage/local/lib'
Change of error ! - UnsatisfiedLinkError: no jniopencv_core in java.library.path
So the ../../lib reference is relative to where the cwd is of program execution... NOT relative to the libjnicore so
# cd myrobotlab-0014
# mkdir ../../lib
# cp lib/libopencv*.so ../../lib
Now the next error
ERROR org.myrobotlab.service.OpenCV - using com.googlecode.javacv.OpenCVFrameGrabber
7334 [OpenCV_videoProcessor] ERROR org.myrobotlab.service.OpenCV - ------
java.lang.Exception: cvCreateCameraCapture() Error: Could not create camera capture.
at com.googlecode.javacv.OpenCVFrameGrabber.start(OpenCVFrameGrabber.java:102)
at org.myrobotlab.service.OpenCV$VideoProcess.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
Unsupported Camera?
Seems complicated... glad it
Seems complicated... glad it works!
UnsatisfiedLinkError: no jniopencv_core in java.library.path
Tried running ldconfig - got error of read only file system
The generated cross-compiled libjniopencv_core.so library is Explicitly looking for libopencv_core.so.2.2
I thought Samuel had a filtering policy - I'll ask him about it and strategies of loading.
Exception in thread "OpenCV_videoProcessor" java.lang.UnsatisfiedLinkError: /mnt/usb-E0FD-1813/myrobotlab-0012/bin/libjniopencv_imgproc.so: libopencv_imgproc.so.2.2: cannot open shared object file: No such file or directory
Remote Command Line
# chumby:/mnt/usb-E0FD-1813/myrobotlab-0012# /mnt/usb-E0FD-1813/java/ejre1.6.0_25/bin/java -classpath ":myrobotlab.jar:./lib/*" -Djava.library.path=./bin org.myrobotlab.service.Invoker -service OpenCV opencv RemoteAdapter remote
Grrr - still get cvCreateCameraCapture() Error: Could not create
I am now under the suspicion that the cmake-gui will mess things up.
With OpenCV 2.3.1 if you specify cross-compile the option WITH_V4L disappears !
I put it in the cmake-gui manually, but I have a feeling that it's being ignored.
Sooooo... (running out of options here) I'll create native Makefiles with cmake-gui and select the WITH_V4L but build in an environment that is configured to cross-compile, with gcc linked to the arm7-gcc
Another option is to use the FFMPEG frame grabber, I cross-compiled ffmpeg without any errors.. will be moving it up to the chumby shortly. Potentially, this might be faster anyway.....
This command should test that ffmpeg is working on the chumby correctly
# ffmpeg -f oss -f video4linux2 -i /dev/video0 out.mpeg
The libjniopencv_* files appear to be looking for their corresponding libopencv in a hardcoded relative path location ! I'm guessing javacv unloads the jars into a tmp directory and expects to find the libopencv in /lib (maybe a few other locations?)
I have moved all the libopencv_* into /lib and the libjniopencv_* into myrobotlab-0014/bin this should line up things.
No more unsatisfied link errors YaY !
Partial Success ! (FFMPEG)
This command worked - ffmpeg was streaming video - just didn't have anything to recieve it !
chumby:/mnt/usb-E0FD-1813# ./ffmpeg -f video4linux2 -s qvga -pix_fmt yuyv422 -i /dev/video0 -vcodec mpeg
4 -f rtp -r 15 rtp://192.168.1.11:1234
I'm a noob with ffmpeg - and got the command line from this chumby forum -
http://forum.chumby.com/viewtopic.php?id=4456
Woohoo !
Up again ....
This nearly kiled me, I still have to collect all the pieces and make more notes, check stuff in....
Part of the issue (took me a while to realize it) OpenCV 2.3.1 has decided NOT to support Video For Linux (V4L) and is supporting V4L2 only, which the chumby does not have ...
5 Mega-pixel camera?
This is what your chumby looks like without MRL running
The last issues were resolved.
It turns out that OpenCV 2.3.1 does not support Video For Linux (V4L) instead they support V4L2 (ish). This was a problem as the Chumby uses V4L.
The module responsible for camera capturing is highgui. The file for capturing V4L cameras is cap_v4l.cpp. Within it some #defines might need to be changed. To get the file to actually be included in the build CMakeLists.txt in modules/highgui needs to be changed.
Now I'm getting out of memory errors
Wondering if the upgrade is using more memory :P
Ro-Bot-X tried a 5Mbit camera - and it would not start. Now the old/slow camera will run out of memory (Grrr.)
Possible solutions:
1. Use the ffmpeg frame grabber - I've already cross-compiled ffmpeg and I have heard it is supposed to be faster than OpenCV's frame grabber.
2. Run headless - without streaming the images - not easy to debug or remotely control - but a possibility which will save memory.