Update 2012.08.09 - High Density 3D Point Cloud in Java3D - Video Up

Same scene zoomed in

HSV - unknown scale colored depth over point cloud

High density 3D point cloud of floor, door, & wall.  Finally got some good results and I am beginning to understand the translation of Kinect data to Java3D view.  Additionally, this was done from saved Kinect data (ie. anyone now can look at a 3 dimensional view of my office with MRL & the data file)...  Next?  I want to color code again with HSV over the full range of the Kinect - with a scale which show's the real distance values in centimeters.  E.g. red = 300-400 cm orange = 400-500 cm etc.

Update 2012.08.07

Well lets start scaling !  In Java3D the world is represented in a cartesian with 3 dimensions in float units.  Which means we can make it "any length value".  
For convienence, I have decided to make:

                                   1 virtual Java3D unit =  1 meter

The solid point cloud wall has 640 x 480 points - this becomes 6.40 m x 4.80 m.
Now, how to find the distance of the camera from the center of the point cloud wall ?
Trig !

Solve right triangle :

A = 57.8/2 = 28.9 degrees
a = 320
B = 90 - 28.9 = 61.1 degrees

We are looking for b - distance kinect away from center of plane.

b = a/tan(A) = 320 / tan (28.9)  = 320 / 0.545217 = 586.92 cm

The density is impressive (10X from previous example), initializing all the points of the point cloud with 0 z depth gives me a solid wall.  Unfortunately, I've botched something up in the translation from kinect data to z depth so I can't show a 3d point cloud at the moment.  Before I fix that, I want to work on the translation such that I know its scale, and all the coordinates and projection is known and accurate.  Right now I'm starting to place a representation of the kinect into the model.  After that is done, I'll adjust and caculate the distance to the depth plane using the kinects maximum field of view angle.  This will help to get a visual representation of the location and heading of the kinect camera.

Update 2012.08.06

Artifact gone, but found a lot more work I need to do....  Going to see what MAX res does ....

Whoa ! I found out how to do tranparencies in Java3D - tried it out and everything got all wonky..  Digging deeper into the original PointCloud example I'm amazed that there is such a large amount of data which is NOT being used..  Or rather a Huge amount of data which is not being displayed.  The author chose not to do this as an optimization.  This ties into the differences of the PointCloud generators in Processing vs this particular one.  The Processing demo's looked very detailed, this one "looked" low res.  So, I've found the root cause of it, but I think what would be optimum, is the ability to dynamically change resolutions.. No?

First step is ripping out a bunch of code and have it display ALL the data - preferably at a correct scale with Trig negating the warping effect.  Once that is done, I'll incorporate dynamic resolution and other optimizations like a Trig lookup table.

Figured out what the "artifact" is - it's simply all the points with 0 depth. A plane is drawn each frame and the  Z-depth is calculated the points are "pushed" to the appropriate depth.  I think this can be fixed by determining if the Z-depth is equal to min or max , if so - draw the point as transparent.

Got "Playback" working... now to straighten this wall & floor.

Update 2012.08.04

At this point I want to get some constants for the Kinect :
So many good resources here's one:

• The depth image on the Kinect has a field of view of 57.8
• The minimum range for the Kinect is about 0.6m and the maximum range is somewhere between 4-5m
• Currently the depth map is mirrored on the X axis
• Need to get Z data value and determine how that corresponds to meters - according to this excellent resource - its depthInMeters = 1.0 / (rawDepth * -0.0030711016 + 3.3309495161); 
  Wow, Daniel Shiffman got some great results with Processing.  I'll look at his examples in detail and try to get them to work without Processing and using Java3D.

3D Rainbow monster
Yay - got the simple data to work - so now Playback is at least theoretically possible (not implemented yet). As I dived into the code I could see the original point cloud example used a ColorUtils jar to do HSV to RGB conversion.  The Java JRE can do this conversion, so I'm using the native version now (although it this picture I have the cyclic range too short and you get several cycles of roygbiv).  I found some potential optimizations too. - Non Trig distortions & 0 range artifacts still exist - will work on those next.

Missing Trigonometry...

Looking throught the example code I did not find any trig references to get the depth data into cartesian point form of the display.  Here is confirmation that it's needed and lacking.  Above I aimed the kinect at an angle against a straight wall - you can see that it curves.  At the moment I'm dealing with trying to convert the data and display to use primitive data structures instead of the NIO buffers.  This will allow the "playback" of a capture (The NIO Buffers are not serializable).  It should help me understand how to use Java3D too. I'm pretty impresed with the display, you can make it fullscreen without reducing the framerate.  In the process I inted to fix the Trig issue and the artifacts - in this frame they are red.

Update 2012.08.03 

Adding functionality - currently "capture/stop capture" & "record/stop recording"  work.
Record is a nice addition since we can then save the kinect data and use it without the actual sensor, so SLAM testing can be done without the kinect hooked up.  I'm converting the nio ShortBuffer to a Java primitive short buffer so other services won't need to bring in the nio dependency to work on the depth data.
The current code I got from Dr. Andrew Davison's example will have to be modified to accept primitives.  Then I'll need to find out the source of the artifact which is being created here.

Update 2012.08.01 It's been Borg'd in !

Yep, welcome the new OpenNI Service in MRL !  The example code was well designed.  It had a sensor data layer (the Service) and a display layer (which I put in the GUIService)..   So, as with all Services, the display can be on a different computer and the Kinect can stream the sensor data across the internet if necessary.   Andrew Davison's example has some bugs I need to clean up and I'll bring in more functionality after looking at the Processing examples

Update 2012.07.31

Progress - I got Dr. Andrew Davison's example of a Point Cloud using OpenNI on Java running !
He uses Sun's Java3D, which makes it easy to mouse around the 3D view.  

But, in comparison to the Processing demo's this look inaccurate...    The depth is inaccurately represented as quantum jumps between plains.  While the Processing examples show surface contours of the objects.

Next Steps : 

• Begin organizing the repo - so that MRL can easily support OpenNI on all the various platforms
• Begin looking into the Processing examples for math and display differences
• Start the OpenNI service and control panel

The depth is definately "squished" in the representational display.  Maybe it was just a choice to get everything on the grid.  My head is flat, but not that flat. Going to dive in and look for the translation formula from Kinect data to Cartesian.

A little better, I'm not such a flat head.  This was done by tweaking a few constants in the example.  I'm guessing the Processing examples use a different representation of displaying the "points" of the point cloud.  If I remember correctly at least one of the demos made bounding boxes which reached its neighbor.  This would give a view of more "solid" objects.  I can see a red plane too.  I think this might be "residue on the minimum boundry".  The Kinect has a minimum limit of 18" or so.  This should probably be filtered out.

(Starting to look at OpenNI - ran some of the samples of simple-openni - nifty hand tracking... Wheee !)

(NiSimpleSkeleton & NiUserTracker64 crashes... but UserTracker.net.exe works !)

Well today starts my foray into OpenNI.  OpenNI is an non-profit organization which produced drivers and middleware which supports gesture recognition.  Primasense (creators of the Kinect) is one of the main members of OpenNI.

I figured I'd tackle the windows platform first, since I have a tendency to use 64 bit Windows 7 (work demands it).  There is a download http://code.google.com/p/simple-openni/ for windows 64.  It's a zip which contains 4 seperate installers. There might be another source, I'm not sure about the simple-openni vs OpenNI.  Then of course there is OpenKinect.  Yikes, so much software ... so little time.

This article succinctly explains the differences between OpenKinect & OpenNI.

Time to jump in !

After installing the 4 installers from the "google code" site, I started looking for examples to run.  Some were console output which gave a distance value for the middle pixel.  I could see possiblities for this since I'm interested in a Point Cloud.  At this time, making a point cloud from a Kinect is (almost) old news.  I think its been done from both OpenKinect & OpenNI (at least utilizing their drivers). 

I'm specifically looking for Point Cloud info for Java.  Java implementations often have to have shim adapters to get to data from platform specific drivers.  I've begun googling and below are a few possible resources.

• Nifty Demo of a Point Cloud from Kinect - showing different point resolutions and other effects
• Another Point Cloud demo
• Processing display (Java) + Kinect - Good Point Cloud
• Daniel Shiffman - Point Cloud on OSX (only)
• Java 3D + Kinect
• Kinect Depth vs Actual Distance

One of the goals into this Kinect exploration is to implement a SLAM implementation using the point cloud and camera from the Kinect.  That's what we all want right !  And gesture recognition too, so we can tell our robots to go stand in the corner when they've been bad.

Update 2012.07.30
It does not seem like the simple-openni is the most stable set of software.  Several of the examples crash.  Even the viewers have glitches in the camera feed., and crash.  This was suprising as OpenKinect appears more stable.  Weird no?  The drivers made by hackers are more stable, than the community effort of the manufactures of the Kinect ?  What the what?

Right now I have OpenKinect installed on Ubuntu 32 bit laptop working well & OpenNI on a 64 bit Windows 7 machine.  Still more testing to do...

Holy JNI Wrappers Batman... There appears to be a Jar which came with OpenNI..  They've thought of Everything !   But wait, there's more - someones done an implementation of a point cloud using said distributed jar ! - http://fivedots.coe.psu.ac.th/~ad/jg/nui14/

Hmmm... his "cloud" seems very flat - maybe since its at a greater distance than the Processing videos?  Hmmm... dunno.. will need to explore more... Definately feeling the need for a new Service !

Found an OpenNI programer's guide - http://openni.org/docs2/ProgrammerGuide.html ... hmmm looking for JavaDocs .... Java related shtuff - http://openni.org/docs2/tutorial/smpl_simple_view_java.html

No JavaDocs though .. not that I'm complaining.  This is usually WAY more info than I usually get.  I'm thinking this might not be so hard....    but after the PointCloud in MRL comes SLAM (more work !)