ao2.it
Antonio Ospite, theorist attacks

It looks like Santa came early this year.

KernelLabs sponsored me for a Kinect sensor device unit so I can experiment writing a Linux kernel driver for it.

So in the next weeks I'll be working on a gspca driver for this device. For now the code is going to be hosted in the gspca_kinect repository, as an out of kernel module, this will ease compile/test cycle; if it comes out that changes are needed to gspca itself, then I'll decide whether hosting these changes as patches, or switching to a kernel clone repository; suggestions will be welcome. The driver is working already, but currently you can just use the sensor as a normal WebCam.

About the Kinect

The Kinect sensor is a device used by Microsoft for its Kinect project, which is a system for controller-less Human-Computer interaction firstly targeted for Xbox 360, but rumors say MS could be integrating this technology into its future operating systems; the hardware is only part of the system of course, the software does most of the interpretation work figuring out how many users there are and assigning a meaning to their actions.

Great part of the Kinect communication protocol has been already figured out, thanks to the adafruit people who dumped the communication stream with the Xbox using an usb bus analyzer, and to Héctor Martín Cantero (marcan) who started analyzing the trace for the libfreenect driver. Marcan is also the guy who won the adafruit bounty for a kinect opensource driver, and the one behind AsbestOS. All these efforts have flowed into the OpenKinect project, wisely guided by Joshua Blake.

For an overview about the hardware you can see the IFixit teardown, the part I am interested in for now is the integration of the video capturing sub-device which provides what is usually called RGBD data (Red, Green, Blue, Depth), the Depth component gives spatial information about the captured scene, easing out the process of image segmentation. Just to give an idea, background subtraction is one step of the segmentation process, and can be achieved quite robustly with techniques like Depth keying (the spatial version of Chroma keying). But I am really not an expert in this area.

In the Kinect device, RGBD data is captured from two distinct sensors: a regular RGB sensor and a monochrome sensor which, with the aid of a IR structured light, captures what is finally exposed as a depth map; so what we have is basically a Structured-light 3D scanner.

Kernel driver development

I don't know yet if the gspca driver is going to define two video devices, one for each sensor, or if a combined format would be preferred, I am asking for advice from others more expert in this field.

This is the plan:

• Stabilize the first version of the driver which does device registration/initialization and allows capturing the RGB data with fixed resolution and fixed framerate using a video device node. The RGB data comes as a raw Bayer pattern, precisely in the V4L2_PIX_FMT_SGRBG8 pixelformat.
• Add support for capturing the Depth map data somehow. Each depth value has 11-bit precision, and the data is transferred as a packed array, I still don't know how such format should be presented to userspace.

Follow the discussion on the OpenKinect mailing list or on linux-media.

Thinking outside the (X)box

Ah, don't forget to check out all the fancy stuff people are doing with this toy.