Kinect linux kernel driver

It looks like Santa came early this year.

KernelLabs sponsored me for a Kinect sensor device unit so I can experiment writing a Linux kernel driver for it.

So in the next weeks I'll be working on a gspca driver for this device. For now the code is going to be hosted in the gspca_kinect repository, as an out of kernel module, this will ease compile/test cycle; if it comes out that changes are needed to gspca itself, then I'll decide whether hosting these changes as patches, or switching to a kernel clone repository; suggestions will be welcome. The driver is working already, but currently you can just use the sensor as a normal WebCam.

About the Kinect

The Kinect sensor is a device used by Microsoft for its Kinect project, which is a system for controller-less Human-Computer interaction firstly targeted for Xbox 360, but rumors say MS could be integrating this technology into its future operating systems; the hardware is only part of the system of course, the software does most of the interpretation work figuring out how many users there are and assigning a meaning to their actions.

Great part of the Kinect communication protocol has been already figured out, thanks to the adafruit people who dumped the communication stream with the Xbox using an usb bus analyzer, and to Héctor Martín Cantero (marcan) who started analyzing the trace for the libfreenect driver. Marcan is also the guy who won the adafruit bounty for a kinect opensource driver, and the one behind AsbestOS. All these efforts have flowed into the OpenKinect project, wisely guided by Joshua Blake.

For an overview about the hardware you can see the IFixit teardown, the part I am interested in for now is the integration of the video capturing sub-device which provides what is usually called RGBD data (Red, Green, Blue, Depth), the Depth component gives spatial information about the captured scene, easing out the process of image segmentation. Just to give an idea, background subtraction is one step of the segmentation process, and can be achieved quite robustly with techniques like Depth keying (the spatial version of Chroma keying). But I am really not an expert in this area.

In the Kinect device, RGBD data is captured from two distinct sensors: a regular RGB sensor and a monochrome sensor which, with the aid of a IR structured light, captures what is finally exposed as a depth map; so what we have is basically a Structured-light 3D scanner.

Kernel driver development

I don't know yet if the gspca driver is going to define two video devices, one for each sensor, or if a combined format would be preferred, I am asking for advice from others more expert in this field.

This is the plan:

  • Stabilize the first version of the driver which does device registration/initialization and allows capturing the RGB data with fixed resolution and fixed framerate using a video device node. The RGB data comes as a raw Bayer pattern, precisely in the V4L2_PIX_FMT_SGRBG8 pixelformat.
  • Add support for capturing the Depth map data somehow. Each depth value has 11-bit precision, and the data is transferred as a packed array, I still don't know how such format should be presented to userspace.

Follow the discussion on the OpenKinect mailing list or on linux-media.

Thinking outside the (X)box

Ah, don't forget to check out all the fancy stuff people are doing with this toy.


CommentsSyndicate content

Great weekend project if you

Dee's picture

Great weekend project if you ask me. Considering the Kinect was one of the fastest selling accessories ever really separates it from the pack as well. It will really be interesting to watch what the community does once the SDK is released.

I have a feeling that the SDK will really propel the Kinect Hacks community and the commercial aspect of its use. We all heard about the Roomba hack, now check out hte latest from them. AVA ready Robot, and thats really a 2.0 level technology.

Thanks for spreading the word! The Open Kinect community should really be thanked as well.

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account associated with the e-mail address you provide, it will be used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

More information about formatting options

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Z
3
3
v
G
G
Enter the code without spaces.