Fully Articulated 3D Skeletal Hand Tracking

Post Reply
melax
Posts: 42
Joined: Mon Apr 24, 2006 4:55 pm

Fully Articulated 3D Skeletal Hand Tracking

Post by melax »

I've posted a couple of youtube videos showing markerless hand and finger tracking based on depth sensor data. Here's the link:
http://www.youtube.com/watch?v=8ctQm1VOC6M
There are a couple of others on the same youtube channel. The tech provides the 6dof pose of all 17 bones in the hand (palm, wrist and 3 per finger).

Most of the hand/finger tracking solutions you'll find out there are based on 2D computer vision (image-centric) approaches that may require the palm to be directly facing the camera so it can look for blob center and extents (palm and extended fingers). We instead use a fully general 3D model tracking approach. (Its different than the particle swarm method used by the FORTH reseachers.) The goal is to provide dataglove output without having to wear a dataglove. Robustness and accuracy isn't yet perfect, and there are limitations based on depth sensor quality.

So you might be wondering why this would be relevant for the Physics forum. IMHO the physics and simulation crowd (you guys) are an audience that might be interested in the advancement of such technologies since you are the developers having the right skills to understand and potentially use skeletal pose as input. For example, we've been doing some initial experiments that use the pose data to drive a powered rag doll based on the relative joint angles. This virtual hand then can interact with objects in a physics scene. Imagine applications being development just by having an artist create content (physically rigged digital machines), and then, put the assets into an environment the user can reach into - no programming required. Admittedly, that's a fairly lofty vision as things can be a bit clumsy today: no tactile feedback, the hand can travel out of the field of view of camera, fingers may become occluded thus making it impossible to fully squeeze that object you are holding when the back of your hand happens to be facing the camera, etc.

This stuff got shown at CES, but there is no news on product or availability for any of this (or any other) full 3D tracking yet. If there's actual news, i'll follow up. Anyways, I've been having fun working on this interesting new frontier of interaction and (now that its public) I felt it was time to share. :D

Cheers.

-stan
RBD
Posts: 141
Joined: Tue Sep 16, 2008 11:31 am

Re: Fully Articulated 3D Skeletal Hand Tracking

Post by RBD »

Nice! Does look like fun just to interact with rigid bodies in a virtual physics world playground. Mostly I can picture using your hand as a proxy to control other articulated machines or objects.

BTW, nice music Mr. Melax.
http://www.youtube.com/watch?v=EuPfrAOqjo8
http://www.youtube.com/watch?v=7LzLQzPE8qs
melax
Posts: 42
Joined: Mon Apr 24, 2006 4:55 pm

Re: Fully Articulated 3D Skeletal Hand Tracking

Post by melax »

Here's a follow-up gameplay concept video I just made showing how hand tracking might be used in a 3D environment:
http://www.youtube.com/watch?v=sAcTshfZCU8

To interact with stuff it uses a typical powered-ragdoll technique, which i'm sure everybody here is quite familiar with.

-stan
RBD
Posts: 141
Joined: Tue Sep 16, 2008 11:31 am

Re: Fully Articulated 3D Skeletal Hand Tracking

Post by RBD »

:( Videos are now marked as private? :(
melax
Posts: 42
Joined: Mon Apr 24, 2006 4:55 pm

Re: Fully Articulated 3D Skeletal Hand Tracking

Post by melax »

I'm extremely sorry about that, we were asked to go "dark" for a few weeks. :(
Same links should work then.

stan
RBD
Posts: 141
Joined: Tue Sep 16, 2008 11:31 am

Re: Fully Articulated 3D Skeletal Hand Tracking

Post by RBD »

*poke* :wink:
Will this 3D skeletal finger tracking be included with Intel's Perceptual Computing SDK? (already supports 2D "close range hand/finger tracking").
http://software.intel.com/en-us/vcsourc ... puting-sdk


BTW Those interested in this for physics applications might also be curious about Microsoft Research's Digits Hand Tracker prototype:
http://www.youtube.com/watch?v=Tm2IuVfNEGk
http://research.microsoft.com/en-us/new ... 00812.aspx
http://research.microsoft.com/en-us/projects/digits

Also some hand tracking stuff currently being done with the existing Kinect:
http://www.youtube.com/watch?v=-jWMaY0WO7c
melax
Posts: 42
Joined: Mon Apr 24, 2006 4:55 pm

Re: Fully Articulated 3D Skeletal Hand Tracking

Post by melax »

Will this 3D skeletal finger tracking be included
Not an officially supported product, but feel free to try:
http://software.intel.com/en-us/article ... al-release
Post Reply