Head-Tracking Pointer is an application that, using an inexpensive camera, lets users control a mouse pointer by aiming their face around the screen. It works with virtually any camera that can be connected to a computer, and without the user wearing anything. This solution provides computer access for those unable to use traditional pointing devices due to temporary or permanent physical impairment.
> Easily-Used Mouse Replacement
I beg to differ. I’ve just sat for 15 minutes stupidly waving my head from side to side for nothing. I read the training section in the manual and the thing tracked me fine during the side-to-side stuff, but still wouldn’t lock on.
I’m sure it works great in the lab, though…
I’m sorry you had trouble training the system. Being a vision-based system, variations in lighting and image background will affect the performance. This has been a one-man part-time project, so I have only able to test the HeadTracker in a limited range of circumstances. Indeed, one of the things I would like to get out of the alphaWorks release is feedback on the performance in a wider range of environments.
The problem you had, a failure to lock on to the face, is very often the image of the head being too large in the image, but there can be other causes.
When you have trouble like this, it would be very helpful if you could send me an image or video showing exactly what is happening. You can capture such data by right-clicking on the video window and selecting either Save Current Image, or Start Saving Video (In the case of video, let the system run for a short time and repeat, selecting Stop Saving Video).
Send the result to me at [email protected] and I will use it to improve the system, and hopefully get back to you with a fix.
Thanks!
Question/Suggestion. Since I can’t run your software on my system (BeOS) I could not test out the idea myself.
I don’t know how you lock onto the face of your subjects but would adding a brightly coloured/reflective headband make it easyier to track head movements? Even if not with the present software maybe as an option. To a handicap person using your system a coloured headband probably would be of min. interferance and to your non-handicap users it probably would considered a minor glitch if it makes for a major improvement in performance.
Also if a headband would help, could the right type of hats work too? I already have my programming hat ready for future use.
Sounds like a perfect toy for those who hate to move their hand off keyboard and reach for the mouse. Also, do left/right eye blink works like left/right mouse button click? How do I install more “buttons”?
Unfortunately small facial movements, like eye-blinks, can’t be detected reliably enough without a pan-tilt-zoom camera. To make it work with a web-cam, I had to rely on gross body movements.
Also, remember that this is intended as a tool for users with physical disabilities that prevent them from using a mouse or other pointing device. For that audience, dwell click is a standard technique. A large part of that audience may not have good control over fine movements such as eye blinks, etc.
Can you elaborate on what you mean by “install more buttons”? Are you refering to using the middle mouse button? I’m redesigning the click control dialog yet again in a way that will allow access to the middle button. The trick is to come up with a design that is easy to use and doesn’t take up too much real estate. As you add more buttons, the complexity climbs quickly. The current design is a compromise between utility and simplicity.
But I’m always open to new ideas, so if you have any, please pass them on to [email protected].
Thanks!
Rick
I second Anonymous’ prosition: while great in concept, it needs to be flushed out a lot to actually work. Not to be a troll and take away from this programmers’ hard work, but I bet if this were open sourced, it would quickly gain much better controls. However, then IBM would never be ablt to perfect it and sell it, which is probably better in the long run. I see this as being big within few years. Also, there need be a better way to click, say, stick out your tounge?
Rick, apologies for the ‘blunt’ — no, let’s be honest, ‘rude’ — appraisal. I do appreciate the effort involved in this kind of thing, and I guess my ire was directed more at the fantasy summary than the work, as I basically expected the results I got. I really don’t fancy sending stupid videos of me looking like a moron, though — sorry!
What bugged me a little was that it tracked me fine during the side-to-side movement (keeping the red box in the right place), but never followed it up with a lock-on. I recently played about with the demos from the Intel CV library ( http://www.intel.com/research/mrl/research/opencv/ ) and they worked reasonably well (eg. keeping a mesh of dots positioned correctly in 3D as you move your head).
I do know this stuff is not at all trivial, so just wanted to offer apologies for seeming to rubbish your hard work like that (1st post). Keep plugging away!
For we annoying and Discovery.Ca watching Canadians, this isnt news to us. Less so if your in HCI to any extent. Im curious as to how similar the tracking is to the project “Nouse” (I believe that was the name). Nouse controlled the mouse cursor using head movements, just as this system does, but used a tracking system which only relied on the participants nose (Hence Nouse, er, if I remember the stupid name correctly). Appearantly that particular system was able to track in different lighting etc, due to the ehhh.. distinctive shape of the “pointer” attached to our faces.
It doesn’t do anything. It’s just a little video box with red lines around my face.
RTFM 🙂
If you still can’t get it to work, please send me an image or video of what the system sees (see instructions in some of my other posts).
Thanks!
RickK
[email protected]
There are several face trackers commercially available that use retro-reflective dots and active (infared led) tracking hardware. The dot is one of the main complaints about those systems – people don’t like having to put something on their face / head to use them.
This head tracker is a spin-off from my main research focus (gesture recognition). As part of that work I developed several methods for tracking arbitrary 3D objects under difficult conditions, and I thought head tracking would be a good project to demonstrate some of my algorithms. Several people in the accessibility community saw the demo and liked that it solved some major complaints with existing head trackers (no dots, cheep hardware), so encouraged me to refine it. I’ve done that off-and-on over the last several years.
The detailed tracking works quite well once you are past the training step, so I don’t think there is a need to “give up” and resort to dots. I am still working to refine the training, but for various reasons, the powers that be wanted to release it on alphaWorks now. I’m confident the training reliability will improve, especially if I get the feedback I hope to from this release.
So send me those images / videos when it doesn’t work! No-one will see them but me – and I don’t care how silly you look 🙂
Once the face tracker locks on to your face, it tracks very well – no dots required. But the system needs a way to initially find a face to lock on to. Users who can’t use a mouse can’t train it manually, and existing automatic face detectors don’t work reliably in the wide range of environments a tool like this has to.
Head tipping works well as a simple way for a user with no other way to interact with the system to signal they want to train it (or re-train, if tracking starts to fail, or it locks onto a bad part of your face). Kind of a “here I am” or “you’re not working well, try again” signal.
Head tip detection uses a completely different method of finding a face than the detailed tracking – it has to be very effecient so it runs transparently in the background, and work with no assumptions about the environment, lighting, shape or color of the user’s face. Watch carefully sometime how different faces can look depending on where the lighting comes from, not to mention glasses that can look opaque or have moving glints on them in different lighting. Once I find the head with the head-tip signal, I can tell the detailed tracker what to look for.
I’ve put lots of constraints on the head tip detector, so that it does not inadvertantly train as the user uses the system, when someone walks behind the user, etc. Unfortunately I think I may have made the conditions too restrictive.
The most common problem is when the users head is too large in the image (I ignore large moving blobs to prevent inadvertant training by people walking by, or the camera being bumped).
Another common problem is people not moving their head side-to-side far enough. I make them move it fairly far to prevent inadvertant re-training from the small head movements when they are using the system (moving the mouse left and right).
The trainer essentially looks for head-shaped moving blobs, so if stuff is moving in the background while you try to train, or your shoulders are in the image and moving while you train it, it may ignore you.
To help me refine the trainer, *please* send images or video of when it does not work. Right click on the video window and use “Save Current Image” or better yet, “Start Saving Video”. Send the resulting file to [email protected].
Thanks!
RickK
I’ve seen Nouse, and played with it. I don’t know in any detail how it tracks the face, so I can’t say how similar our algorithms are.
My understanding is that Nouse relies on the distinctive appearance of the nose or nose/eye intersection. I wanted to avoid any assumptions about the appearance of the face, so it would work in almost any environment. In fact, if you wave your hand back and forth in front of the camera, like a tipping head, it will happily lock on to it 🙂
RickK