ZDNet's David Coursey talks with Microsoft Research about a new technology that uses simple hand gestures to control a personal computer. The video requires Real or Windows Media.
Post a Comment
I like this idea. It does make alot of sense as screens get bigger and the environment where you use computers is not just the traditional office anymore. I think they are also going in the right direction by not using a glove with sensors, it really isn't that practical to put on "computing gloves" every time you want to use your computer. This has alot of neat applications, especially in games, imagine playing a boxing game on your holographic big screen with life size characters that respond to your punches 
Didn't I hear about this being tried in the early or mid ninties, but then with IR light or something?
Anyhow, now we have devices that actually can make use of these kind of pointing devices, for example the TV or as MS like to call it the large central screen of the house 
And pracitcally anything that is currently controlled by a remote control. However, it's pretty much useless for the desktop, which is why previous attempts failed I guess.
What I'd like to see now is some kind of eyecontrol for the desktop at a reasonable price, and then later some mindcontrol perhaps
So you wanna use this technology to beat people up is that what you're saying?
we experimented with the nintendo powerglove back in the day, jaron posted the schematics for getting it all to talk, he also wrote the driver for *gasp* sls linux and we were go!
But that was'nt quite as nice as the thumbcode project...
this is good.. microsoft should pull a big iron and do a ibm.. go all out for awhile on r&d.. think of all of those yummy billions going into technology research..
Innovation. Linux Zealots, Microsoft is where it is because it always innovates. And don't fucking complain that Windows 95/98/ME are unstable anyway - When these versions were releasedd, Linux was just a wish. We have WinXP to pretty beat down every linux distro.
INNOVATION, AND NOT COPYING like all linux devels want to do. Copy Windows, copy MS, copy this.... copy that...
Now that Windows is being seen more and more on handheld machines ( WinCE, Tablet PC, Remote Desktop ) the death of the mouse is overdue
Personally I have all the windows keyboard shortcuts memorised and use both a rollerball and a pen-pad for other inputting-control
Death to Rodents. Especially zillion-button intellimice.
Although I must admit to a personal plan for computer control which is unlikely to make it to production
Just about every application has certain menus.
File Edit View Format Tools Window Help ( Minimize Maximise Close )
Most operating systems map these to Alt-F Alt-E Alt-V etc...
So why not have a keyboard key above the Function Key bars for every common menu?
To DBarros, your inflammatory remarks do look stupid, but you do have a point.
For once M$ seem to be coming up with something original. Of course, its not on sale yet and until it works properly the innovation is useless.
As for saying Linux never innovates, seen any M$ clusters recently? If you want a Linux cluster, just head off to google.com.
"Just about every application has certain menus.
File Edit View Format Tools Window Help ( Minimize Maximise Close )
Most operating systems map these to Alt-F Alt-E Alt-V etc...
So why not have a keyboard key above the Function Key bars for every common menu?"
It might be useful, but as you have to use the mouse anyway to select an option from the menu, I'm not sure it would be an improvement. It is generally nicer to either stay on Mouse throughout a procedure or to stay on Keyboard, and not have to switch between them.
Remember that in some OSes the menus only stay visible so long as the mouse button is held down (I prefer this as it is faster), while in others they stay visible until you click a second time.
" I think they are also going in the right direction by not using a glove with sensors, it really isn't that practical to put on "computing gloves" every time you want to use your computer. "
The big advantage of gloves is that they can in principle give pressure feedback.
Will probably cause the machine to BSOD though HAHA!
It's a joke, laugh!
Now what's pissing me of is that Microsoft very likely will file patents for this "technology", and anybody who wants to implement this on his own has to pay Microsoft because they have patent lawyers.
For my own OS project, I was talking to a co-developer about a concept we have, "UI modules", and how one 2D GUI could be significantly different from another in look and feel. I came up with an UI that uses simple gestures (up, down, left, right, select) for input.
I had that idea, I "stole" it from no-one, but very likely I will not be able to create an UI module like that for my very own OS because Microsoft would sue me to hell and back.
The whole patent system is just rotten.
Will probably cause the machine to BSOD though HAHA!
It's a joke, laugh!
Hahahahahahaaha I am really laughing hehehe but if it was linux it would have probably caused a kernel panic hahahaha
Particulary with Microsoft conception of business and, on another side, the DGMA act, any "hand's patent" seem to me at least dangerous, if not incapacitante...
Anyway, i will no more want to reboot my hand every times, so every 12 hours, i will need a patch to make the wonderfull OS of Mister Bill working approximately secured ;-)))
Another problem : how those guy visiting the sex site will do their stuff "working" ? Hm ? ;-)))))))
Definitely a bad idea.
like unreal or quake....oh my god !.
by the way, i think that my arm is more relaxed using a mouse (or better a trackball) and not pointing the finger and moving left or right.
Of course, Microsoft waved good by to the mouse, but unfortunately it formatted the hard-drive. 
"Hey, where's the coffee machine?"
"It's right over there..."
<file deleted>
"Doh!"
Of course, it would save a lot of clicks if the middle finger were used to reboot...
I was using this technology back in 1989 on my Nintendo Entertainment System via Brøderbund's U-Force (very different from Mattel's Powerglove). Sure, it wasn't quite as advanced and had a smaller sensor area, but the concept was there. Similar technology is also prevelant in certain arcades, even recently -- Circle Center's Gameworks' Tekken 3 machine, for example.
Didn't I see the same thing in the movie Minority Report? I know that was just special effects, but the general idea is the same.
And what was the major problem with it?
He turned to talk to the other guy and wiped all of his work away. Critical to stop a pending murder with limited time, and all the work is wiped away. Perhaps it's just off screen, but that's still pretty pathetic. I can only imagine what would have happened if he had to scratch somewhere.
Minority Report is a VERY good movie in that it shows all the ways NOT to make a UI. In practically EVERY instance, the computer finds a way to get in the user's way. I'm surprised more people weren't going off in computer frustration induced office building killing sprees, I certainly would have. If time was important, they should have found a way to start calling up the computers the moment something was reported.
I remember that they had several pictures of suspects on a card, which then had to be walked around the room and plugged in. Sneaker net?!?! They have retinal code scanners everywhere flash blinding people, real time news"paper"s, holographic imaging, and they're using SNEAKER NET in what is arguably one of the most advanced facilities on the planet?!?! Why not simply have all the available info on these guys already called up and ready to go? Not necessarily visible, but all there? Nope, they had to choose one, get the info, and then figure it all out. Seems to me they could have shaved at LEAST two or three minutes off all that, even BEFORE he started his little arm waving performance...
Also, the interface in the movie itself is not overly useful. Hold your arms over the keyboard, say about 6 inches or so... Do that for 5 minutes. Tired yet? Now try standing and doing this. This is very limited use technology....
I don't see this being useful to several years at the least, and certainly not to the extent of Minority Report. Not in its current state.
So why not have a keyboard key above the Function Key bars for every common menu?"
It might be useful, but as you have to use the mouse anyway to select an option from the menu, I'm not sure it would be an improvement. It is generally nicer to either stay on Mouse throughout a procedure or to stay on Keyboard, and not have to switch between them.
Arrow Keys and Enter do the trick very nicely for me.
There are a lot of companies (hardware/software) who researching/developing things like this, this is not a specific Microsoft thing but its called evolution..
Is this why some people call Microsoft innovative?
Because they got metioned?
It's hard to get rid of them. I mean you can lay traps or buy a cat, but the little suckers are smart, and they breed like rabbits!
Seriously though, for the foreseeable future there's always going to be some form of a mouse, if not for the older people who grew up with them and are comfortable with them, then for the people who are handicapped in some form and thus need a manual tool to move around as opposed gesturing and such.
I myself love having a multi-button mouse which allows one to assign different buttons to different tasks, dependant upon the program I'm using. While I understand that once gesturing is mature there'll likely be similar methods of handling program-dependant gestures, I think it'd be a lot easier to remember something like "left-side button equals COPY" and "right-side button equals PASTE", as opposed to something like "right half-circle with a swirl equals COPY" and "two sqares plus a 'poke' equals "paste".
Also, for us who make our living with a PC, removing our hands from the keyboard to gesture at the screen is a lot less productive than keeping one hand on the keyboard and then using the other to manipulate content with a mouse that's 6 inches to the right of said keyboard.
Good idea (and a step away from mouse gestures which are just now starting to catch on), but I don't think the mouse is going away anytime soon.
the reason is becasue who the hell wants to hold their arm in the air for hours a day to direct the pointer? the mouse you rest your arm. dumb dumb dumd
You're joking, right? Look, when Win95/98/ME were released, Linux was indeed in its infancy; still, Linux was far more stable than these programs (I've used 98SE and Linux extensively). You're also forgetting that OS/2 was around, which was infinitely more stable than 95/98/ME and is still used as an embedded OS today because of its reliability (though it is no longer cutting-edge enough to be a major desktop OS--no offense, you OS/2 fans out there!). And you've neglected the *BSD family, which has been around in one stable form or another longer than Windows has. The fact is that a company with that many programmers and that much cash in the bank should have put out a stable OS loooong before WinXP. This isn't a flame, but rather simple product analysis.
Now, go to your room...you're grounded.
Oh, forgot to actually address the main topic in my post above (sorry, I just get off track in the presence of idiotic flame-throwers). Anyway, this glove concept looks like it could definitely work. I don't think it's going to explode--most people will be happy with the mouse, at least for a while. Still, the power users just might be interested enough to make this thing work.
/me wonders how long before something like this appears on sourceforge
Even if its not that advanced.. it could require you to wave your hands infront of it, so it will know what the 'hands' are. And then just track the pixels.. Of course, if hands go out of the picture...
By the way, anyone know why it needs 2 cameras?
By the way, anyone know why it needs 2 cameras?
Simple: Binocluar vision. Its what gives true depth perception, I beileive.
Chris
Several months ago, Intel released libraries which would allow someone with an Intel-compatible machine and a webcam to read video inputs to interact with something on the computer. They even had a demo "game" where you could use your hands to "pop bubbles" which floated toward you on the screen. It needed significant amounts of computing horsepower; don't attempt this with your old Pentium-90 machine. More than a few people dismissed it as a way for Intel to drum up business for its newer CPU's. The press release regarding it is dated December 11, 2001.
http://www.intel.com/pressroom/archive/releases/20011211tech.htm?ii...
Took Microsoft this long to use it. I wouldn't describe Microsoft as the innovators, here.
http://www.atarimuseum.com/movies/mindlink.avi
This is the only link I could find. ITS REAL SLOW RIGHT NOW!
The Atari Mind-Link; I guess it was ahead of its time after all ;-)
This was like back in ~1984; never officially released, but the commercial shows a working demo and some really cheezy music
It was for Atari 8-bit computers. It is good to know Microsoft is catching up though . . .
See how its the m§ people that always uses the word "Innovative" one too many times? i once read this one page article by microsoft that had the word innovative in it atleast 60 times...."Innovative" this, "Innovative" that... the all new never seen before "Innovative" "Boot-manager".....
-
Its so dull....really.....If you want to complement microsoft please lay Off the friggin Innovative word... Everything microsoft is inventing HAS 100 % Been done before or has been thought of.
Regards from an "Innovative" User because im probably the only one that has thought of this before...not
We've seen stuff like this in movies... if I remember correctly Tom Cruise did something like this is Minority Report... and someone already mentioned the idea that it was already attempted using IR. Overall this doesn't seem that different from certain gesture recognition done with some AI work. Like when the guy in front of the camera waves "hi," the computer says, "Hello Bob, glad to see you today."
I've already done something like this, except it ran a bit to slow for realtime use (I wrote it in perl).
It sorta got boring before I got around to re-coding it in C, but I did find many examples of similar technology, including some Intel libraries with some cool demos.
Use your nose as a mouse: http://www.cv.iit.nrc.ca/research/Nouse/index2.html
And loads of other programs (usually just demos of research people have been doing at universities) can be found by just searching for "mouse gesture recognition" on google. That, and MANY papers on the subject 
>by the way, i think that my arm is more relaxed using a mouse (or >better a trackball) and not pointing the finger and moving left or right.
My exact thoughts
The USAF came up with this idea loooooooong ago (try before Microsoft was founded). The technology was not around to really do the task as well as it can be done today, but all it is , after all, is motion detection with edge detection mixed with gesture recognition.
I remember watching a TLC special about computers the mouse (invented WAY back in the 50s!!) and the next revolution expected in computers. Voice was expected to be next, but they brought up privacy concerns and stated that some company (not M$) had developed a simple hand gesture system to control the system cursor, but went bankrupt. Then they also SHOWED someone 'mentally' moving a mouse cursor (actually just careful eye movement scrutiny).
All this stuff has been around for a long time, computers are not new, only the wide-spread use of them.
--The loon
PS: Flame me all you want, I rarely check back on non-BeOS threads.
There! I said it!! I'm racing to the defense of Microsoft! Whahahahahahahahahaha....
Okay, so I'm no MS apologetic. However, I do have to say this: while this glove concept isn't new, we should keep in mind that it IS Microsoft that's really making it work (assuming, of course, this does actually pan out). So sure, they didn't come up with the idea--and sure, we might be a little remiss in calling this "revolutionary"--but it certainly IS "evolutionary," as has been pointed out above.
By the way, remember that this is still in development. I'm sure that Microsoft developers wouldn't be stupid enough to create such a glove that must be held out and above the keyboard for hours at a time, as some have suggested above. They're a little smarter than that!
There are plenty of problems with such a system, first and foremost it would be akward to keep your hands raised. What about multiple computers. For instance here is my home configuration: http://alex.phataudio.org/boxennew.jpg If each of those computers had motion monitors, they would conflict. Turning them on and off each time i wanted to move somthing would be a pain in the ass. What happens if someone taps me on the shoulder and a swing around with my hands raised? Oops, you just formated your HDD.
..well, this certainly is innovation, but i'd say it's
utterly useless for 99% of computer users.
now, after years and years of improvements in mouse
ergonomics, you are going to tell me that it's better
to hold your hand up in the air for eight hours a day
than to drag your hand on the mouse pad? Think about it,
or better yet, try to hold up your hand in front of the
screen and make gestures like those for only 30 minutes
without resting.
If your arm is not burning and aching by then, then i'll
admit you are way stronger than me :oP
All jabs at Microsoft aside for claiming to "innovate" an interface that was on the NES years ago, this could be quite useful for laptops, whose pointing options are still sub-par. Laptops are always clear, have very small screens, and have fixed areas for your hands. A laptop manufacturer could incorporate two minimal cameras and an image processing chip above the screen, and use that to A: take images for the person's chat program and B: control the cursor. It would probably shrink the laptop setup slightly, as the trackpad would no longer be needed, and users that so chose could plug in an external pointing device.
Now don't get me wrong, I'm not saying this isn't cool, it most certainly is, but... why do I have to use my whole hand?
You see, I'm lazy. I am the classic example of lazy. I have two assignments and a test due in four hours and I am talking to you guys here. I just don't like to do work that isn't absolutely necessary. Lifting my whole hand and pointing with it sucks. What they need is a way that I can just point with my finger. As it is, the software doesn't seem sensitive enough to do that. If it were, they probably would have been finger gesturing instead of making that really stiff handed approach.
Besides, it is much more natural for me to point at a spot on the screen with one finger than to slide my hand around like a mouse.
Keep your hand raised long enough, and it wont just feel awkward, but the blood will run out of your hand. Do it for long enough, and I'm sure it'll start to turn slightly purple and feel tingly. I can easily say I work at my computer way too often to use this unless it lets me rest my "mousing" hand on the table and still operate. At that point, I'd rather just have a mouse with 7 buttons!
It's something MicroSoft can push as innovative because it sounds really cool. It'll probably go the way of MS Bob. Not that it's bad to get some research done on this topic, but I just doubt it'll be profitable once people start using it.
Actually, I'll bet something like this would look AWESOME in a demonstration! Imagine being an office worker who only knows that you click the Word icon to write things. Watching someone use this for 5 minutes would look damn cool! Then ya get it in the office, and turn it off when your arm aches.
...that this might not be meant for the average user, but for those who cannot use the keyboard or mouse?
For me it might be useless, but there are many for whom this would make the computer and the internet more accesable, especially since it needn't be a hand.
Also, it might be possible to have the cameras aimed at your feet so that you can input commands with your tootsies while the fingers are at the keyboard.
This is not inovation, it is just some way to make more cash. They will launch a new fassion and call all mouse-users "lame" or whatever. The younger generations will of course buy it because kids&teens don't understand the sharks behind theire products.Technologies are developed in advance, so by the time most of them are released they are finished. It is only marketing so that they can make more money, because once a technology has matured and doesn't need improvements it is killed by a young technology, same goes for software. And to answer that other question about the crappy OSes from MS: They did not have any intention of investing too much, but they wanted to gain $$$. What helped them? The licensing techniques and some other stuff including the mass stupidity and lack of apps for other OSes and "good" publicity. Now people start crying 'cause they can't be stoped, so it is stupid to complain about them without doing anything...
You see, I'm lazy. I am the classic example of lazy. I have two assignments and a test due in four hours and I am talking to you guys here.
Ah, dwilson, dwilson...get your homework done! Yes, I understand the animal magnetism that keeps drawing you to us when you should be attending to other matters, but get a hold of yourself, man!
And why would you not want to use your whole hand?! It's so much cooler to flick your entire hand (and maybe even yell "Shazam!" when no one else's home!) than to just twitch your little old finger. Just keep that in mind, young man!
You do have to strive and convince people to use Linux on their computers, and sometimes you try to keep yourself convinced you should not erase it off from your own hard drive.
Windows is different. People just go and use it!
I can't imagine how an author would easily convey the equivalant hand gestures in software instruction mauals. Right now they can simply list the key combinations, but how the heck will they demonstrate "swirl your finger around and point down with you index finger." Just seems clunky in this respect. Maybe we'll have animated manuals so you can mimc the examples.
I very much doubt this will be targetted at the home-user, and much like RonG said, it's more likely to be used in specialised fields. I can see this being the sort of technology you'd use in hospitals before surgery on 3D models etc.
Obviously, no, it isn't occurring to anyone else.
The UI in 'Minority Report' was based on interviews from researchers working in gesture-based interfaces, more than likely including some who are at Microsoft Research. The movie implementation was Hollywood-ized, to be sure, but a <em>lot</em> of the world as presented there was based on concepts from 'futurist' researchers, not special effects artists.
I think there's a recurring pattern in (some, not all) commentary on OSNews--an inability to give credit where credit is due, in both directions. By which I mean, either no credit is given at all, or far more credit than should be given. So here, we have "Microsoft never does any innovation at all" set against "Nothing in Linux is more than cheap copies of Microsoft's work." Thematically, this isn't different from our more typical fight of "Macs are five times faster than PCs at half the price" versus "Windows Uber Alles Resistance Is Futile Pathetic Apple Loser."
Microsoft Research <em>is</em> a pure research lab, guys. They're one of the few commercial research facilities still around doing this sort of thing, along with PARC (no longer part of Xerox), Bell Labs and IBM's various research divisions. And no, Microsoft hasn't come up with anything that compares with the best of those labs--but they're a lot younger.
Yeah and you go through those manuals with your trusty mouse.
I think people with a physical need would be better off with voice controls. As a technology display, it's kinda cool, but as a practical technology it is not. Waving your arms and legs around is going to get tiring real quick.
why do some of you think that these kind of pointing devices are meant for desktop use? there are computers elsewhere you know
and no, it's not innovative, I still haven't seen a single innovation from MS research, they have though made improvements to some technologies, like optical mice for example.
One needn't wave ones hands. A wand attached to a forehead ( for those without fine motor muscle control ) may be enough for some people. Or as I mentioned before you could use your feet, which is a lot less tiring.
Also, I don't see why it is required to wave your hands in front of you. It could be set up in such a way that hands hanging at your side could operate it.
By the way, don't Blackhawk helicopter controls allow the use of eyemovement to control the machine? Why don't we see mmore of that in the computer field?
But all in all, I think the keyboard mouse combination for input will be very hard to beat, and the 2D desktop anology is hard to beat for output. Any particular 3d i/o tech may have niche applications, but from what I've seen 3d input will never be as easy or as fast ( until computers can read minds ) and any 3d representation of my home computer system seems like more trouble than it's worth.
"A loud clutter of gunk music flooded through the Heart of Gold cabin as Zaphod searched the sub-etha radio wavebands for news of himself. The machine was rather difficult to operate. For years radios had been operated by means of pressing buttons and turning dials; then as the technology became more sophisticated the controls were made touch sensitive - you merely had to brush the panels with your fingers; now all you had to do was wave your hand in the general direction of the components and hope. It saved a lot of musular expenditure of course, but meant that you had to sit infuriatingly still if you wanted to keep listening to the same program."
The Hitchhiker's Guide To The Galaxy - Douglas Adams (1979)
Of course MS does bear a striking resemeblamce to the Sirius Cybernetics Corporation.
On a more serious note, if mice can give you RSI then having to gesture at the screen to perform actions isn't going to improve mattes.
Share and enjoy. :>
Also, it might be possible to have the cameras aimed at your feet so that you can input commands with your tootsies while the fingers are at the keyboard.
Now this is ingenious. The worst thing for me is to have to take my hands off of the keyboard. The switching between keyboard and mouse is one of the worst innefficiencies in modern computing. If I could leave my hands on the keyboard, and point and click with my mouse when necessary (using a foot, or even toe)... my God, somebody get these gestures tailored to feet NOW!
I honestly think this could be a huge productivity increase for those of us who type fast and use a keyboard well, but are frequently chained to using a mouse.
"Also, it might be possible to have the cameras aimed at your feet so that you can input commands with your tootsies while the fingers are at the keyboard.
Now this is ingenious. The worst thing for me is to have to take my hands off of the keyboard. The switching between keyboard and mouse is one of the worst innefficiencies in modern computing. If I could leave my hands on the keyboard, and point and click with my mouse when necessary (using a foot, or even toe)... my God, somebody get these gestures tailored to feet NOW! "
...?
So why don't you buy a trackball and drive it with your feek? wacko! LOL
You gotta love how Microsoft wastes money to invent dumb technology and then will force governments, corporations, and average Joes to pay for it.
Nothing like a monopoly. And as others have said, I'm sure the physical therapists and doctors are on board with this system.
I can't wait to see how Microsoft Hands works with Palladium. Now that will be funny. You gesture and the machine gives you the Microsoft Finger.
actually google for "jaron lanier" and be stunned
How is it that MS "will force governments, corporations, and average Joes to pay for" these new products? Have you received your copy of Microsoft's "compulsory purchase" agreement? Please...MS-paranoids credit the corporation with waaaaay too much power. They're probably a monopoly (at least they acted in illegal ways), but they can't "force" you to buy anything. This is just another toy that the vast majority of us will do without.
(And how do you estimate that our tax dollars have gone into this? Does MS have some special Federal funding we don't know about?)
//seen any M$ clusters recently? //
You mean the distributed failover cluster for SQL Server 2000 that I built 18 months ago, with (so far) 10 minutes downtime?
Yah, I see it every day. Along with the smile on my boss' face.
Next?
RonG:
By the way, don't Blackhawk helicopter controls allow the use of eyemovement to control the machine?
You're probably thinking of the IHADDS system on AH-64 Apaches. Very cool stuff, but probably still mucho expensivo. At least compared to a $5 mouse. All in due time..
rock_the_casbah:
You mean the distributed failover cluster for SQL Server 2000 that I built 18 months ago, with (so far) 10 minutes downtime? Yah, I see it every day. Along with the smile on my boss' face.
Your boss has been smiling for 18 consecutive months because a Microsoft product (several thousands of dollars worth, no doubt) is, lo and behold, working like it is supposed too? I find that rather telling, don't you?
PS. A "distributed failover cluster"? Is that anything like an "ATM machine", "PIN number", or "salsa sauce"?
>Also, it might be possible to have the cameras aimed at your feet so that you can input commands with your tootsies while the fingers are at the keyboard.
http://www.footmouse.com/
No camera required.
While I could not view the videos since I don't have Media Player or Real One, it sounds totally stupid.
//I find that rather telling, don't you? //
Hard to tell if you're being sarcastic. If you are, what's your point? If not, what's your point?
//PS. A "distributed failover cluster"? Is that anything like an "ATM machine", "PIN number", or "salsa sauce"//
Uh ... yah. Whatever! Guess it's my fault that you're ignorant about this stuff. More veiled sarcasm? Bah! Not worth thinking about...
So what if the idea is not original, so long as it works I don't care. I also doubt Microsoft targets this at home computers. This technology is best suited for changing slides in Powerpoint presentations, for disabled people etc.
I bet that the day someone posts a story about the first Warp-drive, someone will say "What's the big deal? Star Trek had this technology years ago!".
