Whether you like Microsoft or not, the Redmond giant does have one thing going for it: the company’s research division. Working together with several universities and other institutions, Microsoft Research works on the soft and hardware of the future, ranging from research operating systems to insanely cool things like what Microsoft’s chief research and strategy officer Craig Mundie showed off during the Microsoft College Tour ’09 (more videos).
I’m sure most of you remember the flashy gesture-based interface used by that certain actor in the science fiction film Minority Report. Standing in front of glass panels, the actor swooshed his hands in the air, manipulating objects on the “screen”. It looked pretty cool.
Microsoft – but loads of other companies too – have been working on implementing Minority Report-style concept interfaces and hardware for a while now, and during the Microsoft College Tour ’09, Craig Mundie showed off working hardware which looks pretty much exactly like what you see in the film.
It’s a glass panel, standing in front of Mundie like a normal monitor. Mundie lays a tablet computer in front of it, and by flicking an object on the tablet upwards, it appears on the glass panel. He then proceeds to use gestures to manipulate the object on the glass panel – without actually touching the panel (the embedded video requires Silverlight, but you can download an .mpg too).
I’m not so sure it will be comfortable to use such gestures on a regular basis, but the glass panel display is something that has me really excited. I’ve long had the belief that computers – and thus, displays – will become ever more seamless parts of our daily lives; transparent, if you will.
Currently, a computer is still a dedicated device which you carry around or have stationary somewhere in your house. It is clear where the computer is and what it looks like. It is hooked up to another dedicated object which carries some serious presence with it: the display.
Displays, whether they be computer monitors or televisions, are getting ever thinner, and I think we’ll soon hit a point where a display is nothing more than a sheet of thin glass, barely noticeable when not turned on. I would love for my computer monitor or television to be a nearly invisible pane of glass stuck to the wall – as it stands today, the TV and computer monitor are the centrepieces of many a household, even when turned off. Wouldn’t it be great to have displays that more or less disappear when you’re not using them?
Thank god for companies like Microsoft who are willing to spend countless millions of dollars on research, together with universities. Some of this research will benefit the entire world. Like any other company, I dislike Microsoft – but not their research division. Those boys and girls are doing some damn interesting work.
The Minority Report interface was pretty. However, it seems to be more of an audience interface, something to let the person watching the show know what the actor is doing on the computer. Unfortunately the video refuses to work for me so I can’t comment on it specifically. However, I’ve never seen a use case for a wall display/figure gesture interface that wasn’t horribly contrived and less useful then what is available now.
I used to think the very same thing (and, to some extent, I still do). But these days I can at least think of a reasonable motive behind all that thing: I believe they aren’t thinking about the usage of computers we have today: Not the personal computer where you sit (or lay down) and access things.
I think it’s more like renewing the concept of a plugged house. One central computer with various kinds of interfaces around, mostly less powerful to the way we use them today, but possibly more expressive for small tasks and things you’d do in a “Surface” iface and this Minority Report-like thing.
Having a few of those things (in a not-so-near future) around in the house, and having access with simple gestures and touches to things like notes, videos, music, video-conference and stuff like that.
That’s, obviously, just my view of the thing, but I could at least see it being somewhat useful to that experience.
You’re right. As Quato said (in Total Recall), “Free your mind, Quaid.” If you’re going to dream about the future, don’t start with what you have today — think about what you would like. Current interfaces (eg. the desktop) use physical/spacial metaphors to represent data. A friend of mine refers to the desktop as his “alternate junk drawer”. But there’s no reason you can’t think in other terms — temporal, for example … using a timeline. While we have a tough time visualizing how useful that would be, it was completely useful in “Minority Report” because they were essentially sifting through temporal data. Yale computer scientist has been working on this concept for years. He calls it “Lifestreams”. Check it out. Worth reading, if you’re interested in this sort of thing…
http://www.wired.com/wired/archive/5.02/fflifestreams_pr.html
I don’t feel like ranting about Microsoft, but I do think I know what you do make an interresting point. I always imagined people would have one device.
They would carry it around like a mobile/pda and when they are home, it would act like a remote for their TV and a control-system for their home automation-system and tv, etc. recording system (possible also from outside the house).
But maybe not, maybe it will be more like the lightswitch in each room. Which you flick on when you go in the room and flick off when you leave again. Obviously for other functions, sound/music, curtains, lighting.
Interface wise the movie and video both look terrible usability wise. I think it would only work for narrow applications. But maybe that’s just me.
This is basically what I imagined as well. With my iPhone I actually already have some of this, it’s a phone/PDA but when I’m on my home wifi network it also controls my media, being able to control iTunes in my case. Now, if we just had something like that but obviously based on an open protocol so people could use whatever devices they want. Then, once this starts getting built into other household devices, we’d really have something cool.
Farenheit 451, anyone? Hide yer books … it’s coming …
I don’t see any use for those kind of things. For example typing is much faster and doesn’t strain you as much then writing by hand. Also for some reason Microsoft seems to have an unhealthy obsession for rotating and zooming things. Most tasks I know don’t evolve rotating and zooming things, maybe in the future everybody is rotating and zooming away like crazy on floating sheets of glass but for now we need the precision and speed of a keyboard and mouse is stead of making gestures to our screen.
I’ll admit my muscles hurt a little from the Mexican flu but if I try to copy the gestures he made to zoom the windmill, my arms start to tire pretty quick, don’t want to do that all day.
Rotating: yes.
However, dude: zooming and panning with your eyes. That would be a salve to quite a few Window Manager conundrums: you look at what you want to have the largest size and most detail. Zooming interfaces are awesome, but difficult to balance controls of, if you allow multiple windows at a time.
The great thing about microsoft research is that they are well removed from the rest of the company. Researchers seem to have more or less free reign to do their thing, so a lot of what comes out of there is wildly impractical, pure research, blue sky stuff that certainly won’t end up making microsoft any money anytime soon. That’s great to see from a company research division.
They may be well removed from the rest of the company, but they can only bring forward, pardon my expression, crap. To many security holes. They need to get rid of registry, they need to make it an OS without holes (security vulnerabilities). They also need to make it more modular so that if a virus gets into a third party software it will stay in that software. When you have people actually endorsing wiping the hard drive and reinstalling every 3 months to maintain functionality, there is something wrong.
http://livingfortruth.wordpress.com/2009/11/05/why-i-like-linux-and…
You don’t seem to be very familiar with what MS Research does. They actually do a ton of research across a wide spectrum of technologies (voice/handwriting/gesture recognition, database technology and data mining, data federation, cloud computing, alternate operating system designs, security improvements, etc, etc). Some of it is total blue-sky stuff that never seems to make it past academic papers, but a lot of it does get into product development. The problem of malware has less to do with what Microsoft COULD DO and more to do with fundamental application compatibility constraints that limit what it CAN DO without breaking every application on the planet. Researchers know how to eliminate malware: Remove interop points or sandbox applications in a VM. Virtualize every resource. Don’t allow malware to affect the system.
But that’s easier said than done. There are a lot of legacy applications which many thousands of people depend upon which leverage interoperability points in the system. If MS changes the behavior, it not only causes the application to break, but it creates an outcry from customers and potentially introduces legal troubles, as third party software developers scream about big, bad Microsoft trying to kill them; even if it’s inadvertent or well-intentioned, it’s a problem. For example, installing global mouse or keyboard hooks, injecting DLLs into other processes, etc. Most malware gets installed through social engineering. Click on this cute game sent to you in email. Aw, what damage could that possibly do? Click Yes. UAC. Click Yes. Boom!
How do you prevent users from shooting themselves in the foot when you allow those same users to install applications? It’s a thorny problem. And no platform completely eliminates this problem unless you prevent users from elevating privileges and installing software.
Microsoft Research has funded research into alternative operating system technology — such as Singularity — that is built almost entirely in managed code and which has an excellent security model. Microsoft has shared a lot of information about Singularity (http://research.microsoft.com/en-us/projects/singularity/), and you might find it interesting.
Edited 2009-11-07 03:00 UTC
From the looks of his blog, he’s been using computers for about 5-6 years, so he really doesn’t know much of anything.
You should also visit http://www.oblong.com to read about g-speak and the people who were the actual technology advisors for Minority Report and who have been working on this technology for well over a decade.
This is the kind of stuff we should be working towards and reading Kroc’s “What Would Make Your Perfect OS?” thread left me rather depressed by the fact that most people seem completely satisfied with the current WIMP paradigm and large, monolithic applications.
A bit of an aside…
The fact is that UNIX got it very right with the idea of having numerous small, simple programs that can be chained together to operate on data and produce results. It’s just that so far, that concept is stuck in the CLI and no one has been able to translate it to the graphical world. My instinct tells me that our UIs and our methods of physical interaction with our computers have been the limiting factors in this area.
g-speak attempts to address these problems by following the UNIX philosophy, using small tools and even including complete network transparency for all the X-lovers out there. I’m not sure if the Microsoft solution approaches it in the same way, but it’s nice to see them exposing people to the possibility that there may be better ways to interact with a computer.
Unfortunately, we’ll never see this actively pursued by Microsoft because it is financially detrimental for them to do so. Anything that breaks WIMP will effectively end their domination of the desktop market, and in fact, may destroy the “desktop” market altogether.
RANT (didn’t have enough sleep):
I’m sorry but I’m not impressed. This company has not just millions but billions of ‘excess money’ – they have one of the highest profit margins to be found in any market, yet manage to get away with a stunningly low amount of innovation (compared to their competition) and a huge bureaucracy. Which makes sense considering their market position (Bell System, anyone?).
Now they move some of that money towards a few universities doing flashy, academic stuff – and spend even more $$$ advertising the fact they ‘do research’ (I’m betting the marketing part of the research budget is twice the size of the actual research). And you, Thom, go ‘oh I love Microsoft’. There must be a cool English saying for this but I can’t be bothered to look it up.
If half the marketing budget of MS in the last 10 years was spend on actual research, we’d laugh about ‘minority report’ interfaces by now.
I laugh about those interfaces now. They don’t work, as you need to move your arms up and down, and that stresses too much to do it all day for 8 hours or more. Surely they are pretty, but not practical for work. For home use and multimedia they would work better.
To watch “the UI of the future”, one must be running Microsoft OS and have Silverlight installed. Isn’t that ridiculous? That’s not an innovation, that’s pure marketing.
You need to make something based on, “how does this solve more problems than it creates?”
The problem I see created, is, once again, adding handling time, for the sake of simplicity, to get typical things done. We need a good noun-verb-adverb UI first, then let it flow to something like this, but with less touch (unless there’s full 3D+T tactile feedback, at which point ordinary pr0n will be a bygone relic, and we still won’t get work done any better
).
The touch bit, and floating hand gestures: eh, RSI, go away for anything but plain language and formula recognition, thanks. Real work flows use too much precise repetition for me to consider a workstation touch interface a good thing. Holding your arms up like that gets tiring very quickly, as well. Trying to make touch or gestures work better (phones excluded) is like trying to make a cat embarrassed. You give the cat attention it wants, but no real progress is made. But cats are cute and cuddly, and everyone loves seeing them, so you obliviously keep at it for decades.
Now, that said, this is why the GPGPU needs our attention for non-servers. The implementation of something like this (but without the touch or hand gestures) with a robust GPGPU system becomes trivial, once the UI itself is figured out (provided there is sufficient room for horizontal integration, to add stuff for your own kinds of crap, which MS is usually bad about).
P.S. (I’m being whimsical, today)
I will not download Silverlight
It-is more open than Flash, but still not right
I will not take Moonlight–great Gnu!
It will not suit me, tech I’d rue
I will take flash, but with a grudge
I have a Geforce, just quicker’n sludge
I do not like egg bacon and spam (background: Viking yelling, “Spam” over and over)
I do not like it, Sam I am
Edited 2009-11-07 16:52 UTC
The minory report-like technology was acquired from a startup founder who tried really hard to get it working.
Just to note that such technology requires the hard pushing of a visionary rather than a big slow enterprise company division.
I fell for that the first time I saw it, too.