The Future of Computing Part 4: The Next Dimension

No, I’m not going all “New Age” on you, this time I’m looking at how computers are going to get a 3rd dimension and how this will change the way we interact with them. The previous parts of this series have been based on extrapolations or previous history. This time I’m looking further forward, when technologies currently in long term development become available and open up a whole new realm of possibilities.

A third dimension is going to require large amounts of computing power, luckily enough thats going to happen so I’ll start by telling you how your computer is about to get faster – a lot faster.

Architectural advances in microprocessors seem to have slowed down in recent years, indeed what on the market today is much more advanced than mid 90s designed Alpha 21264?. Instead we are starting to see the addition of additional cores on the same chip and special purpose elements being added. All the major CPU companies are now committed to adding multiple cores on a single die and this along with the additional cache memory made possible by smaller geometries will bring us large rises in computing power.

But CPUs are not the only thing getting more powerful. Graphics Processing Units (GPUs) get more performance at a much higher rate than CPUs but have to date always been limited to producing graphics. These days modern GPUs have programmable vertex and pixel shaders and these are starting to be used for general purpose computations[1] in research and even in some applications. With the next generation shaders they will become not only more powerful but also more general purpose. Just as vector units (e.g. SSE, Altivec) have boosted computing power expect GPU shaders to be utilised in a similar way to provide potentially massive performance boosts – in the order of 1000%.

In part 3 I predicted office and casual computer users will turn away from traditional desktop systems. Desktop PCs will still remain, the numbers may be smaller but their computing power will be massively higher than anything available today. When combined with 3D visual displays all this power will transform what our computers are capable of and how they will be used.

Visual Interfaces are just starting

Some people believe the command line is king and all the graphical stuff is superfluous. The command line is just one way of interfacing with a computer and as such it has it’s own strengths and weaknesses, it offers a great deal of power but at the cost of complexity and having to remember strange commands and their syntax. The GUI has other strengths and weaknesses, features are more obvious so remembering how things work is not difficult or not required at all, this is much better for the beginner. The power of a command line is difficult to present on a GUI so their simplicity can be a drawback, there is no one “perfect” GUI, each interface method has it’s own place.

A third interface method is on the way, it exists in research labs and in select, specialist areas of industry, the 3D GUI is coming.

The human brain has a very large portion devoted to the processing of visual information. Some regard the human ability of abstract thought to be our greatest strength but when you combine this ability with visualisation it becomes a great deal more potent. If you could visualise an equation wouldn’t mathematics become easier? Wouldn’t the progress of science become faster if when we manipulated equations we could see the results visually in real time? Albert Einstein could do this in his head, If everyone had the tools to do the same, think of the progress that could be made.

I for one think exactly those sorts of tools are coming. I think the future will become more visual, I expect the type of interface in shown in Minority Report will actually appear and will be useful for many tasks. I don’t know if we’ll do physics this way but it would certainly have interesting results if we could.

The implementation of the interface in Minority Report was somewhat clumsy with all sorts of weird hand movements required and moving could accidentally send files flying off all over the place. I don’t expect to see an exact replication of this system but something similar could emerge.

Alternatively, it could be something better: Take a large screen and view it through LCD shutters mounted on the screen, have cameras monitor your eye and hand movements with tactile feedback through gloves so you can feel what you can see. I don’t know what such a system will be used for but I bet playing Doom 5 will be awesome. Does this sounds like fantastic futuristic technology which will never appear? Think again, this system already exists [2].

Already more advanced techniques are in development such as Holographic screens which give even better displays. Of course Sharp are already starting to sell 3D display screens today.

The 3D GUI

When I say the 3D GUI is coming I mean a GUI displayed in three dimensions, not a 2D representation of a 3D space such as games deliver. I mean a real 3D display where objects on screen have Height, Width and Depth. Our displays are one way or another going to gain another dimension, 2D displays are going to seem somewhat quaint by comparison.

There are problems though, one major know problem with real 3D displays is that they can confuse the viewers eyes no end. The human visual system expects to point both eyes at a single point and focus them there. 3D displays create images in front of or behind the screen but don’t focus them there, to remain in focus your eyes must focus on the screen yet point at an image elsewhere. The human visual system is not designed to do this and it is the major reason why Virtual Reality has never caught on. Using a large screen (say 60 inch) will mitigate the need for this as it will be further away and depth of field [3] means images not too far off the screen will remain in focus.

The key to this will be subtlety, making sure the images are not projected too far in front or behind the screen. Your brain may be able to get used to this without causing too many problems. Another technique is make the displays bright, your eyes’ irises will shrink and the resulting improved depth of field should allow images to move further away from the screen.

Compute encounters of the third kind

Once we all start getting these 3D screens we will also get applications for them and I expect just as there was for 2D, there will be a lot of experimentation in new kinds of user interfaces and new ways to use them.

There are some pretty obvious candidates for 3D displays, 3D modelling of course stands out (pun intended) as an obvious candidate, as of course are 3D games – where monsters will really try and bite your head off. I will leave the reader to probe the possibilities of porn.

There has been plenty of research into 3D desktops [4] and Sun have recently shown their work [5] but I for one can’t really see the ability to spin 2D windows around being of any particular use. Sun put post-it notes on the back of the windows but that means having to remember to look. It could however be an interesting way of hiding complexity but you could do this just as well in 2D.

No, in order to create a 3D GUI we will have to do better than this, the 2D windows will have to become 3D, and that won’t work for many applications. In order to create a 3D GUI we will have to forget about the way existing desktops work and think of something completely new.

In the following I describe some ideas for interfaces which could work, this area is pretty much impossible to predict but that’s never stopped me before. In the cases below I have tried to find areas where a 3D interface will benefit the application and improve it. This will be the key to a successful 3D GUI, there is not point trying to make a Word processor 3D, it doesn’t gain anything (though displaying 3D images may be useful). There are many application areas which I think will gain from 3D and I have described some of these possibilities below.

File Manipulation in 3D

This seems to be the one area where a lot of 3D GUI research has been concentrated on, unfortunately this seems to be a horrendously difficult problem as many seem to end up with a screen full of thousands of files.

I can’t see how a command line could be represented in 3D but rather than trying to change a command line into a 3D application it could be augmented with a 3D display of the part of the directory tree, files etc. you are working on. The 3D display would be linked to the terminal display so what is done in one is reflected in the other. File manipulation could be done either by command or by selecting and moving files by hand, you could change directory by dragging a folder from the 3D part onto a terminal window.

The 3D representation could change depending on the commands you are using, you could use “ls” to look at which files are oldest in a text display or you could display them from above as pillars, the shortest would be further away and thus oldest.

Some things would be easier or faster in 3D others in 2D, by displaying both 2D and 3D we will get the best of both worlds.

Sounds with depth

Audio doesn’t sound like an area which could benefit from an extra graphical dimension but I think it is one which will benefit more than most. Applications could have controls where moving them would be easier, the mouse is a pain for musicians and many control surfaces exist to supplant it. If a virtual synthesiser is displayed the control knobs could be turned by gripping and then turning them, this means no more mapping from screen to an external control surface so it will be easier and cheaper. To really work properly this will require tactile feedback but this can be done in gloves.

The basic creation of sound could benefit from 3D also, if an synthesiser’s envelope generators or filters responses are displayed in 3D they can be manipulated by hand, creating new sounds becomes a whole lot easier. The depth can be additionally used to represent volume so the response can change as volume changes and this too can be manipulated.

The BeOS had a very good demo of mixing in 3D [6] where instruments could be moved around to change their balance and volume. Extending this to a 3D display device would allow the musician to move instruments by hand instead of mouse. Mixing will never be the same again.

An interesting possibility is to move multiple instruments by hand simultaneously, (think about moving multiple chess pieces with an outstretched hand). With a mouse you have to first select all the pieces then move them, a hand will be a much better and more immediate tool for this.

3D Painting

Depth can be added to existing applications in a 3D GUI to enhance them. 2D Paint programs could benefit from this, This wouldn’t make the program into a 3D modeller but rather the painter could use multiple 2D layers in front of one another to create depth. Traditional 2D painting uses a number of techniques to fake depth and this is really just an extension of this.

3D could however create a better interface for the drawing tools:
In computerised painting there are a number of different pallet types to chose from, rather than selecting from a menu and picking them one by these could all be placed on screen at once in a window, the window would then be shrunk and moved to the background. When a new pallet is required just pointing at the window would bring it forward, pallets could then be dragged out or exchanged. When done the window is just pushed back out of the way. To get rid of a pallet when it’s not needed simply grab it and throw it in the direction of the shrunken window.

If you are an OS X user this may sound familiar, it has similarities to the dock’s “warp” effect. This technique could work for different kinds of application, you could be doing 2D painting or even word processing, it’s a technique which will allow the 3D interface to be used to enhance a 2D interface without replacing it, again you get the best of both worlds.

A scenario:

3D can benefit many areas which can be used together, in the following scenario I put a number of different areas together with a potentially very powerful end result.

A guy goes out with a 3D camera (actually 2 2D ones spaced apart). He proceeds to take a few pictures of buildings and other objects from various angles.

Then he gets home and plugs the camera into a desktop PC (this is one of the people who has one) which is connected to a 3D screen. He then uses the screen to go through the photos combining the multiple photos into 3D objects, the computer makes guesses for any parts he hasn’t got images for. This is all an easy task as it’ll be done by picking up the objects and manipulating them by hand, the computer will scan his eyes and hands so it knows where he’s looking.

He then takes these objects and inserts them into a 3D modelling package and creates landscapes. Into the landscapes he then adds objects of people, these aren’t just dumb 3D representations of people. They have AI software controlling them so they can be made to walk around scenes, the AI software knows how people look and how they move so when directed they will move properly.

This sofware will be better though, it will also know how people look close up and even how they talk. Our user shall then give the actors instructions to move about and talk to one another. The computer will render this in 3D in real time, no previews will be necessary as this computer will have the horsepower to generate ultra high quality “photon mapped” [7] images in real time. He can view the outcome and adjust things if he doesn’t like how it looks. As with any 3D package he can move lights around and change the camera parameters.

In this manner our future computer user will be able to make an entire movie to almost Hollywood grade visual standards at home, whether the movie is any good is a different matter altogether however. This may not be the future of real movie production but it’ll help speed up the process a lot and will enable TV stations to produce low cost TV series.

Most of this has already been predicted and some has already been done – You think all the armies in Lord of the Rings were real? This is an application where a 3D display with a 3D interface and massive computing power will work to greatly enhance the process. It’s exactly the sort of thing computers will be capable of in the future and will be used for.

Add more interactivity, remove the actors lines and you’ll have a system for producing games. In the future when you play the game of a movie perhaps you’ll just be playing a different version of the movie.

Unemployed Mice

It seems with a 3D interface we are not going to need mice. Perhaps there will be a 3D equivalent or we’ll just use our hands. I think 3D interfaces will enable us to build “proximal” interfaces [8] where the tool becomes part of us, these have the promise of making computers easier to use than ever before.

Whats next?

The next stage to this is where all 3D interfaces will end up if you think about them long enough. I’ll not even bother describing it, I’ll give you a clue instead and let you figure it out: The Matrix.

In the film The Matrix there was a 3D world which everyone inhabited, it is designed to pretty much emulate the real world. The Matrix is the ultimate 3D interface and I fully expect it to happen in some form but something so realistic is obviously a long time away.

We could start with having a “Personal Matrix” on our own computers, this doesn’t have to abide by same conditions or rules as the Matrix in the film, there are no reasons why you couldn’t have it set to be a in a zero gravity environment, on Mars, or both. There are no limitations on how the world could be set up their world other than the power of the computer it’s running on and the imagination of the user.

I do think plugging cables into the back of your head is some way off yet. That said I’ll be reluctant to plug my brain into anything, what happens if a cracker gets in and 0wN3r5 your head? Who would you call? a Sysadmin, Surgeon or an Exorcist?

The Matreb

Another aspect of this system that could be very different from the original Matrix is the concept I’ve named “Matreb” (short for Matrix-Web). If we can have our own worlds why not connect them up? The possibilities here are virtually endless, even bigger than the web. You can build any world you like and invite anyone you want to join in. You could surf the web to a web site about surfing and actually go surfing. You could have a game of “Finding Nemo” where you actually have to go swimming to find him. Ever fancied flying on the star-ship Enterprise?

The computer could be become a lot more appropriate for socialisation than it currently is. Currently the lack of visuals or audio means a lot of the message you are sending can be misinterpreted, it is this detachment that leads to no end of misunderstandings and unwarranted flame wars. In a Matreb you could set up a world with a virtual pub and actually see and hear the people you are communicating with, enabling much more constructive conversation. It’d also be a valuable business tool for meetings and discussions as you could invite people all over the world to meet in the same room, no need to leave their offices. You can imagine this will apply to much more than social or commercial gatherings, again I’ll leave it up to the reader to feel the possibilities of futuristic porn.

Of course we would also get the web’s downsides – spam would be worse, who knows what viruses will be like and pop up ads will be a lot more annoying – of course all these will pale beside the experience of clicking on a 3D Goatse link!

Building a Matreb

Building such a system is going to be a highly complex but I’ve no reason to think it’s impossible. The initial Matrix like system requires a tactile body suit and a very powerful computer, but these will come, we’ll need to go back to Virtual Reality type headsets to create the “being inside” illusion properly but perhaps the screens can be focused away from the wearer to mitigate the focusing problem.

To build a Matrix web is a more difficult problem because it will require the user to interact in a world elsewhere and this will bring up the problems of transferring a large amount of data at very high speed. In order to enter a world you first need to have a full description of it sent and that could be large and complex. After that all the movements within that world will have to be transferred at very low latency, the more realistic the world is the more data will have to be sent.

There are ways around these problems but they are complex and going to need a lot of research and work before we ever see them. The beginnings of this are already being researched and we will see this system developing in the future.

Perhaps one day we wont sit down at a computer to use it but rather plug into it and enter it to use applications. When we do this the computing experience will have changed to something which will not be recognisable today.

But this will still be software running on CPUs, right?

Wrong. I think there are going to be fundamental changes to come in both software and hardware. Not only will the applications be very different but the software they use and hardware they run on will also become unrecognisable. The technologies are all in place, another journey is about to begin.

Stay tuned for part 5…

—————————-

References

[1] General purpose computing with Graphics hardware.

http://www.gpgpu.org/

[2] Not the link I wanted but look at the end of this and you’ll see similar technology.

3D Displays

[3] Depth of Field.

Depth of Field

[4] Nooface has a 3D GUI section, 3dcgi is about 3D graphics

http://www.nooface.org/

http://www.3dcgi.com/

[5] Sun’s Looking glass 3D GUI.

http://wwws.sun.com/software/looking_glass/

[6] The BeOS’s 3DMix.

http://www.skycycleonline.com/images/3dmix.html

[7] Good article on 3D graphics which mentions Photon Mapping and other techniques.

The Future of 3D Graphics

[8] Andrew Basden’s Proximal User Interface.

http://www.basden.u-net.com/R/proximal.html

Copyright (c) Nicholas Blachford February 2004

Disclaimer:
This series is about the future and as such is nothing more than informed speculation on my part. I suggest future possibilities and actions which companies may take but this does not mean that they will take them or are even considering them.

32 Comments

  1. 2004-02-19 8:23 pm
  2. 2004-02-19 8:33 pm
  3. 2004-02-19 8:34 pm
  4. 2004-02-19 8:49 pm
  5. 2004-02-19 8:49 pm
  6. 2004-02-19 9:19 pm
  7. 2004-02-19 9:24 pm
  8. 2004-02-19 9:28 pm
  9. 2004-02-19 10:00 pm
  10. 2004-02-19 10:07 pm
  11. 2004-02-19 10:11 pm
  12. 2004-02-19 10:15 pm
  13. 2004-02-19 10:16 pm
  14. 2004-02-19 10:21 pm
  15. 2004-02-19 10:22 pm
  16. 2004-02-19 10:44 pm
  17. 2004-02-19 11:01 pm
  18. 2004-02-19 11:17 pm
  19. 2004-02-20 12:19 am
  20. 2004-02-20 12:58 am
  21. 2004-02-20 1:28 am
  22. 2004-02-20 1:37 am
  23. 2004-02-20 1:47 am
  24. 2004-02-20 1:56 am
  25. 2004-02-20 2:59 am
  26. 2004-02-20 3:02 am
  27. 2004-02-20 4:15 am
  28. 2004-02-20 5:06 am
  29. 2004-02-20 7:55 am
  30. 2004-02-20 10:09 am
  31. 2004-02-20 8:14 pm
  32. 2004-02-20 10:11 pm