“Microsoft and Perceptive Pixel Inc. (PPI) today announced that they have entered into a definitive agreement under which Microsoft will acquire PPI, a recognized leader in research, development and production of large-scale, multi-touch display solutions.” Yes, Jeff Han is now a Microsoft employee. This demo still amazes me – from 2006. Before the iPhone. Before Android. Before the iPad. Remember that the next time you wind up in a discussion about who supposedly invented what.
I built a multi touch tracking panel, years ago now. There were actually tons of people building these, and it looked so easy I had to try for myself. Just take a pane of glass and put it in a frame full of LEDs on the sides…seal it up and touch the glass, now fingerprints will shine through. Place something partially opaque against the glass to project an image onto (some plastic bags can work, or thin paper). Now you’ve got a touch screen!
Note: They say to use IR so it doesn’t interfere with visible light, but visible spectrum works never the less and could be filtered/tracked easily with a web cam.
One component I needed and couldn’t afford was a video projector, I improvised with a cheap static photo projector just to prove the concept. I still have alot of ideas for MT applications on large surfaces. It looks like alot of multitouch work is being done in universities now.
http://www.youtube.com/watch?v=71BfgXZVBzM
http://www.youtube.com/watch?v=3jxbS1c_HU4
http://www.youtube.com/watch?v=5VNTPwVvLzE
http://www.youtube.com/watch?v=6E36k70cKAw
I can’t wait for cheap hardware to be available (something the size of a wall rather than a monitor). I predicted these interactive surfaces would start to show up everywhere in the public spaces, still hasn’t happened yet as far as I know.
I have a small glass-top computer desk that is screaming to be used as a multitouch input device. It’s slightly larger than the glass panel in the demo that Thom linked (which was freaking awesome btw).
I’m picturing an interface similar to what you see in a lot of sci-fi themed anime, where the glass surface changes to reflect whatever type of input you need at the time: Keyboard, vector manipulation, musical instrument, etc.
I once worked in an acoustics lab where people detected tapping and dragging motion using a flat surface (regular table) with a few cheap piezo microphones stuck on it and a computer program.
It’s crazy that modern touchscreens use something as complicated, unreliable and expensive as capacitive sensing, when so many simpler approaches exist…
Edited 2012-07-10 10:53 UTC
I’m not convinced that a microphoned approach would be any more reliable than capacitive sensing. In fact I’d wager it would be far less reliable as it would be subject to outside interference. It’s one thing having such a technique work in lab conditions, but when you’re at a noisy train station and the phone is in your pocket or trying to detect multi-touch gestures under such environments. Certainly in the for former scenario, you could run the risk of triggering all sorts of false positives. Sure, with enough code you could account for most scenarios, but then you’d create something more complex then detecting the variable voltage change from a persons fingers.
Just to entertain the idea and play devil’s advocate here:
If the system relies on the speed of sound reaching sensors, I suspect that’s usually fairly constant in most environments. I’d expect the air density to be consistent across the surface, windspeeds of 15+ MPH (sorry, Km/H) would be unusual and that’s still only about 1% error against the speed of sound.
Tuning the sensors to a specific frequency should be trivial, and one could even implement frequency hopping and/or multi-spectral tones which are unlikely to be duplicated in any natural environment.
Whether it is superior or not to capacitive touchscreens somewhat misses the point that it may be “good enough” and far more affordable. It could work on natural surfaces too, like walls and floors.
Edit:
Thinking about it further, it should be possible to compensate for the wind-speed issues as well by pairing up the microphones with speakers and detecting the sound waves from the other microphones. This could additionally help automatic calibration and maybe even adhoc placement of the microphones on arbitrary surfaces without any frame at all.
With an appropriate sensor array, I think an acoustic system might be made to work in 3 dimensions. Capacitive sensors can’t really do that.
Edited 2012-07-10 14:38 UTC
Kelvin-meter per Henry? That does seem like an unusual wind
I think you’ve missed my point because wind speed should never need to come into the equation unless you’re using the general purpose front mic (which, quite frankly, is a silly idea).
Laurence,
“I think you’ve missed my point because wind speed should never need to come into the equation unless you’re using the general purpose front mic (which, quite frankly, is a silly idea).”
Firstly, if the mic picks up the frequencies you need, then it’s not silly at all. If it doesn’t, then you’ll need something more specialised, but as a hobby project it makes plenty of sense to start out with off the shelf equipment when possible.
Secondly, any acoustic system will necessarily be affected by windspeed regardless of the equipment you use. I think you may have misunderstood me, I’m not talking about noises caused by wind, I’m talking about the physics of audio propagation and how wind will offset the triangulation of a ping from it’s true origin.
Therefore, IF wind is anticipated (ie outdoors), then it must be compensated or it will affect the accuracy of the system.
Edit:
Neolander is talking about detecting taps on against a board, but I’m talking about tracking a stylus in 2d/3d space using emitors and microphones. It could scale to large natural areas like the football example, or maybe even in swimming pools.
Of course, it makes more sense to use radio instead, but the discussion was about DIY projects, where audio is easy to do.
Edited 2012-07-11 15:08 UTC
you have completely missed the point because we’re talking about this being a possible alternative for embedded systems. Thus everything you’ve posted is completely irrelevant:
Engineering smart phones are not “hobby projects”. So nothing you’ve said here bares any relevance to anything I was discussing earlier.
I repeat: the mic would be inside the casing. You simply wouldn’t want nor need to use the general purpose mic for this. Thus, unless the phone itself has it’s own internal weather patterns, then wind speed is not an issue what-so-ever. Period.
Exactly, you’ve missed the point of what we’re discussing. What you’re chatting about is completely irrelevant to anything which I was discussing earlier.
Fair enough this idea has scope beyond what I’m specifically chatting about (I never disputed that – in fact I’ve actually stated this already), however don’t tell me that my points were wrong when you’re quite clearly chatting about a whole other topic than I am.
So please, if you want to disagree with me (and you’re more than welcome to
), then please do so with the same context to the points I raised. If you want to discuss a related tangent (as you have done and which is also an interesting topic), then don’t raise it as a counter argument. That’s all I ask 
Edited 2012-07-11 15:53 UTC
Laurence,
Thank you for clarifying, however I already know what your talking about. Please quit being so patronising and insisting that I am the one missing the point. I am the OP of the thread, we’re covering a variety of approaches for DIY tracking – you’re obviously welcome to criticise whatever you like and better yet come up with your own ideas.
Edit: Maybe I’m being too abrasive, but I don’t like being spoken down to and it feels like that’s what you are doing.
Edited 2012-07-11 16:52 UTC
You may have been the OP of this thread, but you were not the guy i was responding too. Thus whatever context you opened with was irrelevant.
As I said before, it’s not the criticise from you that I objected about (I welcome such comments), it’s the fact that you were completely missing the points I was making (as we’ve now established).
As I said before your ideas are interesting, they’re just not related to anything I had posted.
Anyhow, we’re not straying right off topic now so I’m going to shut up
Laurence,
It’s probably better for me to ignore this, but human nature…
My source of frustration here is continued accusations of “missing the point” and being “irrelevant” based on a faulty understanding of what I meant. My response to your claim that a microphone wouldn’t work may not have been what you were expecting, and you can fault me for not having expressed myself very clearly, and even for changing the goal by adding an emitter to the mix. But that’s truly the first thing that popped into my head when considering the problem of low amplitude tapping in that project. To say I’m missing the point over and over again is totally uncalled for.
When you’re counter arguing a post I made with points about a topic I wasn’t discussing, this it is somewhat missing the point of my post. “Apples and oranges”, as some say. So I’m sorry if that sounds condescending, but that is the simple crux of it.
As I said before, you made some very interesting points, stuff I’d enjoy chatting to you about any other time. But they were not related to my post. And this is why I got frustrated and made my remark.
So I’m genuinely sorry if I’ve annoyed you. I do have a lot of respect for the ideas you were presenting, I just didn’t see how they were related to the point I was raising.
Laurence,
I know it’s “apples/oranges”, it was from the get go. I think a misunderstanding over my hastily worded post was wrongly interpreted as my misunderstanding of yours. That doesn’t mean I missed the point, which is what’s annoying me, but you seem willing to drop the matter, so I can too.
My brain naturally goes on tangents all the time, pulling out ideas that are well outside the unspoken borders of the discussion, and if I don’t filter myself I’ll sometimes say things that are orthogonally related to the discussion. Anyone who knows me well will probably recognise this instantly. I do it all the time and I can understand how that can be annoying and even counter-productive sometimes, especially when someone else wants to bring the discussion in a specific direction. I’m actively trying to be clearer when I talk, however that goes out the window when I’m in a hurry. It’s not a good excuse though, and I certainly need to work harder at it.
Edited 2012-07-12 13:51 UTC
To be honest mate, being able to think on a tangent like that is the best way to be.
Outside noise is not necessarily that much of an issue. If you stick the piezos inside of the device’s casing, they will be far, far more sensitive to finger tapping and friction on the device’s casing than to outside noise, since there’s a lot of acoustic reflection going on at the air-casing interface. That, and acoustic power decreases as the square of the distance that is travelled from the source. To see how big the difference is, just gently tap the surface of a microphone when recording sound : you should get a peak that is far more intense than anything you were recording, even for a very gentle tap.
Pockets are indeed an issue, but on this front microphones are just as bad as capacitive touchscreens, which will also run amok if you forget to turn off your phone before putting it in your pocket. One specific pitfall of acoustic sensing is friction on the back or the sides of the device during device use though. Accounting for these could indeed require complex noise cancellation systems.
Edited 2012-07-10 15:09 UTC
Or it could be exploited to expand possible control schemes (like PS Vita with a touchpad on its back; and, IIRC, some time ago we discussed how this could be one of the approaches for “more 3D” control than touchscreens allow)
I do remember that
The problem, however, would be to differentiate taps from below or the sides from taps from the top.
If you have only one homogeneous flat surface (as with the regular table with mics stuck on it that I saw), detecting taps on it is a simple 2D problem. Discriminate microphone signal pulses from background noise, find out correlations between them through signal processing algorithms, and you can estimate relative propagation delays. Then, knowing the speed of sound inside of the surface, you get relative spatial positioning information, that can itself be turned into absolute spatial positioning given some hardcoded calibration.
Now, if you are dealing with a thin box full of electronics (in which sound does not propagate homogeneously) and want to detect taps and friction in a 3D space, things become quite a bit more complicated. You need microphones in other places than the screen, and at least a rough model of how sound propagates inside of the device. Which is why I said that this could be an issue if we were to try to integrate acoustic touchscreens into small devices.
Edited 2012-07-11 12:06 UTC
Of course; the microphones would need to be throughout the thing, interpreted through quite a bit more complex model (maybe also aided by most of the case being one solid shell relatively isolated from the insides, or maybe a structure with “sound barriers” at the edges to limit round-trips of sound, or maybe different surfaces of the device having slightly different texture hence also sound) – but I imagine it would be doable, and not significantly harder (and, what particularly matters: not significantly more expensive in mass production)
But ultimately, only one way to find out – experimental research
Edited 2012-07-17 23:52 UTC
I understand the acoustics of it all (I’ve spent a lot of time working with mics as well). While I appreciate that outside noises isn’t a huge issue most of the time, it will be an issue some of the time. For example, in night clubs where the sound system is well tuned for punchy kick drums. Such a touch pad under those conditions could be worse than useless.
You also then have to account for the amplitude of the sound (depending upon how heavy handed the user is), the different refractive properties of the casing for each unique phone handset and the subtle differences in the tune of the touch depending upon the make up of the casing (crude example: tapping a metal table produces a different sound to tapping a plastic table).
So I just think the amount of code that would have to go into understanding the difference between sounds originating from physical contact and those from external sources could amount to a more complex solution than having sensors detect physical contact via the resistance of the users finger. Though I will grant you that some of the above can be configured at run time by a control panel-type app – but even then, that means you don’t have an “out of the box” solution unlike with the current method.
Further more, attaching a new phone case (eg rubber sleeve or patterned cover) would then also change the acoustics and thus potentially break the touch pad. Which would be a terrible step backwards in terms of usability.
Don’t get me wrong, I think the mic’ed method is a fascinating idea and it sounds like great fun for a lab / home environment. I just think it’s not better (or even practical) than current capacitive inputs.
Regarding night clubs, I don’t know if that would be enough to perturbate an acoustic touchscreen, but I agree that doing the experiment with a prototype would be a good idea…
I think that tap amplitude and tune are not very important for this design, however, as long as the following conditions can be met :
1/It can be assumed that the acoustic properties of the screen are uniform and there are some absorbers on the sides to address the problem of reflections
2/It is still possible for the mics to detect a reproducible signal that is reasonably far above the noise level
That is because, for the design which I have seen at least, finger position tracking is based on sound propagation delays, not attenuation or wave shape. If we detect a pulse on mic 1 at a time t1, on mic 2 at time t2, and on mic 3 at time t3, we can measure t2-t1 and t3-t1, then deduce spatial positioning information using such calbration data as the speed of sound inside of the screen material and the well-known mic network geometry.
I would gladly draw you a sketch that explains this in more details, but my main computer is currently undergoing repairs and cellphones are not very good for that kind of things.
Since we are only interested in the direct signal flowing from the impact region to the mics without undergoing reflections, rubber casing should not be a problem. In fact, they could even help addressing the problem of reflections and the one of friction on the back and the sides of devices that I mentioned above.
To conclude, I am not sure that acoustic touchscreens could play the same role as capacitive touchscreens either. The guys who worked on them at that lab preferred to market them as a path towards cheap large touchscreens (such as touch-sensitive shopwindows). I was just playing with the idea in thoughts experiments to see what would be the advantages and drawbacks, and found that it might work surprisingly well in that scenario, with much simpler hardware than the capacitive electrode mesh insanity. But I’d probably need to build a prototype (which is way beyond my knowledge of embedded electronics) in order to test some of the potential issues that have been raised here.
Edited 2012-07-11 20:43 UTC
Neolander,
Do you think acoustic reflection is that big a deal for software to handle in your model?
Granted, I don’t know how much attenuation to expect, but I’d be surprised if it’s difficult to differentiate between the primary and reflected signals. My guess is that secondary waves would loose high frequency components far more quickly than low frequency components.
Instead of trying to absorb/eliminate reflections, we may be able to use the information from reflections to our advantage. Consider a sensor on the right hand side of the table that picks up a pulse, shortly followed by one on the left hand side. Ordinarily the algorithm would measure the time lapse difference and register a corresponding touch on the right side. However the pulses may have been triggered by environmental sources off the table.
If the reflection isn’t too week, we might be able to prove that the pulse originated on the table because from anywhere else the timing of reflections would be off.
Instead of attempting to write an explicit algorithm for this, it might be easier to use a generic pattern matching algorithm and perform a hough transform across it. This would produce a 2d graphic with intensities corresponding to the areas that have the strongest matches. This could be projected onto the table so that it gets lit up in the spots where it is touched. Then try to fool it with external noise.
Edited 2012-07-11 21:19 UTC
Well, theoretically-speaking, a few reflections on the sides of the screen from time to time are not that big of a deal. Without even going into waveform analysis, they would be detected as taps coming from outside of the screen, which can be eliminated by means of simple bound checking.
The first potential issue with reflections, however, is when you have lots of reflections going on with little damping. In this case, standing waves of significant amplitude would form inside of the screen after a short while, and potentially bring a significant amount of noise into the mics for a few seconds (think of a guitar string), which is bad news.
Another potential issue is reflection on the back and the top of the screen. These should probably be quickly damped, but if they aren’t, they would result in the detection of multiple taps on a direction of space which the mics aren’t sensitive to. I don’t know if there would be an easy way to ignore those, maybe an heuristic like “Ignore taps which originate from nearly the same position within a delay smaller than 1 ms” could work.
I don’t have enough of a signal processing and acoustics background to know whether your proposal to discriminate taps from the outside from sound from the inside would work, but spontaneously I would think that once an exterior pulse has hit the screen at some point, hard enough for mics to detect a signal, the perturbation would propagate inside of the screen just like a regular tap.
I follow what you’re saying so no diagram is required (thanks for the offer though).
Actually, you’ve raised some interesting points there – to the point of nearly convincing me (it’s a real pity I can’t up vote once I’ve posted).
I don’t think there’s necessarily a “right” or “wrong” answer though, at least not without building a prototype and testing it (sadly it’s above my level too). I do think you’re right about the huge potential of using such a solution for larger surfaces though (if it’s not already in wide-spread use).
Is this how some of the marker whiteboards work?
It’s a standard whiteboard, nothing electronic in it, used with special markers and a detector on the side. It records everything written on the whiteboard to a program.
I’ve seen this same system hooked up to projection screen and used to interact with a windows desktop.
There’s a simple way to test this hypothesis : tap the whiteboard with your finger. If the surface is soft, you are likely dealing with a resistive touchscreen. If nothing happens, the system likely detects the stylus in particular, which is generally done using Wacom-like inductive sensors. If your finger tap is indeed detected, you may be onto something there…
Neolander,
“There’s a simple way to test this hypothesis”
I don’t have access to one now, so I can’t tell you.
Up to now I was thinking about the “pen” containing an emitter and having sensors on the side. However it seems the opposite should be equally feasible and far more scalable.
An example might be to go to a football field, broadcast different frequencies at each corner and place a wireless microphone on players. A computer could listen to an arbitrary number of mic’s on the field, pinpointing them on a map by analysing audio timing and/or phase differences. Kind of like a localised “GPS” built out of commodity audio equipment.
This could be used to help in statistically analysing athletic behaviour and recognising trends to gain a competitive advantage. Even speed could be recorded accurately using doppler effects. Probably been done, but would be neat to experiment with.
well, that might have worked in a lab, but think about real life: noise interferences and all the problems that a “cheap piezo” would have when exposed to daily heavy usage.
capacitive sensing is not at all unreliable, just take a look at how well it works on an ipad or similar devices. by now it’s the best technology we came up with for touch surfaces. too bad it’s still expensive and not usable on large surfaces.
I’ve already discussed the issue of noise in another reply, so I’ll just explain what makes capacitive touchscreens so unreliable as compared to all other touchscreen techs : their sensitivity to water. If your screen becomes humid for any reason (rain, sweat, freshly washed hands…), the device will start become unresponsive or register false positives in a matter of minutes. Since when do we build mobile devices that can’t stand a few drops of one of the most widespread chemicals of planet Earth ?
And there is a reason why the tech is so expensive and sensitive to external perturbations too : it requires ridiculously complex hardware by its very nature. As if resistive touchscreens and their need for fine-tuned mechanical properties weren’t complicated enough, the best which hardware engineers could come up with as a successor was a fine mesh of transparent electrodes stuck in the tiny space between a LCD and a protective plate, following the tiny capacitance difference induced by the presence or absence of a human finger milimeters away while shielding itself from its direct electromagnetic environment somehow ? Honestly, if they just wanted to come up with something bizarre enough that it would create tons of jobs in R&D and stimulate the economy, they could have stated the goal right away…
Edited 2012-07-10 15:35 UTC
Some folks at Nokia Research Center Tampere built such style of touchscreen out of ice, some time ago:
http://www.youtube.com/watch?v=bbtrI6GjBsk
http://research.nokia.com/news/11362
Not sure how useful the experiment was, for Nokia, but it looks like they had some fun (so perhaps team-bonding usefulness)
And in the related of the above video, there’s http://www.youtube.com/watch?v=O7ENumwMohs – ~AutoCAD on… Perceptive Pixel.
Bringing back the drafting board could end up awesome (but make it properly tilted…), when expanding it further in ways which ~desktop UI cannot (present CAD user interaction definitely also lost something when “upgrading” from drafting boards).
Similar with music software, video editing, photos. Perhaps that’s what MS sees with Metro (and forcing the thing now, a bit, so the software will be ready when inexpensive large displays arrive in a few years) – sure, it’s a bit awkward now, but their desktop OS also really took off only with 3.x…
BTW, webcam sensors are sensitive to near-IR, and many webcams can be easily modified to see it: just remove the IR filter in the lens (or, to see it only: replace the IR filter with visible light filter that lets through IR – the dark areas at the ends of photographic film work decently, IIRC)
Edited 2012-07-10 17:29 UTC
I really don’t. I prefer the ones who take a certain technology, iterate over it, ship a real product and make it available to people at a reasonable price.
I interviewed with this company a few months ago and I really liked the small startup atmosphere. It was very informal and relaxed. Hopefully they don’t lose that.
That is one demo Samsung and others need to bring to court with them, in the US just file a memorandum of law with that demo as the subject and no court can block it as evidence.
Jeff Han gave a cool W8 on Perceptive Pixel 82-inch screen at the WPC a couple days ago:
http://www.digitalwpc.com/WPC2012/Pages/WPC2012.aspx#fbid=6BHzisCbW…
Starts at the 1:54 mark.