Projection/Vision Devices

William Buxton
Buxton Design/Gallery 888
888 Queen St. East
Toronto, Ontario
Canada M4M 1J3
+1 416 465-3836
http://www.billbuxton.com
Created: May 16, 2003
Last Revision:  May 29, 2003

Note:

This is a kind of "open source" document.  I have collected this information and posted it.  I will try to keep it up to date.  However, like many things, the field is changing quickly, and time is not infinite. Hence, I will miss things.  Therefore, I encourage those interested to email me with  suggestions, corrections, or anything within reason that would improve the value of this page.

Introduction

Just like every conventional loudspeaker can also be used as a microphone, for some input devices there is a complimentary form where they can also be displays.  However, just as few loudspeakers are used as microphones (so few, in fact, that most people forget - if they even knew - that this was possible), very few input devices incorporate this duality into their design.  Force feedback devices are one exception.  With them, the "display" is felt rather than seen.  Touch screens and other direct input devices appear to have this property, but in fact, this is appearance only, since their input/output duality is accomplished by designing two separate technologies into one integrated package.  The acoustic analogy would be integrating a microphone and speaker into one package, a bit like a telephone handset, rather than using the same transducer for both the microphone and speaker functions.  It is interesting to note that this is not the case with force feedback devices since with them, the same motors that generate the force output also serve as the encoders that capture the actions of the user.

Recently a new class of device has started to emerge which is conceptually rooted in exploiting this input/output duality.  They can be called Projection/Vision systems,  and/or Projection/Scanning or Projection/Camera technologies.  In the "pure" case, these are devices that use a laser, for example, to project an image of the input controller - such as a slider or keypad - onto a surface. In doing so, they are performing a function analogous to an LCD displaying the image of a virtual device under a touch screen. However, in this case, the laser is also used to scan the same surface that it projecting onto, thereby enabling the device to "see" how your fingers, for example, are interacting with the projected virtual device.

In a slightly less pure "hybrid" form, the projection and scanning functions can be performed by two separate, but integrated technologies.  For example, instead of a laser projector, a conventional video or data projector could be used, and an integrated video camera (supported by vision software) used for input.

Both the "pure" and "hybrid" classes of device have been used and have strengths and weaknesses.  Since laser projection is far less advanced than conventional data projection, the hybrid solution sometimes has advantages on the display side.  However, 2D and 3D scanning using lasers is far more developed than 2D and 3D vision using video based vision techniques.  This is partially due to the degree to which the laser technology can extract 3D information.  Going forward, one can expect laser projection technology to advance extremely quickly, especially in its ability to deliver extremely small, low power, bright, relatively high resolution projection capability.  This will likely have a strong impact on how we interact with small portable devices, such as PDAs, mobile phones and even wrist watches.  Not only does this technology provide a means to couple large (virtual) I/O transducers with small devices, it provides the potential for sharing and interacting with others, despite using devices as small as a wrist watch.

On the other hand, these technologies have strong potential on the other side of the scale, in large-scale interaction, where what is scanned are bodies in a room, rather than fingers on a surface, and the projection surface may be the floor or ceiling of a room, rather than a desktop.

Besides the obvious, there are a couple of interesting challenges with this type of system.  First, it is generally not sufficient to simply know where the fingers are over the display.  One has to be able to distinguish the difference between pointing or hovering, versus activating.  (For background, see A Three-State Model of Graphical Input.)  This must be reliable, and responsive.  And, to avoid "the chess player's conflict" ("You touched that piece!", "No I didn't!!") the system and the user must agree as to if and when activation takes place.  Also, since the device is virtual, a means (acoustic of visual) is likely needed to provide some form of feedback at the device level.  Since, especially in the mobile case, the projection surface, and hence the input control surface, is arbitrary, so there would be no opportunity for any tactile feedback, vertical or lateral.  Of course, if the projector was fixed, then there are a range of techniques that could be used to provide tactile feedback.

Electronic whiteboards that use projection technologies coupled with touch screens, such as those available from Smart Technologies, and 3Com, for example, are related to this class of device.  However, they differ in that the input transducer is integrated with the projection surface, rather than with the projector.  This is a significant technological difference (but one which may be transparent to a user).  The same could be said of touch screens, especially in the future as touch screens become thinner and more inobtrusive, such as if/when they are made with OLEDs, for example.  That is, they could appear the same to the user as "pure" projection vision systems. However, I treat touch screens and this latter class of projection boards separately.

What is unique, dintinct, or new, from the usage/user perspective of the type of projection/vision systems that I highlight in this section is that they are not fixed in position.  The same unit may project/sense in different locations, on different surfaces, and in many cases be mobile.  That is, there is no specific surface, other than the (perhaps) arbitrary surface on which one is projecting, on which the system operates.  This is especially true of the miniature laser projector/scanner systems.  But is even true of installed systems, such as the IBM steerable projection/vision system.  In this later case, while the projector and vision systems are fixed in architural space, they can be directed to work on different surfaces/areas in the room..

Projection/Vision systems constitute an area where products are beginning to emerge.  Below is a listing of some of the companies who are playing in this field.  As well, there is a body of work emerging from the research community around this type of interaction.  (Actually, there is a relatively long, albeit sparse, history already out there.)  For current work, the next clearing house for research in this area that I am aware of is the IEEE International Workshop on Projector-Camera Systems to be held Oct. 12, 2003 in Nice France.
 

Companies/Products/Technologies

Canesta Inc.

Canesta Inc.

The Canesta Keyboard is perhaps the best known laser projected keyboard.  The way that the projector works is like a slide projector.  The laser does not move.  Rather, it illuminates a hologram-type slide which diffuses the light in the pattern of the keyboard.  Hence, nothing in the image can change unless you change the slide.  This makes things simple and relatively inexpensive, but it also means that the projected display cannot be used for anything but a keyboard.  It cannot, for example, be a substitute for the devices smaller LCD display.

The keyboard is implemented by a separate optical system mounted at the base of the unit. It senses the fingers breaking the invisible light fanning out from the light source, close to, and parallel to, the desk top. Software determines when the fingers have broken the fan of light, and where they have done so, thereby determining which virtual key has been struck.

Details of their system can be found in:

Tomasi, C., Rafii, A., & Torunoglu, I. (2003).  Full-size projection keyboard for handheld devices.  Communications of the ACM (CACM), 46(7), 70-75.

While one of the early players in this space, the company appears to have abandoned it in favour of developing 3D imaging technologies.  See VKB, below, for a company offering essentially the same thing that Canesta did.

Canesta, Inc.
2833 Junction Avenue, Suite 200
San Jose, California 95134-1920
USA
tel: +1 (408) 435-1400
http://www.canesta.com

InFocus

InFocus is one of the leading companies in providing video and data projectors.  Their projectors are conventional, in that they do not use laser technology.  This has that advantage of delivering high quality colour images with a mature technology. However, it has the disadvantage of larger size, lower contrast, and higher power requirements, compared to laser projection systems.  In 2000, InFocus merged with Proxima, which had been one of its competitors.  I include InFocus/Proxima in this survey not only because they make projectors.  In their early days, Proxima developed one of the first commercially available projection/vision systems.  It was called Cyclops, and they still hold a patent on the technology.  Cyclops augmented the projector by adding a video camera that was registered to view the projection area.  The video camera had a bandpass filter over the lens which passed only the wavelength of a laser pointer.  The system, therefore, enabled the user to interact with the projected image, using a provided laser pointer as the input device.  The camera detected the presence of the laser pointer on the surface, and calculated its coordinates relative to the currently projected image. Furthermore, the laser pointer had two intensity levels which enabled the user to not only point, but to have the equivalent of a mouse button, by the vision system interpreting the two levels as distinguishing button up and down events.
InFocus
27700B SW Parkway Avenue
Wilsonville, Oregon,
USA 97070-9215
tel: +1 503-685-8888
tel: +1-800-294-6400
info@infocus.com
http://www.infocus.com

Senseboard Technologies AB

The Senseboard SB 04 technology is an extreme case of a hybrid approach.  The sensing transducer is neither a laser scanner nor a camera.  Rather, it is a bracelet-like transducer that is worn on the hands which captures hand and finger motion.  In fact, as demonstrated, the technology does not incorporate a projection component at all; rather, it relies on the user's ability to touch type, and then infers the virtual row and key being typed by sensing relative hand and finger movement.  The system obviously could be augmented to aid non-touch typists, for example, by the inclusion of a graphic representation of the virtual keyboard under the hands/fingers.  In this case, the keyboard graphically represented would not be restricted to a conventional QWERTY keyboard, and the graphical representation could be projected or even on a piece of paper.  I include it here as it is a relevant related input transducer which could be used with a projection system.  The technology has patents pending, and is currently in preproduction proof of concept form.
Senseboard Technologies AB
P.O. Box 1077
SE-164 25 Kista
Sweden
tel: +46 8 555-272-01
info@senseboard.com
http://www.senseboard.com

Symbol Technologies Inc.

Symbol technologies is the world leader is laser barcode scanner technology.  They have recently announced a miniature Laser Projection Display (LPD).  The technology is in the preliminary concept stage, rather than a product.  The technology described is projection only, and does not currently incorporate a scanning/vision component.  However, it is relevant to this class of devices, and is described as being relevant to incorporating projection displays into portable devices such as PDAs and cell phones.
            

Symbol Technologies Inc.
One Symbol Plaza
Holtsville, New York 11742-1300
United States
tel: +1.631.738.2400
tel: +1.800.722.6234
fax: +1.631.738.5990
info@symbol.com
http://www.symbol.com/products/oem/lpd.html

Virtek Vision International Inc.

Virtek Vision is an example of how projection/vision systems can scale.  Most of the technologies discussed on this page are concerned with providing relatively large (on the order of 10-30 c.m.) virtual display and input transducers to small portable devices.  Virtek works at the opposite extreme.  They use laser projection systems for very large scale functions, such as in manufacturing large wooden or metal parts.  In this case, their technology is used to project patterns or templates onto the materials.  They also provide technologies for using laser-based vision systems to do quality assurance.  They are not currently applying their technology to what we would normally call "human computer interaction", and they would probably not consider themselves as enabling "virtual interactive devices," nevertheless, their technology is relevant to the class of system being described.

Virtek Vision International Inc.
Waterloo, Ontario
Canada
tel: 519.746.7190
fax: 519.746.3383
info@virtek.ca
http://www.virtekvision.com

VKB Ltd.

Have developed a proprietary infra-red (IR) technology that captures hand interactions with a projected virtual device.  They are a technology company that appears mainly to be involved in licensing chipsets, optical components and designs for which they have applied for patent protection.
 
VKB Ltd.
5 Nahum Hefzadi,
Beit Ofer
Jerusalem
Israel 95484
tel: +972-2-659-5611
info@vkb.co.il
http://www.vkb.co.il/

Virtual Devices Inc.

Have a patented technology, the Virtual Keyboard, or VKey, that incorporates laser projection with a "camera" to enable you to work with a virtual keyboard.  They currently have working prototypes but are not shipping product.
Virtual Devices Inc.
2661 Clearview Drive
Allison Park, PA
USA 15101
tel: +1 412 492-8500
Peteh@virtualdevices.net
http://www.virtualdevices.net/


Related Links


http://www.procams.org
http://www.wirelessnewsfactor.com/perl/story/17756.html
http://www.wirelessnewsfactor.com/perl/story/14762.html