Figure 1: Four tiers of media space study
The Ontario Telepresence Project is investigating the effectiveness of media spaces in the work environment. An important part of the research involves prototyping new media space applications, and evaluating them through experimentation and field studies outside of the laboratory environment. As researchers apply user-centred design techniques towards the development of the social interface, testing the systems in the field becomes a critical research component. The problems and logistical issues of undertaking media space field studies are becoming more important. For the past two years we have been increasingly involved in this kind of study. In this paper we report on some of our practice and experiences in order to benefit others who may be about to embark on this type of research.
KEYWORDS: Multimedia, media space, desktop videoconferencing, computer supported collaborative work, groupware, social interaction, user interface, field studies.
The Ontario Telepresence Project is a three year joint government, university and industry research project investigating the effectiveness of media spaces in the work environment. We design, build, test and use technologies in our own facility which explore new concepts in media space design. An important part of the research involves prototyping new media space applications, and evaluating them through experimentation and field studies outside of the laboratory environment.
As researchers apply user centred design techniques towards the development of the social interface, testing the systems in the field becomes a critical research component. The problems and logistical issues of undertaking media space field studies are becoming more important.
For the past two years we have been increasingly involved in this kind of study. In this paper we report on some of our practice and experiences in order to benefit others who may be about to embark on this type of research. After a brief review of previous work, we outline a four-tiered framework within which our research is organized. These tiers move the research progressively from contained local tests and experiments to remote arms-length field studies. We discuss the structure in terms of both its practical and methodological implications - in particular how one can better incorporate user-centre design in this class of system.
While we argue that testing systems in the field is important, our experience thus far has made it abundantly clear that doing so is fraught with problems, especially when resources are limited and time is short. Consequently, a part of our discussion reports on the roles assumed by the various players so as to minimize overhead, and improve the quality of the study.
Our hopes are that the practice and experience reported here "from the trenches" will benefit other researchers who may be about to embark on this type of research.
The current round of media space research began in 1985 at Xerox PARC [25], though earlier exploration by artists Kit Galloway and Sherri Rabinowitz [10] and by AT&T is well documented. Recent years have seen a marked increase in research efforts on media spaces and their application in support of collaborative work. Some of the research facilities involved in media space research include Bellcore [9], Xerox PARC [2], Rank Xerox EuroPARC [11], Olivetti [15], and CAVECAT [20].
Investigations of media spaces have traditionally fallen into two categories: relatively short experiments and laboratory studies [12,13,23,24], or longer term self-investigation [2,9]. All of these studies have been internal to the respective organizations, and have provided valuable insights into the impact of media spaces on work behaviour.
In 1992 part of the CAVECAT project evolved into the Ontario Telepresence Project, a joint government, university and industry research project. A large part of our research involves prototyping new media space applications, evaluating them through experimentation and taking them to field study. The primary focus of the project is on issues of shared presence, with secondary attention on shared task spaces [4,16].
The project is distributed between two distant geographic locations, with one site in Toronto and the other in Ottawa. The team is deliberately not co-located in order for team members to experience working in the media space we are implementing. In this way we are able to continue our investigation of media spaces through practice and through controlled experiments.
In earlier work we evaluated the effectiveness of a media space within our own organization. Other research projects, such as those previously cited, have similarly focused on in-house use. In many cases the subjects have had a vested interest in the use of the systems, often because either they themselves or a colleague have been involved in developing the technologies.
This initial in-house testing is fundamental to the development of the technologies, but it is not sufficient to make broad claims about the success or failure of the techniques. To evaluate more thoroughly the value of media spaces it is essential to move the technology beyond the laboratory, into other workplace settings. The dilemma is how to do this early enough to influence the design of commercially developed systems, yet no robust commercial systems exist with which to conduct a comprehensive study. We needed to develop a strategy for providing field study sites with robust hardware and software without the expense of developing a commercial product.
Part of our strategy for dealing with this dilemma has been to adopt a four-tiered model of evaluation: Experiments and Laboratory Study, In-House Study, Wrist's Length Study, and Arm's Length Study. (See Figure 1). This model allows us to investigate media spaces both in our lab and in other workplace settings.
In experiments and laboratory studies we place subjects in a controlled environment and assigns them specific tasks [23,24]. This type of study often uses groups of two to four people. Subjects are not performing ordinary work tasks, and are not usually in their work environment. Frequently the subjects have no previous exposure to media spaces, and have less than an hour of time in the environment. The tasks the subjects perform must be quite simple, since there is little time for them to become familiar with complex tasks, or to form comprehensive cognitive models of the media space. Any user interface must be trivial to learn. Laboratory studies allow the researchers to hold all other variables the same, and measure the effect of changes to the media space. We use both formal experimental method and informal experiments at this level. Other media space research projects also engage in this type of study [12,13].
The hardware and software installations in our laboratory are quite complex, but allow a great deal of flexibility. We can rapidly reconfigure the resources to test novel concepts and to run experiments. The compexity and flexibility is essential for an experimental system where the range of potential uses are unknown in advance.
At the second level we investigate the impact of media spaces on ourselves. These investigations are different from laboratory studies in several ways. They allow a larger population to be studied, over longer periods of time, where the subjects are involved in actual work related activity. The better, more practical ideas from the experimental level migrate to in-house practice. The subjects have an advanced competency level with technology, and are quite familiar with the technical structure of the system. Their level of comprehension of the system allows us to supply them with applications that have sophisticated feature sets. We are able to make changes to the system quite regularly, and it is not uncommon to have the behaviour of certain system features change weekly during heavy development cycles. The users are usually tolerant of change, don't require lengthy training, and have little necessity for manuals. Technical installation and support is conducted by the system developers. Users frequently take responsibility for enhancing the system.
This is by far the most common type of study in media space research [2,15,20](??? OLIVETTI). For in-house studies the researchers use systems that they are themselves developing. In some cases the systems are used by other colleagues in the same organization [9].
There is still much to be learned within a research environment and from users familiar with the system and tolerant of change. This user group is willing to use and test application software and interface designs that are still under development. The facilities are more complex than necessary for any particular experiment, to allow us to quickly reconfigure the environment for new experiments, and to explore feature sets in our software. The offices are well equipped with technology, and little attempt is made to make the technology unobtrusive. We keep the devices and wires accessible for rapid reconfiguration, with the unfortunate side effect of cluttering and intruding on the workspace.
We believe that most research in media spaces to date falls into one of these first two categories. The next two categories of study involve media spaces outside a laboratory environment, and the implementation of these studies differs drastically from the others.
The field studies at wrist's length place media spaces in a workplace outside our laboratory, in an organization that has a working relationship with our researchers. The site is sympathetic to our research endeavour, yet is not directly involved with nor has vested interest in it. The sites considered to date have had offices in two or more locations, where there was a requirement for close collaboration between employees across sites. These sites are small, with between 10 and 30 people expected to participate in the media space.
Our system is being used by many organizations (Telepresence, Xerox PARC, Bell Canada, Eurecom and others). Each of these sites are involved media space research, and have highly technical staff to support their installation. A site at wrist's length may not have such advanced technical support. Their main interest is simply to provide a new method to support communication between their geographically separated employees. Since their communication requirements are very different than ours, we must develop custom applications which match their needs, usually a subset of the tools we use in-house.
Our relationship with wrist's length sites allows us to occassionally try new concepts, make changes to the system, test new questionnaires, and generally have a greater presence in their environment than would be appropriate for sites at arm's length. We work with these sites to test installation and maintainance procedures and the useability of our various manuals. At the same time we are undertake longitudinal studies to acquire useful data about acceptance, usage, and social impacts of the media space. We are getting from subjects without any vested interest in the research.
At the beginning of studies at wrist's length we work closely with the study site to provide technical support, counseling on installation, and training on the system. The software is be relatively robust, but the site is are aware they are a beta-site of sorts, and are expecting occassional modifications and enhancements. At the end of the studies the software is very robust, the Telepresence project withdraws, and the system continues to be operated and maintained by the field study site.
The research methods and design we use will vary from study to study, but can be broadly characterized as follows: experimental method, surveys, user and useability surveys, field research and network analysis. More details can be found in [gmoore].
The study at arm's length is a full-fledged field study in a workplace that is completely unrelated to us. An example of this might be a financial institution, a natural resource concern, or a government agency. These sites will have between 30 and 100 subjects connected to the media space. They will have their own technical staff who will install and support the installation, and Telepresence will function as consultants to them in this regard. The technology fielded must be more robust, compact, and efficient than in previous levels. There will be a simplified user interface to the media space. A complete set of manuals will be required by the site before the study begins, and may be required to encourage their decision to become a study site. We will undertake longitudinal studies that were refined in the earlier studies at wrist's length. When the field study comes to an end the system will continue to be used, operated, and maintained by the field study site.
We are currently in the planning stages for an arm's length field study.
Field studies of media spaces bring together researchers from many different disciplines, each of whom have distinct research practices. The studies at wrist's length provide an opportunity for methodology design and team building among the various disciplines in the field trial efforts, and they afford the incremental acquisition of skills and confidence to support the more difficult arm's length studies. Studies at wrists' length provide us with a vehicle to incrementally learn the techniques of field trial deployment and execution. Therefore, it is the strategy of the Telepresence Project to start studies locally and, as the underlying test-bed, applications and social science techniques mature, move toward sites that are more at arm's length from us.
The deployment of a media space outside the laboratory environment is entirely different than experimental in-house use. In Figure 1 the four levels of study are represented as four concentric circles. As the circles get larger the following become more true:
• the technology fielded must be more robust, compact, less functional and more constrained.
• the cost of doing the study and the planning time increases.
• the users will become less "friendly" to the scientific goals of the project and more concerned with the way the system affects their workplace and daily tasks.
• the less control we have over the users and their environment and an increased risk of scientific failure due to confounding causes.
• the value of the information to our industrial partners increases.
Some additional characteristics of field studies are:
• the participants in the studies must be able to continue in their normal job functions with minimal personal disruption.
• issues of cost, support, aesthetics and ethics become factors in selecting and installing equipment.
Several factors affect the selection of equipment for use in offices.
Cost. The study site finances the purchase of equipment. The same economic pressures that guide all of their purchases affect their decisions here. The least expensive video monitors available to us today are 13 inch consumer televisions.
Aesthetics. The equipment must "fit in" at least as well as any other technological device in the subjects' personal environments. One of our goals is to make the appliances invisible and ubiquitous. We select equipment to fade into the background.
Features. Cost takes precedent over features, but the following elements are important to consider. We look for equipment with the fewest dials and knobs. They reduce the intimidation factor and discourage subject from making unecessary adjustments. Fewer controls means there are fewer opportunities too push the wrong button.
Theft. Until recently the least expensive colour video cameras available to us were small consumer camcorders, which are very attractive to would-be theives. We now one of the several small, inexpensive, and commercially available cameras designed for desktop video conferencing.
Audio quality. It has been suggested that the audio channel makes more of a difference than the visual channel [pye] and our own anecdotal evidence is in line with these observations [mantei]. We find it entirely unacceptable to use the speakers in most television monitors, as their frequency response and tendency to send the sound in all directions make it difficult to prevent feeback. We use professional audio equipment to keep the quality of the audio channels high, though we are quite satisfied with consumer grade PZM microphones.
Wires. One unfortunate characteristic of using analog technologies is the abundance of wires that are required [bux,moran]. When installing a media space out of a laboratory environment, the wires must be installed in a manner coordinated with the overall design of the environment. If all other wires are neatly hidden, then so must be those for the media space. We anxiously await the arrival of digital architectures to support our work [19].
Prior literature has indicated some of the technological obstacles to using off-the-shelf audio and video technology in media spaces [3,20]. We have found that subjects are very concerned with the way the appear on the monitor, and have witnessed significant rearrangement of office space to accommodate an interesting view from the camera's perspective.
Several media space installations have cameras in public areas to increase awareness and accommodate informal interactions [2,11]. One of our wrist's length sites does not have a public space that is appropriate for viewing in this way. Through questionnaires and interviews we have found that many subjects, before installation of the media space, are concerned about the intrusion of cameras on their privacy, and they have no desire to have cameras trained on them in public spaces. At the beginning of our study subjects were inclined to leave their equipment turned off when they arrived in the morning until such a time as they desired to contact someone. This resulted in numerous occasions when people were notified of a communication attempt on their workstation, and had to hastily turn on all their equipment. (We attempted to hide all of the power switches and other controls, but this was not possible for the monitors and speakers we selected.) In an attempt to remedy this problem we pointed a camera out a window with an interesting view. This had the immediate effect of encouraging people to turn on their equipment the moment they arrived in the morning.
In another experience an individual decided the did not want a media space node in their office, but they did accept our offer of a video monitor, to provide them with an electronic window. The individual became comfortable enough with the presence of the monitor that they soon requested a full media space node. A different subject likened a 'black' monitor (one with no image on the screen) to a "black hole". It's mere presence was intimidating. Another subject, who had a large picture window in their office, would frequently notice changes in the weather on their electronic window rather than the wall-sized 'real' window.
These anecdotes suggest that putting an interesting image, on the monitor may decrease anxiety and encourage adoption.
The Telepresence project is a small team of people with a full palette of research activities on our agendas, so we have little remaining resources to install and support additional media space installations. The enabling technologies which we deploy in field studies are assembled in non-standard ways, so traditional hardware vendors haven't the vaguest idea how to support media space installations. The software that we provide is relatively complex and complete installation and operation manuals are still under development. There is no comprehensive source of information giving instructions on how to setup, operate and maintain a media space. All of these add up to a potential technical support nightmare.
Figure 2 illustrates a simplified view of the relationships among the participants in Telepresence field studies. The three main players are illustrated as ovals. They are: the Telepresence researchers, the field trial subjects, and the industrial/commercial partners.
The subjects are observed working on the technology foundation. Observation is carried out by the social scientists of the Telepresence project. The technology foundation used by the subjects was designed and tested by the Telepresence researchers in conjunction with the industrial/commercial affiliates. However, the actual deployment and maintenance of the system is undertaken by the industrial/commercial affiliates (in consultation with the Telepresence researchers, with diminishing involvement over time). The technology foundation is financed by the user/clients, and commissioned directly from the industrial/commercial affiliates.
Central to this arrangement is the view that the deployed technology must have a life in the client organization after the field studies. The Telepresence Project has a central role in the initiation, evaluation and revision of the foundation technology and deployment. At the end of the study, however, the Telepresence project will withdraw, and the system can continue to be maintained. Everything to the left of the vertical dotted line is self-contained.
By this, we avoid the common trap of field studies where the technology being tested is deployed temporarily, for the duration of the study. If the technology is not going to remain in use, the users have no real motivation to modify their work habits and integrate it into their workaday habits, and any results that might be obtained from the study will be suspect. Our approach helps avoid this common pit-fall.
This organization means that studies can be undertaken with only minimal resources from the Telepresence project. The primary requisite resources come from the main stake-holders in the private sector and the user/clients.
Our first field study at wrist's length has provided us with insight into to process of moving experimental technologies out of the lab. Moving the technology outside the lab requires a degree of robustness and documentation not normally required in a community of technically advanced users. For the technical staff, who are experienced in design in a research environment, these requirements required a significant shift of priorities.
The system architects/developers, who were accustomed to having the media space running under their watchful eyes, were suddenly charged with the task of insuring the system would run effectively and reliably in an environment over which they had no control. Reporting and handling error conditions suddenly became more important, as did performance, security, and reliability. Having made commitments of deliverables to outside organizations to install and operate systems by certain dates, we found our management and programming practices had to change to reflect more of a product development cycle. This was often at odds with the traditional research practice in our environment.
Even in our own environment the successful adoption of the media spaces by approximately 24 people has been a double edged success. On the one hand we have demonstrated the usefulness of the tools. On the other hand we have created a demand for the system that requires that it be fully operational at all times, thus decreasing our ability to explore, test, debug and otherwise research new media space concepts. We are trying to manage apparently opposing objectives of research and exploration on the one hand, and production and dependability of the system on the other.
- study of media spaces outside of the laboratory environment is different
- need to worry about support
- need to worry about reliability
- involvement of industrial affilliates helps with support issue
-
For research support the authors are indebted to Technology Ontario, Information Technology Research Centre and the Telecommunications Research Institute of Ontario. We acknowledge the contributions of Tracy Narine, Beverly Harrison, Tom Milligan, Chris Passier, Barry Wellman, Janet Salaff, Dimitrina Dimitrova, Laura Garton, Dick Dillon, Jo Tombaugh and Barbara Whitmer for their invaluable efforts in our first field study. Thank you. Special thanks to Barbara Whitmer for comments and suggestions on this paper.
1. Baecker, R.M., Nastos, D., Posner, I.R. and Mawby, K.L. The User-Centred Iterative Design of Collaborative Writing Software, In Proceedings of CHI'93 (Amsterdam, April 24-29, 1993), ACM Press, pp. 399-405.
2. Bly, S.A., Harrison, S.R. and Irwin, S. Media Spaces: Bringing People Together in a Video, Audio, and Computing Environment, In Communications of the ACM , Vol. 26, No. 1 (January 1993), ACM Press, pp. 28-47.
3. Buxton, W. and Moran, T. EuroPARC's Integrated Interactive Intermedia Facility (iiif): Early Experiences, In Proceedings of the IFIP WG8.4 Conference on Multi-user Interface and Applications (Heraklion, Crete, 1990), Elsevier Science Publishers, pp. 11-34.
4. Buxton, W. Telepresence: Integrating Shared Task and Shared Person Spaces, In Proceedings of Graphics Interface '92 (Vancouver, May 11-15, 1992), pp. 123-129.
5. Dourish, P. and Bly, S. Portholes: Supporting Awareness in a Distributed Work Group, In Proceedings of CHI'92 (Monterey, May 3-7,1992), ACM Press, pp. 541-547.
6. Dourish, P. The Affordances of Media Spaces for Collaboration, In Proceedings of CSCW'92 (Toronto, Nov 1-4, 1992), ACM Press, pp. 17-24.
7. Egido, C. Videoconferencing as a Technology to Support Group Work: A Review of its Failures, In Proceedings of CSCW'88 (Portland, Sept 26-28, 1988), ACM Press, pp. 13-24.
8. Elliott, C. High-Quality Multimedia Conferencing Through a Long-Haul Packet Network, In Proceedings of ACM Multimedia 93 (Anaheim, Aug 1-6, 1993), Addison-Wesley, pp. 91-98.
9. Fish, R.S., Kraut, R.E. and Chalfonte, B.L. The Video Window System in Informal Communication, In Proceedings of CSCW'90 (Los Angeles, Oct 7-10, 1990), ACM Press, pp. 1-11.
10. Galloway, K. and Rabinowitz, S. Hole-in-Space. Mobile image videotape, Santa Monica, California, 1980.
11. Gaver, W., Moran, T., MacLean, A, Lövstrand, L., Dourish, P., Carter, K. and Buxton, W. Realizing a Video Environmtent: EuroPARCs RAVE System, In Proceedings of CHI'92 (Monterey, May 3-7,1992), ACM Press, pp. 27-35.
12. Gaver, W., Sellen, A., Heath, C. and Luff, P. One is Not Enough: Multiple Views in a Media Space, In Proceedings of CHI'93 (Amsterdam, April 24-29, 1993), ACM Press, pp. 335-341.
13. Gaver, W., Smith, R. and O'Shea, T. Effective Sounds in Complex Systems: The Arkola Simulation, In Proceedings of CHI'91(New Orleans, April 27-May 1, 1991), ACM Press, pp. 85-90.
14. Heath, C. and Luff, P. Disembodied Conduct: Communication Through Video in a Multi-Media Office Environment, In Proceedings of CHI'91 (New Orleans, April 27-May 1, 1991), ACM Press, pp. 99-103.
15. Hopper, A., et al. The Pandora Multimedia System (a videotape), Cambridge, England, Olivetti Corporation, 1991.
16. Ishii, H. and Kobayashi, M. ClearBoard: A Seamless Medium for Shared Drawing and Conversation With Eye Contact, In Proceedings of CHI'92 (Monterey, May 3-7,1992), ACM Press, pp. 525-532.
17. Li, J. and Mantei, M. Working Together, Virtually, In Proceedings of Graphics Interface '92 (Vancouver, May 11-15, 1992), ???, pp. 115-122.
18. Lu, I. and Mantei, M. Idea Management in a Shared Drawing Tool, In Proceedings of ECSCW'91, Kluwer Academic, The Netherlands, pp. 97-112.
19. Lyles, J.B. and Swinehard, D.C. The Emerging Gigabit Environment and the Role of Local ATM, IEEE Communications Magazine, Vol. 30, No. 4, April 1992, pp. 52-58.
20. Mantei, M.M., Baecker, R.M., Sellen, A.J., Buxton, W., Milligan, T. and Wellman, B. Experiences in the User of a Media Space, In Proceedings of CHI'91 (New Orleans, April 27-May 1, 1991), ACM Press, pp. 203-208.
21. Moore, G. Social Science Models and Strategies for Telepresence User Studies, Ontario Telepresence Technical Report OTP-93-01, February, 1993.
22. Pye, R. and Williams, E. Teleconferencing: Is Video Valuable or is Audio Adequate?, Telecommunications Policy (June 1977), pp. 230-241.
23. Sellen, A.J. Remote Conversations: The Effects of Mediating Talk with Technology, In the Journal of Human-Computer Interaction, 1993. (??? verify this)
24. Sellen, A.J. Speech Patterns in Video-Mediated Conversations, In Proceedings of CHI'92 (Monterey, May 3-7,1992), ACM Press, pp. 49-59.
25. Stults, R. Media Space, Xerox PARC technical report, 1986.