Experimental Study of the Effect of Interruption on Office Work
Experiment 2: Recovery Devices

a study made at UCSD for Electricité de France, in collaboration with

Saadi Lahlou, EDF, by

David Kirsh, Thomas Rebotier, Clarissa Reeves and Mark Remy, UCSD

 

In this second design we looked at the effect of two "recovery devices" that gave users a feed-back on where they were before they were interrupted.  These are mechanisms that might help improve the performance of subjects during the interruption phase of the task.   The first recovery device was a delayed video player, and the second was a toggle highlighting all the changes already made in the document.

To our disappointment none of the recovery device seemed to help.  Indeed, they rather hindered.  Given that our primary task did not contain a major memory or planning component it is understandable retrospectively that subjects showed no improvement when they were encouraged to use the recovery devices.   This was somewhat of a surprise however, since we did expect that subjects would find the track changes feature of MS Word to be helpful in noting what they had done and where they ought to pick up their task.  As the results illustrate, the recovery devices (both of them) make things worse rather than better.

We had initially planned to run 12 subjects on this experiment, but as it became evident from both the data and the subject reports that the recovery devices were harmful rather than useful we did not proceed any further than the first four.

Subjects were initially run in a similar manner to the main experiment, but without a baseline phase.  They alternated between 2 minute phases of no interruption, and short or long interruptions.  Since we had found from our last experiment on 45 subjects that there was no difference between our cognitive and social interruptions we treated all these as identical and used all of them in this experiment.   In addition to the standard interruption phase, subjects were run on recovery one phase, where they were offered the use of a delayed video recovery device, and a recovery two phase, where they were offered the use of the MS Word track changes mechanism.  The order of presentation of the phases was randomized between subjects.

The delayed video recovery device was made by buffering two hundred frames of the video signal.  On the computer this was run on, the buffering introduced a twenty seconds delay.  The process could be halted at any time, so that the buffer would preserve the previous 20 seconds of signal.  When resumed, the process started buffering again, at the same time as the last pre-halting twenty seconds were displayed.

The delayed video was programmed by adapting a sample program of the free multimedia toolkit "DX Media, v.6.0" from Microsoft.  It requires a PC running Windows 98, having Microsoft Visual C++, and a video capture card with DX Media drivers (which most capture cards use).  The changes to the program itself are minimal (delete various color effects, add a buffer for the stream).  [We will provide EDF with a copy of the folder containing the project source files.]  The compiled program is a graphix "filter", run by the DX Media filter editor.   If the EDF team desires so, we will demonstrate/write instructions for that multimedia development environment.  (This environment allows pixel-per-pixel access to the video stream, so that if needed the recovery device could be made much more complicated without requiring more expertise than C/C++ programming).

The other recovery device was the standard track changes feature of MS Word, with highlight changes turned on.

As mentioned before the main results were negative: subjects took longer when they had the added complexity of considering the use of the recovery device.  As shown in Table 2, the cost of adapting to the device exceeded the benefits of using it.  With only four subjects we fall short of significance (.13) but the numbers are bad enough not to turn around with more SS.   Indeed in both cases it was not clear that the devices offered any benefits to the particular task subjects were doing.  It is quite possible that subjects would find either of these devices useful if they were engaged in a task requiring more planning or prospective memory.   Our choice of the MS track changes feature, for instance, was motivated by the well known value similar event driven or change visualizers are known to have in programming environments.  

Table 1: The nefarious effect of some recovery devices.

In summary, we found that the recovery devices (both of them) make things worse rather than better.  With only four subjects we fall short of significance (.13) but the numbers are bad enough not to turn around with more SS.  It seems that for the video recovery device the time spent watching the video was a useless cost.  For the highlighting device, it could be that the visual clutter resulting from highlighting all past changes was bothering the subjects.

From the interviews, it seems that considering the task the only device that might have been useful would be a spatial marker on the paper document showing the corrections to be made.  This suggests that in a work environment the most universal recovery device might be very simple: a tacky pointer.  A sound-responding system (like the whistle-responding key-chains) could be embedded in tacky material (3M does recyclable tacky paste, attaching to various materials just like a post-it, but with the of modeling clay), with perhaps also velcro and/or other means of removable attachment (hook, suction cup for screen).  The whole might be in bright, fluorescent , easy-to-spot color (may come in several colors), and have grossly an arrowhead shape.  When the user is interrupted, he or she could immediately attach the tacky pointer where he was looking  at the time of interruption.