Natapov, D., Castellucci, S. J., and MacKenzie, I. S. (2009). ISO 9241-9 evaluation of video game controllers. Proceedings of Graphics Interface 2009, pp. 223-230. Toronto: Canadian Information Processing Society. [PDF] [software]

ISO 9241-9 Evaluation of Video Game Controllers

Daniel Natapov, Steven J. Castellucci, I. Scott MacKenzie

Department of Computer Science and Engineering
York University, Toronto, Canada
{dnatapov, stevenc, mack}@cse.yorku.ca

ABSTRACT
Fifteen participants completed a study comparing video game controllers for point-select tasks. We used a Fitts' law task, as per ISO 9241-9, using the Nintendo Wii Remote for infrared pointing, the Nintendo Classic Controller, for analogue stick pointing, and a standard mouse as a baseline condition. The mouse had the highest throughput at 3.78 bps. Both game controllers performed poorly by comparison. The Wii Remote throughput was 31.5% lower, at 2.59 bps, and the Classic Controller 60.8% lower at 1.48 bps. Comparing just the video game controllers, the Wii Remote presents a 75% increase in throughput over the Classic Controller. Error rates for the mouse, Classic Controller, and the Wii Remote were 3.53%, 6.58%, and 10.2%, respectively. Fourteen of 15 participants expressed a preference for the Wii Remote over the Classic Controller for pointing tasks in a home entertainment environment.

KEYWORDS: Video game controller, analogue stick, infrared, Wiimote, target acquisition, performance comparison, Fitts' task.

INDEX TERMS: H.5.2 Information interfaces and presentation (e.g., HCI): User Interfaces--evaluation/methodology.

1 INTRODUCTION

Video games comprise a large and profitable industry, totalling $9.5 billion in 2007 sales [3]. A recent and greatly hyped entrant is the Nintendo Wii console, launched in November 2006. Success was immediate, as the Wii gained quick dominance of the console market by outselling the competition – Microsoft's Xbox 360 and Sony's PlayStation 3. For instance, in April 2008, the Wii sold more units than the Xbox 360, PlayStation 3, PlayStation 2, and PlayStation Portable combined [9]. Though the Xbox 360 and PlayStation 3 have superior graphics capabilities, processing power, and storage capacity, the Wii employs a unique controller – the Wii Remote (also referred to as the Wiimote). The Wiimote allows one's gestures to serve as input to the video game. Media reports attribute the commercial success of the Wii to this innovative controller, which makes gaming more intuitive and accessible to a broader user community [16].

Video game controllers are finding other uses in home entertainment. Televisions are becoming the source of increased user interaction. Home theatre PCs (HTPCs) allow users to view pictures from their digital cameras and stream movies from the Internet. Personal video recorders, program guides, and interactive DVDs all require users to perform frequent menu selections. Gaming consoles also provide media player functionalities.

However, navigating graphical interfaces and selecting menu options are not tasks well-suited for the traditional remote control. Instead, HTPCs use the ubiquitous mouse; gaming consoles rely on their controllers. While gamers are passionate about their controllers and gaming environments, independent empirical evaluations quantifying and comparing human performance with game controllers are rare. This paper presents an evaluation, comparing the Wiimote game controller to a traditional analogue joystick controller. An optical mouse is included as a baseline condition. We believe our findings can aid in design decisions for associated pointing devices. As a basis for comparison, we use the measures of throughput and error rate as specified in Part 9 of the ISO 9241 standard [5]. The results of our study seek to provide quantitative comparisons of pointing devices in a home entertainment environment.

1.1 Video Game Controllers

Traditional controllers include the Xbox 360 controller (Figure 1, top) as well as the PlayStation 3 controller (Figure 1, bottom). The only significant difference between the two is the positions of the analogue stick and directional pad. The Xbox 360 has the analogue stick on the left of the directional pad, while the PlayStation 3 has the analogue stick on the right. While we acknowledge that an analogue joystick device employs digital circuitry, we refer to it as "analogue" to be consistent with common vernacular.

(a)
(b)
Figure 1: (a) Xbox 360 and (b) PlayStation 3 controllers

The Wiimote is shown in Figure 2a, and is visibly different from the other controllers. There is a directional pad on the top of the controller, as well as "A", "1", and "2" buttons. There is a trigger on the bottom, and an infrared camera on the front. The optional Nunchuck (Figure 2b) attaches to the Wiimote to provide an analogue joystick if needed.

What further separates the Wiimote, and allows it to have fewer buttons than traditional controllers, are its two methods of input. The first method uses an infrared camera on the front of the controller. The Wii system includes a series of infrared LEDs in a Sensor Bar, which is placed above or below the television screen. The Sensor Bar provides a reference point for the Wiimote, which enables a player to target a screen location by simply pointing to it. The Wiimote's relative position to the Sensor Bar maps to an absolute location on the screen. The second input method uses an accelerometer for sensing motion, thus affording gestural input. Players can shake, swing, tilt, and roll the Wiimote, with each motion specifying a different game action. However, this form of input is not employed for pointing tasks.

(a)
(b)
Figure 2: (a) Wiimote (sideways) and (b) Nunchuck

Another optional attachment for the Wii is the Classic Controller shown in Figure 3. It provides a traditional control scheme for use with previous generation games. The Classic Controller has two analogue sticks, a directional pad, four function keys ("B", "A", "Y", and "X"), and two triggers. Its configuration is similar to that of the PlayStation 3 controller.


Figure 3: Wii Classic Controller

Any evaluation of video game controllers needs to account for the tasks the controllers are used for. The tasks vary depending on the game or application, but the more common ones include navigation, targeting, and button input.

Navigation refers to moving an avatar through a game world (whether 2D or 3D) and is generally performed using an analogue joystick.

The joystick is a good fit for the navigation task, since it intuitively inputs direction. Using a navigation task as the basis for comparison would not produce interesting results, since the different controllers would all use an analogue stick for this task (using the Nunchuck in the case of the Wiimote). We do not consider this task in our comparison.

A targeting task involves on screen point selection. It is a common task interpretable in a variety of ways, ranging from shooting on-screen enemies to selecting on-screen menu options. The traditional controller scheme will generally use one analogue joystick for navigation and a second analogue joystick for targeting. On the Wiimote, targeting is generally done using infrared pointing. We hypothesize that infrared pointing is a better fit for point-select tasks than using an analogue joystick. This hypothesis is the basis of our comparison in the study.

Button input generally involves pressing buttons in a certain sequence, or at a certain time, to perform game actions. The task is performed similarly using different controllers. The Wiimote is sometimes the exception, and for some games gestural input with the Wiimote is used instead of button input. For instance, in The Legend of Zelda: Twilight Princess, it is possible to use the "A" button to cause the protagonist to swing his sword, but it is also possible to shake the Wiimote to perform the same action. Though this would be an interesting comparison to make, it is not considered in our study. We did not consider this comparison because the Wiimote does not enforce gestural inputs instead of button inputs – the Wiimote affords buttons as well, and gestural inputs are not always used. This is different than point-select tasks, in that the Wiimote only uses infrared pointing, while traditional controllers only use the analogue joystick to select on-screen targets, with no overlap.

The present study compares the different controllers with respect to a point-select task. We consider this task ideal for the comparison because it is common and is always performed differently between the two classes of video game controllers. The intent is to reveal which method of targeting is superior and, as a consequence, which controller is more appropriate for this task.

1.2 Related Work

Video game controllers are often used for tasks other than gaming. Consequently, it is important to know how to evaluate and advance such technology. However, no empirical comparison of game controllers yet exists. This paper is a first attempt to provide such information In subsequent subsections, we summarize related research. Specifically, we highlight research that used video game controllers as input devices outside the context of video games and research regarding the quantitative evaluation of pointing devices.

1.2.1 Video Game Controllers as Input Devices

Koltringer et al. used an Xbox 360 controller in a text entry technique called TwoStick [8]. The technique maps characters to cells in a 9-by-9 grid. The characters include letters of the English alphabet, punctuation, numbers, and symbols. Users enter a character using the analogue joysticks: one joystick selects a 3-by-3 grid; the other selects the desired character within that grid.

Previous research involving text entry with video game controllers also includes UniGest, a method of text entry using the Wiimote [1]. UniGest uses the Wiimote's motion-based input to map gestures to letters of the English alphabet. However, unlike TwoStick, it does not yet permit the entry of punctuation, numbers, or symbols. Gardner et al. describe a software interface using a gamepad for drawing in a three dimensional environment [4]. The interface uses analogue joysticks to position a cursor in 3D. The technique is intended for a walk-in virtual reality theatre in a science museum. Research by Schlomer et al. describes using the Wiimote's motion sensors to recognize input gestures [13]. Independent of the gaming console, the system allows the training of arbitrary gestures by users. The gestures are then used in applications such as browsing photos on a home television.

1.2.2 Evaluation of Pointing Devices

The ISO 9241-9 [5] standard is directed at the evaluation of non-keyboard input devices. It proposes a standardized methodology for evaluating performance and comfort. Performance is evaluated according to one of six tasks (in this research we used the multi-directional tapping task) and is measured in terms of throughput, which is based on Fitts' Index of Performance [10].

Throughput (TP, in bps) is computed by dividing the index of difficulty (ID, in bits, averaged over a block of trials) by the average movement time (MT, in seconds):

  (1)

The calculation of ID is a logarithmic term known as the Shannon formulation. It includes D for movement distance and W for target width:

  (2)

Instead of using the presented IDs (Eq. 2) in the calculation of throughput, the standard prescribes use of effective IDs to accommodate the spatial variability observed in responses [10]:

  (3)

The term De represents effective distance – the distance a participant actually traversed along the task axis. The task axis is a straight line from the centre of the source to the centre of the target. The term x is the distance from the participant's click to the centre of the selected target, projected on the task axis. SDx is the standard deviation in x over a block of trials using the same D and W. Note that x can be positive or negative, depending on whether selection was an overshoot or undershoot, respectively. Calculating throughput (Eq. 1) using IDe (Eq. 3) yields an overall performance measure that includes both the speed and accuracy in user responses.

Comfort of a device is quantified according to ISO 9241-9 by having participants subjectively rate the device in a questionnaire, based on different categories. We used such a questionnaire in our experiment. Considerable previous work exists on the comparison and evaluation of pointing devices (both following ISO 9241-9 and otherwise). We borrowed from the methods proposed in those papers for our own comparison and evaluation.

Research by Douglas et al. evaluated the scientific validity and practicality of ISO 9241-9 [2]. An experiment was conducted comparing a finger-controlled isometric joystick and a touchpad. Participants were asked to perform a point-select task, after which throughput for each device was calculated and device comfort was rated in a questionnaire. Research by Isokoski measured the throughput of mice with button-up and button-down events in a Fitts' paradigm pointing test [6]. The experiment recorded the position of the cursor with the button-down event as well as the button-up event, and measured throughput with both. It was determined that button-down events produced slightly better throughput. Other research evaluated remote pointing devices [11] according to ISO 9241-9. In summary, it was found that the remote pointing devices performed poorly when compared to a mouse, and that participants preferred using a mouse over either of the two remote pointing devices.

Klochek and MacKenzie presented five new performance metrics to quantify differences between video game controllers [7]. The experiment compared an Xbox gamepad controller and a standard PC mouse in a target tracking task and compared the performance according to five cursor path metrics. It was found that when trying to track an onscreen moving target, both the mouse and the gamepad allowed participants to track the target's velocity equally well. The significant difference was that the mouse allowed participants more control over acceleration than the gamepad, which in turn helped correct errors in position.

Previous research has also proposed seven new accuracy measures for differentiating devices in precision pointing tasks [12]. The new measures were designed to capture patterns of movement during a trial. Four pointing devices were used in an experiment to validate the measures.

In the following section, we describe our methodology. The goal was to evaluate and compare the traditional Classic Controller with the modern Wiimote according to ISO 9241-9. ISO 9241-9 employs a series of pointing task and we believe such tasks exemplify targeting in video games and menu navigation in home entertainment systems. The mouse was included as a baseline condition.

2 METHOD

2.1 Participants

Fifteen paid participants (6 male, 9 female) were recruited from the local university campus and local community. All were right-handed, though this was not by design. Participant ages ranged from 18 to 28 (mean = 22.2).

When completing the pre-experiment questionnaire, all participants stated they used the mouse frequently. They were also asked about their experience with each gaming device using the following categories:

For the Wiimote, the number of responses was 1 (non-existent), 8 (rarely), 3 (infrequent), and 3 (frequently). For traditional analogue controllers, the numbers were 1 (non-existent), 10 (rarely), 1 (infrequently), and 3 (frequently).

Participants were chosen randomly. We were hoping to study groups of participants who had specific experience with video game controllers. Unfortunately, our participant pool did not reveal such diversity. Still, we believe our participants' varied experience will allow our findings to generalize to a larger population.

2.2 Apparatus

2.2.1 Hardware

The experiment was conducted using a Samsung DLP television (model HL-T5076S) with a resolution of 1024 × 768 and an IBM T42 ThinkPad laptop PC, with Windows XP SP 3. The controllers used were a Wiimote, a Wii Classic Controller, and a Logitech V220 Cordless Optical Mouse. The experiment software was run on the laptop, but the audio and video outputs were sent through the television.

Effectively, participants were interacting directly with the television, instead of with the laptop. The experiment setup appears in Figure 4. Participants sat in an armchair directly in front of a television, situated two meters away. This seating arrangement simulated a typical home entertainment environment. The television was on a stand of height 73 cm. The screen size (diagonal) was 102 cm. The armchair seat height was 42 cm, with 58 cm height arm rests. To allow participants to use the mouse on a hard surface, we placed a shelf across the arm rest to serve as a make shift desk. The shelf's thickness was 2 cm.


Figure 4: Participant performing the experiment.

2.2.2 Software

The experiment software was written in JavaScript and run under Mozilla Firefox (v2.0.0.16). The application was shown on the television (from the ThinkPad) in a maximized Firefox window, thus simulating the feeling of a home entertainment application on the television, as opposed to a browser application on a PC. From the point of view of the JavaScript application, the input devices interacted like a mouse. This was achieved by the use of emulators.

To use the Wiimote as a mouse, we connected it to the ThinkPad using a Bluetooth adapter and utilized software called GlovePIE (v0.3). GlovePIE includes scripts that emulate a mouse pointer using the Wiimote's input. Pointing input used the Wiimote's infrared sensors and a Nyko Wireless Sensor Bar, placed on the bottom of the television. The Wiimote's "A" button was used for selection.

To use the Classic Controller as a mouse, it was plugged into the Wiimote. A different GlovePIE script was used to emulate a Windows Game Controller. To use the Windows Game Controller input as mouse input, we used the trial version of Total Game Control (v3.5). This software maps game controller events to mouse and keyboard events. The Classic Controller's left analogue stick was used for pointer movement, while the "B" button was used for selection. The mapping of targeting to a particular joystick varies between games and is sometimes customizable. Given the presence of buttons on the right of the Classic Controller, we opted to allocate the left joystick for targeting. Figure 5 depicts the Classic Controller in use during the experiment.


Figure 5: User manipulating the Classic Controller.

The JavaScript program implemented a Fitts' point-select task, much like the experiments described earlier [2, 14]. There was a green "Ready" square (referred to as a home square in the aforementioned publications), of size 60 × 60 pixels. When a participant clicked Ready, its colour changed to red and the message changed to "Timing", thus beginning a trial. The participant proceeded to move the cursor to the highlighted target circle and click on it. We logged the duration between clicking on "Ready" and clicking the target. Additionally, we logged the coordinates of the mouse click and their distance along the task axis from the centre of the target. Finally, we logged whether the click was a hit (inside the target) or a miss (outside the target).

At the end of a trial, the "Timing" square reverted to a green "Ready" square and a new target appeared at a different location. At this point the participant could rest briefly. When ready, the participant performed the next trial. Figure 6 and Figure 7 show screen snaps of the application before and after the "Ready" square is clicked. Figure 8 shows a screen snap of the application after the target circle is clicked, and a new target appears.


Figure 6: Screen snap of the experiment before the "Ready" button is clicked.


Figure 7: Screen snap of the experiment after the "Ready" button is clicked.


Figure 8: Screen snap of the experiment after the target circle is clicked and the next target appears.

2.3 Procedure

Prior to beginning, participants were asked to sign a consent form and complete a questionnaire. The questionnaire asked demographic questions such as age, gender, and handedness, as well as multiple-choice questions about their experience with each device (noted above).

Next, participants were given a short instructional session. The instructions specified that they were to repeatedly click the "Ready" square, then the "Target" circle. The participants were also instructed to try to click the target as quickly as possible and as close to the centre as possible. A short practice session (the results of which were not recorded) was also performed.

Each participant performed five blocks of trials with each device. Each block consisted of six ID levels. The six IDs were derived from different target distance and width combinations. There were three target widths (14, 35, and 60 pixels), and two target distances (165 and 330 pixels). These values were chosen based on the size and resolution of the hardware used. The resulting IDs are given in Table 1.

Width (pixels) Distance (pixels) ID (bits)
14 165 3.67
14 330 4.61
35 165 2.51
35 330 3.38
60 165 1.90
60 330 2.70
Table 1: Presented Indices of Difficulty (IDs)

For each ID, the participant clicked a target circle – appearing in one of eight compass directions (N, NE, E, SE, S, SW, W, and NW). Each such square-to-circle movement and click comprised a trial. In total each participant completed 240 trials per device (5 blocks × 6 IDs × 8 trials).

In each of the five blocks, the order of IDs was randomized per participant. Additionally, for each ID, the order of the eight trials (one per direction) was also randomized per participant. As the experiment was fully within-subjects and learning effects were anticipated, participants were divided into three groups with the order of administering device conditions governed by a Latin Square. At the end of each ID level, a progress report was displayed, showing the levels and blocks completed.

After each device, participants rated the device using questions from ISO 9241-9 on device comfort. The questions are similar to those in other experiments [2]. There were nine questions, each with a rating from 1 to 5, as follows:

  1. The force required for actuation was
         (1: too low – 5: too high)
  2. Smoothness during operation was
          (1: very rough – 5: very smooth)
  3. The mental effort required for operation was
          (1: too low – 5: too high)
  4. Accurate pointing was
          (1: easy – 5: difficult)
  5. Operation speed was
          (1: too fast – 5: too slow)
  6. Finger fatigue
          (1: none – 5: very high)
  7. Wrist fatigue
          (1: none – 5: very high)
  8. General comfort
          (1: very uncomfortable – 5: very comfortable)
  9. Overall the input device was
          (1: very difficult to use – 5: very easy to use)

After using all three devices and completing the experiment, participants were given one final instruction:

"Please rate the devices in the order you would prefer to use them for pointing tasks in a home entertainment environment. Please explain your decision to rate them this way, and comment on what you liked and disliked about each device. Feel free to add any additional comments."

The entire procedure, including five experimental blocks per device and the time to complete the questionnaire and instructional session, took just under one hour on average. The experiment was conducted in a university department lounge.

2.4 Design

The experiment employed a within-subjects, 3 × 5 repeated measures design with the following factors and levels:

Input Device:      Wiimote, Classic Controller, Mouse
Block: 1, 2, 3, 4, 5

There were two dependent variables: throughput (bps) and error rate (% error). The goal of the experiment was to investigate the effect of input device on throughput and error rate. In all, there were 15 participants × 3 input devices × 5 blocks × 6 IDs × 8 trials per ID = 10,800 trials.

3 RESULTS AND DISCUSSION

3.1 Throughput

To compare the three Input Devices, we used throughput, as described in ISO 9421-9 (defined above). Throughput is measured in bps and serves as a quantitative measure for comparing input device performance. We calculated throughput for each input device on a per-participant, per-block basis. The results are shown in Figure 9. As expected, the mouse outperformed both video game controllers. The average throughput of the mouse over the five blocks was 3.78 bps. As noted earlier, the mouse was included as a baseline condition. Our findings are in line with previous results for mouse throughput, which typically vary from 3.5 to 4.5 bps [15]. Because the mouse has been previously researched in depth, and since it was expected to provide the best results, our discussion focuses instead on the analysis of the video game controllers.


Figure 9: Throughput (bps) by input device and block

It is worth noting that while mouse throughput was better than those of both video game controllers, it is not necessarily the best choice of input for gaming. A mouse requires a desk or other hard surface on which to track (in our case we used a shelf placed across the arms of the chair). Users might prefer to recline in their chair and hold a game controller in their lap. Furthermore, game controllers are more practical in a home entertainment environment, where users typically sit on a sofa. Both game controllers employed in our study can be held either in the air or in the user's lap.

The next in line, in terms of throughput, was the Wiimote, with a throughput of 2.59 bps, 31.5% lower than for the mouse. While not as good as the mouse, this is respectable for a remote pointing controller. In fact, the throughput for the Wiimote is at the high end of the 0.99 - 2.99 bps values reported for a laptop PC's touchpad [15], and is much better than throughput for the Classic Controller (discussed next).

The Classic Controller, also as expected, falls behind the mouse. The average throughput over the five blocks was 1.48 bps, 60.8% lower than for the mouse. Comparing the two gaming devices, the Wiimote's performance represents a 75% increase in throughput compared to the Classic Controller – substantial, indeed.

Considering these results, it is not surprising that during the experiment, participants new to gaming showed irritation when using the analog controller for point-select tasks. It is therefore not surprising that the Wiimote, with its faster and more accurate pointing, has had such a large impact on the market. Our results show the Wiimote represents a considerable improvement over analog controllers.

As an additional consideration, while all participants rated their mouse usage as "Frequent" (at least once a week), only three stated they frequently used the Wiimote, and only three stated that they frequently used an analog controller. Clearly, experience with the mouse was a confounding variable. If our participants were as experienced with the Wiimote and the Classic Controller as they were with the mouse, it is likely that throughput for the two gaming controllers would have been higher. The impact of input device on throughput was statistically significant (F2,14 = 155.63, p < .0001). The effect of block on throughput was also statistically significant (F4,14 = 14.02, p < .0001). The effect of group on throughput was not statistically significant (F2,14 = 0.64, ns), indicating that counterbalancing was effective. Post hoc analysis using the Scheffé test revealed that differences in throughput were significant between all device pairings (p < .0001).

3.2 Error Rate

Although throughput provides a good measurement of how quickly and accurately participants were able to perform with each device overall, it does not represent whether the task of selecting targets was successful. This information is important for fully evaluating a pointing device. As such, we also analyzed error rates (even though error rate analyses are not required according to ISO 9241-9). The results for error rate are presented in Figure 10.

The impact of input device on error rate was statistically significant (F2,14 = 18.42, p < .0001). Meanwhile, the effect of block on error rate was not (F4,14 = 1.87, p > .05). The effect of group on error rate was also not statistically significant (F2,14 = 1.08, p > .05), indicating again that counterbalancing worked. Post hoc testing revealed that differences in error rate were significant between all device pairings (p < .001).


Figure 10: Error Rate by Input Device and Block

Much like the throughput analysis, we see that the mouse's accuracy was best, with the lowest error rate. On average between the five blocks, the error rate for the mouse was 3.53%. However, unlike the throughput comparison, the Classic Controller performed better than the Wiimote. The average percentage of errors over the five blocks for the Classic Controller was 6.58%, still higher than the percentage of misses for the mouse, but lower than the average of 10.2% errors for the Wiimote. It is worth noting that, with regard to error rate, the Classic Controller's lead over the Wiimote was primarily observed with small and medium sized targets. When considering the large 60-pixel targets only, the Wiimote was more accurate than the Classic Controller, as shown in Figure 11. For large targets, the average error rate was 2.0% for the mouse, 3.0% for the Wiimote, and 3.4% for the Classic Controller. However, when considering only large targets, the effect of input device on error rate was not statistically significant (F2,14 = 1.36, p > .05).


Figure 11: Error Rate by Input Device and Block (large 60-pixel targets only)

This result, while surprising at first, makes sense upon closer consideration. We observed very different patterns of misses for the two controllers. The Wiimote, by its nature, is more susceptible to accidental hand movement or jolts than the Classic Controller, which makes it more error prone. When the target is small, hand shakes and small movements will greatly affect accuracy.

3.3 Questionnaire and Observations

In addition to empirical measurements, we also noted participants' reaction to using the devices. Most participants expressed some form of frustration while performing the task with the Classic Controller. Those who used the Wiimote first expressed that the Classic Controller was much more difficult to use. One participant nearly gave up because she felt she could not accomplish the task at all with the Classic Controller. Conversely, participants who used the Classic Controller first expressed relief and surprise at how much easier the Wiimote was to use. Nearly all participants showed further relief upon using the mouse, and commented how the five blocks of mouse input passed much faster than the blocks using the other controllers. After using each device, participants were asked to rate it according to various comfort categories. The compiled results of the questionnaire are shown in Figure 12.


Figure 12: Questionnaire results

Unfortunately, the questionnaire was not very helpful in quantifying differences between the Wiimote and the Classic Controller. The Kruskal-Wallis test revealed significance only with respect to actuation force (H2 = 6.12, p < .05), smoothness during operation (H2 = 17.06, p < .001), accurate pointing (H2 = 25.80, p < .0001), and overall rating (H2 = 19.40, p < .0001). However, post hoc tests failed to reveal significance between the two game controllers.

In the post-experiment questionnaire, people were also asked to rate the three devices in order of preference for a pointing task in a home entertainment environment. Ten of 15 participants ranked the mouse first, the Wiimote second, and the Classic Controller last. The reasons given were generally that the mouse was the easiest to use, and the Classic Controller was the most difficult.

Four of 15 participants placed the order of preference as Wiimote first, mouse second, and Classic Controller third. They commented that the Wiimote is as intuitive to use as a television remote or simply pointing with one's hand. Furthermore, they noted the Wiimote (unlike a mouse) does not require sitting at a desk or near a hard surface.

One participant placed the order of preference as mouse, Classic Controller, and then Wiimote. This participant was a frequent user of an analogue joystick and deemed the Classic Controller easier to use. Overall, 14 of 15 participants would prefer the Wiimote over the Classic Controller for point-select tasks in a home entertainment setting.

Despite having a higher error rate for small and medium targets, the questionnaire indicates that the Wiimote is the preferred video game controller. This shows that the benefits of the Wiimote's higher throughput outweigh the loss of accuracy. The comments from participants during the experiment support this. It seems that throughput affected participants more than error rate.

The higher throughput of the Wiimote compared to the Classic Controller confirms our hypothesis that infrared pointing is a more intuitive method for point-select tasks than analogue joystick targeting. We believe this is the case despite the higher error-rates of the Wiimote for smaller targets, and this is largely confirmed by the participants' response. The Wiimote is similar to traditional controllers in its affordances for both navigation tasks (using the analogue stick), and button input tasks (either using buttons or gestures). This means that the Wiimote can perform navigational tasks equally well, perform button input tasks equally well, and perform targeting tasks better than traditional controllers. In essence, the Wiimote is more flexible in its affordances and the overall better controller for common video game tasks.

Furthermore, we found that analogue stick targeting is not a good method for performing point-select tasks. Despite this, the method is used in many games that use traditional controllers. Given our findings, it would be a good idea for future generations of consoles to either introduce new controller schemes (possibly similar to the Wiimote), which would make targeting and point-selection easier and more intuitive, or avoid the use of point-selection tasks in games. Though this is limiting, traditional controllers are too often used to perform tasks for which they are not at all well suited.

4 CONCLUSION

This research compared the Classic Controller, the Wiimote, and a mouse using a point-select task. Fifteen participants were recruited for the experiment, and asked to select on-screen target circles, in an ISO-conforming Fitts' task shown on a television. We measured throughput at 3.78 bps for the mouse, 2.59 bps for the Wiimote, and 1.48 bps for the Classic Controller. The throughput for the Classic Controller is likely representative of analogue joystick controllers in general. This shows that the throughput of the Wiimote is a 75% increase over a Classic Controller, and by extension a standard analogue joystick.

Target misses, or error rates, were 3.53% for the mouse, 6.58% for the Classic Controller, and 10.2% for the Wiimote. With small and medium sized targets, accuracy favoured the Classic Controller over the Wiimote. This is likely due to the Wiimote's susceptibility to accidental movements, or hand shakes, which can result in imprecise movement, particularly for small targets.

In a post experiment questionnaire, participants were asked to rate which controller they would prefer to use in a home entertainment environment. Fourteen of 15 participants placed the Wiimote ahead of the Classic Controller.

This study provided a quantitative evaluation of the pointing performance, accuracy, and comfort of video game controllers. No comparison of infrared pointing to analog joystick pointing currently exists. We also showed that the Wiimote was the preferred video game controller for the participants, despite higher error rates. The benefits of higher throughput seemed to outweigh the loss of accuracy for this study. Based on our findings, manufacturers of interactive home entertainment peripherals should consider a Wiimote-like pointing device, if the user is expected to perform point-select tasks.

REFERENCES

[1]    Castellucci, S. J. and MacKenzie, I. S., Unigest: Text entry using three degrees of motion, Extended Abstracts of the ACM Conference on Human Factors in Computing Systems - CHI 2008. New York: ACM, 2008, 3549-3554.

[2] Douglas, S. A., Kirkpatrick, A. E., and MacKenzie, I. S., Testing pointing device performance and user assessment with the ISO 9241, Part 9 standard, Proceedings of the ACM Conference on Human Factors in Computing Systems - CHI '99. New York: ACM, 1999, 215-222.

[3] ESA, Entertainment Software Association (ESA) - Industry Facts.

[4] Gardner, H., Lifeng, D., Wang, Q., and Zhou, G., Line drawing in virtual reality using a game pad, Proceedings of the 7th Australasian User Interface Conference - Volume 50. Hobart, Australia: Australian Computer Society, Inc., 2006, 177-180.

[5] ISO, 9421-9 Ergonomic requirements for office work with visual display terminals (VDTs) - Part 9: Requirements for non-keyboard input devices, International Organization for Standardization, 2000.

[6] Isokoski, P., Variability of throughput in pointing device tests: Button-up or button-down?, Proceedings of the 4th Nordic Conference on Human- Computer Interaction - NordiCHI 2008. New York: ACM, 2006, 68-77.

[7] Klochek, C. and MacKenzie, I. S., Performance measures of game controllers in a three-dimensional environment, Proceedings of Graphics Interface 2006. Toronto: Canadian Information Processing Society, 2006, 73-79.

[8] Koltringer, T., Isokoski, P., and Grechenig, T., TwoStick: Writing with a game controller, Proceedings of Graphics Interface 2007: Toronto: Canadian Information Processing Society, 2007, 103-110.

[9] Kuchera, B., Nintendo Wii outsells 360, PS3, PS2, PSP combined in April: ars technica - the art of technology NPD report, 2008.

[10] MacKenzie, I. S., Fitts' law as a research and design tool in human-computer interaction, Human-Computer Interaction 7, 1992, 91-139.

[11] MacKenzie, I. S. and Jusoh, S., An evaluation of two input devices for remote pointing, Proceedings of the Eighth IFIP Working Conference on Engineering for Human-Computer Interaction – EHCI 2001. Heidelberg, Germany: Springer-Verlag, 2001, 235-249.

[12] MacKenzie, I. S., Kauppinen, T., and Silfverberg, M., Accuracy measures for evaluating computer pointing devices, Proceedings of the ACM Conference on Human Factors in Computing Systems - CHI 2001: New York: ACM, 2001, 9-16.

[13] Schlomer, T., Poppinga, B., Henze, N., and Boll, S., Gesture recognition with a Wii controller, Proceedings of the 2nd International Conference on Tangible and Embedded Interaction New York: ACM, 2008, 11-14.

[14] Silfverberg, M., MacKenzie, I. S., and Kauppinen, T., An isometric joystick as a pointing device for hand-held information terminals, Proceedings of Graphics Interface 2001. Toronto: Canadian Information Processing Society, 2001, 119-126.

[15] Soukoreff, R. W. and MacKenzie, I. S., Towards a standard for pointing device evaluation: Perspectives on 27 years of Fitts' law research in HCI, International Journal of Human-Computer Studies, 61, 2004, 751-789.

[16] Villoria, G., Wii Fit Review - Is Nintendo ushering a fitness revolution?: GameSpy, 2008, Media review of the Wii Fit.