Natapov, D., & MacKenzie, I. S. (2010). Gameplay evaluation of the trackball controller. Proceedings of the 2010 Conference on FuturePlay, 167-174. New York: ACM. [PDF]

Gameplay Evaluation of the Trackball Controller

Daniel Natapov and I. Scott MacKenzie

Dept. of Computer Science and Engineering
York University Toronto, Canada M3J 1P3
{dnatapov,mack}@cse.yorku.ca
ABSTRACT
We present a study of user performance in a First Person Shooter game comparing a prototype trackball controller, a standard game controller, and a keyboard and mouse. The prototype controller replaces the right analog stick of a standard game controller (used for pointing and camera control) with a trackball. To measure the performance of the three input devices, participants played two games. Penguin Hunt measured the number of target hits per minute, which was 28.1 with the keyboard and mouse, 22.9 with the trackball controller, and 21.7 with the standard controller. Modern Warfare measured average completion times, which were 26.8 s with the keyboard and mouse, 31.8 s with the trackball controller and 35.5 s with the standard controller. The trackball controller represents a 5.5% target hits increase over the standard controller in Penguin Hunt, and a 10.4% speed-up in trial completion time in Modern Warfare.

Categories and Subject Descriptors: H.5.2 User Interfaces. Input devices and strategies.

General Terms: Design, Experimentation, Human Factors.

Keywords: Game controllers, trackball, joystick, analog stick, pointing devices, performance evaluation, video games, game input

1 INTRODUCTION

Game playing on the PC offers a different experience than game playing on a console. Not surprisingly, there is considerable Internet debate on which is better. The differences are numerous, ranging from insignificant to major. For example, the TV screen used with consoles is generally larger than the PC's smaller screen. In terms of performance, a good PC has more processing power and better graphics than a console. In terms of price, consoles are generally less expensive than a PC. The list of differences goes on, but an important difference, which is the focus of this research, is the input method.

PC games are played using the keyboard and mouse, on a supporting surface such as a desk. Console games are played using handheld console controllers, either sitting or standing, and without a supporting surface. Console controllers come in a variety of types. The standard input device is a gamepad, such as the Microsoft Xbox 360 controller, or the Sony PlayStation 3 controller. We refer to these as "standard controllers", as they have similar arrangements of buttons, triggers, and analog sticks. Another method of input is motion sensing, such as the Nintendo Wiimote or the upcoming Microsoft Project Natal and Sony PlayStation Move. Other methods of input include specialty controllers, such as guitars, mats, guns, etc. This paper focuses on the difference between PC controls and standard game controllers. We are not investigating specialized controllers or motion sensing controllers, as they are task-specific and often used for tasks of a different nature than standard PC controls, making for an invalid comparison. On the other hand, PC controls share a lot of similarities with standard controllers. Both types are generally used for similar genres of games, with the same games often playable both on consoles and the PC. Both types of controllers are also similar in their use of buttons for input. The main differences between the two are the analog thumb joysticks used by gamepads, which are not present on PC controls.

Analog sticks became standard on game controllers when gaming became three-dimensional. The first analog stick was introduced on the Nintendo 64 controller in 1996. Analog sticks are useful for 3D games, since they allow for motion in more than eight directions (as offered by the directional pad). A second analog stick was added, since in 3D games camera position and orientation is player controlled. The second analog stick was used to pan the camera in a larger number of directions than possible with buttons, and at varying speeds. At present, in most 3D games, and First-Person Shooters (FPS) specifically, the left analog stick is used for character movement, while the right analog stick is used for camera control and pointing. On the PC, the W, S, A, and D keys are used for motion, while the mouse is used for camera control and pointing.

The left analog stick performs well compared to the W, S, A, and D keys. It specifies a direction of motion, and allows for movement in more directions and at varying speeds. The right analog stick, on the other hand, performs poorly compared to the mouse for pointing and camera control. The mouse specifies a position, while the analog stick specifies a panning direction, with the former being faster and more accurate. In terms of throughput, which is a measure of pointing speed and accuracy [6], the mouse is vastly superior to the analog stick - the throughput of the analog stick was measured to be 61% lower [7]. The purpose of our research is to bridge the gap between gamepads and PC controls in terms of pointing.

Using the mouse is not feasible with gamepads, as a mouse is operated on a supporting surface, while gamepads are handheld. Instead, we created a prototype which replaces the right analog stick of a gamepad with a trackball (Figure 1). Earlier, we evaluated the trackball controller in terms of pointing in a Fitts' law experiment [8] and found that the trackball controller offers approximately a 60% increase in throughput over the analog stick for novice and advanced gamepad users. We also found that cursor path was less direct with the trackball as compared to the analog stick. The previous study involved a simple pointing task, and as a consequence a number of questions remain unanswered. It is uncertain whether the controller offers performance benefit in a realistic task, as opposed to a pointing task (as pointing and camera control are similar but not identical). It is also unclear whether a less direct cursor path has a detrimental effect on camera control.


Figure 1: Trackball controller

To address these questions, this study compares the trackball controller to a standard controller and a keyboard and mouse, in a realistic gameplay task. Participants used the three input methods in two FPS games: Penguin Hunt [3] and Infinity Ward's Call of Duty 4: Modern Warfare [1]. Our hypothesis is that while the keyboard and mouse are still the best method of input (performance wise), the trackball controller will yield an improvement over the standard controller. Our aim is for this new prototype to bridge the gap between standard gamepads and PC controls, offering an alternative that is better for camera control and pointing, but is still usable in a handheld fashion.

1.1 Related Work

Klochek and MacKenzie presented five new performance metrics to quantify differences between video game controllers [5]. The experiment compared an Xbox gamepad and a standard PC mouse in a target-tracking task. Performance was compared according to five cursor path metrics. The significant finding was that the mouse allowed participants more control over acceleration than the gamepad, which in turn helped correct errors in position. The trackball will also allow more control over acceleration than the analog stick.

Isokoski and Martin report on a number of studies comparing performance of input devices for FPS games. In one, they compared an Xbox 360 controller (gamepad) to a mouse and keyboard combination for use in FPS games [4]. Participants played a simple FPS game called Penguin Hunt. The number of targets hit in a trial was used as the measure of performance. Results showed that the keyboard and mouse combination performed better in terms of the number of target hits than the Xbox 360 controller. In a similar study, they compared the Xbox 360 controller to a keyboard and mouse, and to a keyboard and mouse aided by eye tracking [2]. As before, performance was measured in terms of target hits. The keyboard and mouse again performed better than the Xbox 360 controller, which confirms their previous result. The eye tracking assisted keyboard and mouse performed on par with the Xbox 360 controller. In the following section, we describe our methodology. The goal was to compare the prototype trackball controller to the standard controller and to the keyboard and mouse, in terms of performance in FPS games.

2 METHOD

2.1 Participants

The problem with comparing our prototype to existing devices is the confounding influence of participants' prior experience. It is reasonable that participants will perform better with the device with which they are familiar. Unfortunately, as no participants could be experienced with our prototype, it created the risk of a biased comparison. As a consequence, we initially planned to only use participants who had no experience with FPS gaming using any input device in order to eliminate prior experience as a confounding variable.

Unfortunately, such a participant pool turned out to be unusable. We performed a pilot study with novice game players, who had no experience with either input method, and found that the task of playing even simple FPS games is too complex for novices to obtain any clear results during a short study. Participants were frustrated, confused, and often unable or unwilling to complete the task. The resulting data had too much variance from participants learning the task; the variance overshadowed any differences resulting from the input methods used.

As a result, we used participants who were advanced FPS players, despite this biasing the comparison against our prototype. As a consequence, the differences presented here are understated. For the final study, we recruited 18 participants, self assessed as advanced FPS players (on either consoles or the PC). Participants' adeptness at FPS gaming was also validated based on the task. Some participants who performed below expectation, or were outliers, were removed. In the end, we used the scores of 16 of the 18 participants (discussed below).

Prior to beginning, all participants completed a pre-experiment demographic questionnaire. A question asked whether participants had previously played Call of Duty 4: Modern Warfare. Although all participants were advanced FPS players, they ranked their platform experience on PCs and consoles, and with gaming in general using the following categories:

Participants were recruited from the local campus and the local community, and paid for their participation. We divided participants into three groups, based on platform experience: players advanced on consoles, players advanced on the PC, and players advanced on both. Each group had 6 participants for a total of 18 participants. The average age was 23.2 (SD = 4.13). On average, participants took one hour to complete the study. Eleven of 18 participants stated they previously played Modern Warfare.

2.2 Apparatus

2.2.1 Hardware

The experiment was conducted on a Sony VAIO laptop running Windows 7. For the keyboard and mouse conditions, the laptop's keyboard was used with a USB mouse (Logitech Laser). A Microsoft Xbox 360 wired controller was used for the standard controller condition (connected through USB). For the trackball controller we built a prototype using a Microsoft Xbox 360 wired controller and a wired Logitech Trackman Wheel trackball. We used the Xbox 360 controller as the housing for the prototype so that the form factor for both is the same, except for the trackball replacement. The prototype controller replaced the right analog joystick with a trackball. The optical camera and original housing of the trackball were glued to the underside of the top half of the Xbox 360 controller. The trackball was placed in the housing, and a plastic lid was attached so that the trackball was moveable but secure. The rest of the trackball, excluding all unnecessary parts (such as button and scroll wheel controls), were attached to the bottom of the Xbox 360 controller, where the battery pack is on a wireless controller. The resulting prototype is seen in Figure 1. The prototype connects to the PC with two USB cables: one for the trackball for mouse motion, and the other for the game controller for all other inputs.

2.2.2 Software

The first task involved shooting targets in a game called Penguin Hunt. This game was previously used to measure the FPS performance of input devices [2, 4]. We used the same software for our analysis so that our results will be comparable to previous work. The game world in Penguin Hunt consists of a square-shaped randomly generated terrain, with different height levels, and randomly placed trees. The game also contains 10 randomly placed penguin targets, which move slowly in a random direction. The goal is to shoot the targets. Trees block the movement of the player and are also impenetrable to the weapon. When a target is shot it disappears and another appears at a random location. Ammunition is unlimited. A screen shot of the game is shown in Figure 2.


Figure 2: Screenshot of Penguin Hunt

For the standard controller condition, the left analog stick was used for movement, while the right analog stick was used for turning/aiming. The right trigger was used for shooting. The developers of the software reported that the angular rotation velocity was 270 degrees per second, following the Xbox version of Halo. Following the instructions of the Xbox controller programming manual, the displacement-velocity transfer curve was cubed [4]. For the trackball controller condition, the left analog stick was used for movement, and the right trigger for shooting, exactly like in the standard controller condition. The trackball was used for turning and camera control. For the keyboard and mouse condition, the W, S, A, and D keys were used for movement and the mouse was used for camera control. The left mouse button was used for shooting. The mouse and trackball had the same sensitivity, and pointing acceleration (gain) was turned on for both.

Each game trial lasted 120 s. The top right corner of the game screen showed the remaining time, and the number of targets hit so far. Hits and misses were accompanied by sounds to provide auditory feedback. The game logged the number of targets hit, the distance between the player and the target when a target was hit, and the number of misses.

The second task involved playing the PC version of Infinity Ward's Call of Duty 4: Modern Warfare, run at a resolution of 1366 × 768. This commercial game was chosen due to its training level. Part of the game's introduction involves the player completing an army-like obstacle course called the cargo ship mockup. The mockup involves navigating a "ship", moving from one station, or "position", to the next and completing objectives in each. Each position involves some offensive action, such as shooting cardboard targets, or throwing flashbangs (blinding grenades) into rooms. In total, there are 6 positions, 2 of which require throwing a flashbang, and 10 cardboard targets spread out across the positions. The entire event is timed, with the goal to complete all objectives as quickly as possible.

Looking directly at a thrown flashbang blinds the player (represented by the screen going white for a few seconds, and sounding a ringing noise). This has a negative impact on completion time. To avoid being blinded, the player either moves behind a wall or pans the camera away from the blast area. Overall, this level was a good comparative task as it is representative of actions in FPS games, while being short and identical between trials. Specifically, it involves navigation, camera control, aiming and shooting, taking cover (to not be blinded by flashbangs), sprinting, etc.

Completion of the obstacle course represents a trial, which participants repeated multiple times. Before a trial, participants restocked on flashbangs and ammo. The gun held 30 bullets, which was enough to shoot all targets without reloading, but reloading was still possible. At the end of a trial, the game displayed the time to finish the course and an accuracy score from 0 to 3. The accuracy score is a "black box", in that we do not know how it is calculated. The accuracy score is subtracted from the time to obtain the final time. In essence, the final time represents both speed and accuracy. Having no access to the game code, we manually recorded the time and accuracy of each trial. Figure 3 shows a screenshot of the cargo ship mockup level.

We used the PC version of Modern Warfare, with the default control configuration used for the keyboard and mouse condition. Specifically, the W, S, A, and D keys were used for movement, while the mouse was used for camera control. The right mouse button was used for looking down the sight of the gun (zooming in) and the left mouse button was used for shooting. SHIFT was used for sprinting, R for reloading, and '4' for throwing flashbangs.


Figure 3: Screenshot of Call of Duty 4: Modern Warfare

To use the standard and trackball controllers with the PC version of Modern Warfare, we used an emulator called XPAdder [9], which allows emulation of game controller input as mouse and keyboard input. For the standard controller condition, we used the default key configuration of the Xbox 360 version of the game. Specifically, we mapped the left analog stick to movement (W, S, A, and D) and the right analog stick to camera contro (mouse motion), with the left trigger for aiming down the sights and the right trigger for shooting. Pressing down the left analog stick was used for sprinting, 'X' was used for reloading, and the left shoulder button was used for throwing flashbangs. For the trackball controller condition, XPAdder was configured exactly as it was for the Xbox 360 controller, except for aiming and turning, for which the trackball was used instead.

XPAdder allows setting the analog stick sensitivity when emulating a mouse. The emulated sensitivity needed to match the sensitivity on the Xbox 360 version of the game. To that end, we compared the turning velocity of an emulated controller on the PC, to a controller on the Xbox 360 version of Modern Warfare. We set the sensitivity so that turning 360 degrees by maximum analog stick displacement takes as long with the emulated controller on the PC as on the Xbox 360 game.

Despite our attempt to emulate the Xbox 360 controller on the PC, the mapping was not exact. The Xbox 360 version of the game has a number of features, or "handicaps", which are not present on the PC version of the game. Specifically, aiming down the sights, or zooming in, on the Xbox 360 version of Modern Warfare slows down the camera panning speed, but it does not do so on the PC version. Additionally, pulling the left trigger to aim down the sights "locks on" to the nearest target (positions the crosshead on it) on the Xbox 360 version of the game, but not on the PC version. That Infinity Ward includes targeting assistance mechanisms on the console version of the game, but not on the PC version, indicates that developers are aware of the shortcomings of standard controllers in terms of accurate pointing.

2.3 Procedure

The purpose of the experiment was explained to participants. They were then asked to give informed consent for their participation and fill out a pre-experiment questionnaire. The first task was playing Penguin Hunt. Before each control scheme was used, participants were given a 120 s practice trial to familiarize themselves with the controls. The results of the practice session were discarded. Participants performed four recorded trials of 120 s each, with each controller.

After completing all trials on Penguin Hunt, with all controllers, participants switched to Modern Warfare. Participants were given a number of practice trials to familiarize themselves with the game, the goals, and the obstacle course. These familiarization trials were not recorded. Once participants were reasonably familiar, recording started. The results of the first trial with each controller were also not recorded, as they too were considered practice trials. Participants performed 10 recorded Modern Warfare trials with each controller. Each trial took approximately 30 s, which allowed us to have more trials than with Penguin Hunt. After completing both tasks, participants were asked to fill out a post-experiment questionnaire.

2.4 Design

The Penguin Hunt part of the experiment employed a 3 × 4 × 3 mixed design. There were three independent variables:

Input Device and Trials were within-subjects variables. Experience was a between-subjects variable. Input Device was counterbalanced according to a Latin square. Each trial lasted 120 s. In total, participants performed 4 (Trials) × 3 (Input Device) × 3 (Experience Groups) × 6 (participants per group) = 216 trials. The dependent variables were hits (count), misses (count), and distance to target (pixels).

The Modern Warfare part of the experiment employed a 3 × 10 × 3 mixed design. The independent variables were:

Input Device and Trials were within-subject variables. Experience was a between-subjects variable. Input Device was counterbalanced according to a Latin square. Each trial consisted of a completed cargo ship mockup level run. In total, participants performed 10 (Trials) × 3 (Input Devices) × 3 (Experience Groups) × 6 (participants per group) = 540 trials. The dependent variables were final time (seconds) and accuracy (score).

3 RESULTS AND DISCUSSION

3.1 Outliers

After completing the obstacle course in Modern Warfare, the game provides a recommended playing difficulty, based on the player's final time. The recommended difficulties are as follows:

We feel that this recommendation is accurate and we used it to validate participants' self-assessed proficiency. To be included in the study, each participant had to reach the Hardened difficulty on at least one trial, with any input device. In other words, at least one trial had to be completed in less than 26 s.

All but one participant met this requirement, which indicated that overall, participants' self-assessment was accurate. The one participant (from the advanced PC players group) who did not meet the requirement, and was likely not an advanced FPS player, was disqualified. We removed the participant's scores from the data in order to reduce variability. Additionally, a player from the advanced console players group was removed due to being an outlier. The participant had scores which were more than 3 standard deviations away from the mean. As a consequence, the data in the results are based on 16 participants instead of 18: five players advanced on the PC, five players advanced on consoles, and six players advanced on both.

3.2 Penguin Hunt

3.2.1 Hits

We first present the results for the three experience groups combined. Though a trial lasted 120 s, we present the results as the average number of hits per minute. Other research has used different trial lengths with Penguin Hunt, so reducing the score to a ratio makes it more comparable. The results are shown in Figure 4. All figures show standard error bars.


Figure 4: Penguin Hunt hits per minute, across the four trials.

Overall, the number of hits per minute using the keyboard and mouse was 28.1. This is 18.5% higher than the trackball controller's 22.9 hits per minute, and 22.8% higher than the standard controller's 21.7 hits per minute. Comparing just the console controllers, the trackball controller represents a 5.5% improvement over the standard controller. The difference between the input devices was statistically significant overall (F2,30 = 37.63, p < .0001), but a post hoc Tukey-Kramer test revealed the difference to only be significant between PC controls and standard controller, and between PC controls and trackball controller. The difference between the standard controller and the trackball controller was not significant. The order effect was also not significant (F2,30 = 0.33, ns), indicating that counterbalancing was effective. It is not surprising that the keyboard and mouse outperformed both console controllers. It was the expected result since, in terms of throughput, the mouse (4.5 bps) is known to be a faster and more accurate pointing device than the trackball (3.0 bps) and the analog stick (2.0 bps).

Based on the results, participants not advanced with PC controls still performed well with the keyboard and mouse, but participants not familiar with the standard controller performed poorly using it. This may be due to participants' familiarity with the keyboard and mouse for non-game tasks. Advanced PC players had an average of 26.9 hits per minute with PC controls. Advanced console players also performed well with PC controls, and had an average of 27.6 hits per minute. This indicates that it is easy to learn to play with the keyboard and mouse.

On the other hand, while advanced console players performed well with the standard controller, and had an average of 26.0 hits per minute, advanced PC players struggled with the standard controller, and had a lower average of 18.2 hits per minute. This indicates that it is more difficult to use the standard controller without practice. The Experience × Input Device interaction effect was significant (F2,30 = 11.83, p < .0001), with the Tukey-Kramer test revealing that there was a difference between console players and PC players using the standard controller, but no difference between the two when using PC controls.

The gap between the trackball and standard controllers is not as large as was hoped. This stands to reason, as two thirds of our participant pool were experienced with the standard controller, but none were experienced with the trackball controller. Despite this, the trackball controller had a higher average of hits per minute than the standard controller.

Based on the figures in previous research using Penguin Hunt, the average number of hits per minute using the Xbox 360 controller was between 13 and 18 [2, 4], lower than our average of 21.7. On the other hand, using PC controls, participants in previous research averaged between 23 and 28 hits per minute [2, 4], which is comparable to the 28.1 hits per minute observed with our participants. The difference in scores with the Xbox 360 controller may be due to prior participant experience. Two thirds of our participants were advanced players using standard controllers. The experience level of participants in previous research is not specified.

To better illustrate how each group of participants performed with the various controllers, we provide separate analyses based on experience group. The results for the advanced PC players are seen in Figure 5 (left). The average number of hits per minute was highest with PC controls, at 26.9. This is not surprising, since participants in this group were only experienced with keyboard and mouse gaming. The trackball controller had an average of 21.3 hits per minute, which was higher than the standard controller's average of 18.2. The trackball controller represents a 17.0% improvement over the standard controller for participants in this group. The difference based on Input Device was statistically significant, (F2,8 = 54.16, p < .0001), and the Tukey-Kramer test revealed that it was significant between all pairwise comparisons. This result is important, as it indicates that participants who are unfamiliar with either standard console controllers or the trackball controller, perform significantly better with the trackball controller. The value of 18.2 hits per minute obtained with the standard controller is comparable to the range of results seen in previous research [2, 4], which indicates that their participants were not advanced players with standard controllers.


Figure 5: Penguin Hunt hits per minute per group: advanced PC players (left), advanced players on both systems (middle), and advanced console players (right).

The results for participants who were advanced with both consoles and the PC are shown in Figure 5 (middle). PC controls once again perform best, with an average of 29.5 hits per minute. Participants again performed better using the trackball controller than the standard controller, with 23.4 hits per minute and 21.1 hits per minute respectively. The difference is statistically significant (F2,10 = 34.83, p < .0001), but the Tukey-Kramer test revealed no significance between the trackball controller and the standard controller. Though significance was not obtained between the two controllers, on average, participants performed better using the trackball controller, despite being advanced players with the standard controller.

Finally, we analyzed the results for participants who were advanced console players. These are shown in Figure 5 (right). Participants in this group did not perform best with the standard controller, which is surprising considering their experience. Instead, the highest number of hits per minute was obtained with PC controls: 27.6. Average hits per minute were 26.0 with the standard controller and 24.0 with the trackball controller. The difference was statistically significant (F2,8 = 12.33, p < .005), but a Tukey-Kramer test revealed significance only between the PC controls and the trackball controller. Despite participants in this group being only advanced with standard controllers, they did not perform better with it. This alone indicates deficiencies with the standard controller. It is likely that with equal experience the trackball controller, as well as PC controls, will perform better than the standard controller.

3.2.2 Misses

In addition to the number of hits, we counted the number of times participants missed with each device. On average, participants missed 6.6 times per minute using the keyboard and mouse, 4.1 times per minute using the trackball controller, and 6.2 times per minute using the standard controller. The difference was not statistically significant (F2,30 = 0.96, ns).

3.2.3 Distance to Target

For each target hit, we logged the distance in pixels between the target and player. We did not expect to obtain interesting results, due to participants' varying preferences and play styles, but a relationship did surface. The results for distance to target per trial are shown in Figure 6. The average distance to target was 130.9 px using PC controls, 114.6 px using the trackball controller and 101.4 px using the standard controller. The difference was statistically significant, (F2,30 = 11.43, p < .0005), but the Tukey-Kramer test revealed significance only between PC controls and the standard controller.


Figure 6: Penguin Hunt average distance to target, per trial.

Targets appear smaller when they are shot from a distance. This result signifies that participants felt most comfortable shooting at smaller, farther away targets when using the mouse, and least comfortable when using the standard controller. This in itself is a statement about the different levels of precision and accuracy of the three devices.

3.2.4 Summary

We have shown that advanced PC players, as well as advanced PC and console players, perform better in Penguin Hunt with the trackball controller than with the standard controller. We expect that with practice the benefit of the trackball controller will grow, and further increase the performance gap, even for advanced console players.

3.3 Call of Duty 4: Modern Warfare

3.3.1 Final Time

We present the results for the final time in Modern Warfare next. As noted, final time is the time for a participant to run the cargo ship mockup level, minus the accuracy bonus. Faster (lower) times are better. The average times per input device, across the 10 trials, are shown in Figure 7.


Figure 7: Modern Warfare final time per input device.

On average, participants were fastest using PC controls, with an average final time of 26.8 s. As hypothesized, the average time using the trackball controller was faster than using the standard controller. The average time using the keyboard and mouse is 18.6% faster than the trackball controller's average time of 31.8 s, and 32.5% faster than the standard controller's time of 35.5 s. Comparing just the console controllers, the trackball controller represents a 10.4% speed-up in trial completion time over the standard controller. The difference was statistically significant (F2,30 = 14.50, p < .0001), and the Tukey-Kramer test revealed significance between all pairwise comparisons. The effect of group was not significant (F2,30 = 1.64, p > .05), indicating that counterbalancing was effective.

This is an important result. Despite none of the participants having experience with the trackball controller, participants performed better using it than the standard controller. There is also an improvement across the trials for all devices, as participants become familiar with the task. Indeed, the difference based on trial is significant (F9,135 = 7.41, p < .0001), with the earlier trials (1 and 2) being slower than the later trials (5 - 10). Despite obtaining significance in the overall results, we analyzed the results individually by experience.

Participants in the advanced PC players group, and unfamiliar with either gamepad, performed better with the trackball than the standard controller by a significant margin. The results are shown in Figure 8 (left). The average time with the keyboard and mouse was fast: 26.7 s. The average time with the trackball controller was 33.9 s, almost 8 s faster than the standard controller's average time of 41.7 s. The trackball controller represents an 18.7% speed-up over the standard controller for this group. The difference was statistically significant (F2,8 = 3.61, p < .005), and the Tukey-Kramer test revealed that the significance was between the PC controls and the standard controller, and between the trackall controller and standard controller. The difference between the trackball controller and PC controls was not significant. This is similar to the results obtained with Penguin Hunt; participants who are unfamiliar with either console controller perform better with the trackball controller.


Figure 8: Modern Warfare final time per group: advanced PC players (left), advanced players on both systems (middle), and advanced console players (right).

The results for participants who are advanced players on both consoles and the PC are shown in Figure 8 (middle). The same pattern appears: the keyboard and mouse is fastest, and the standard controller is slowest; but the gap between the conditions is narrower than for the PC players' group. The average time was 25.1 s with the keyboard and mouse, 30.8 s with the trackball controller, and 33.4 s with the standard controller. The difference was statistically significant (F2,10 = 11.16, p < .005), but it is only significant between the PC controls and the two console controllers, and not significant between the trackball and the standard controller. Despite this, as with Penguin Hunt, the standard controller, on average, still performed worse than the trackball controller, despite participants' prior experience. We expect that given equal experience, the trackball controller will perform better by a larger margin.

The results for participants who were advanced console players are shown in Figure 8 (right), and it is clear that no well defined relationship exists. The average final time was 28.7 s for the keyboard and mouse, 29.6 s for the trackball controller, and 30.3 s for the standard controller. Not surprisingly, the difference is not statistically significant, (F2,8 = 0.52, ns). But the fact that participants, who are advanced at using the standard controller and nothing else do not perform any worse with other controllers, is telling.

3.3.2 Accuracy

We recorded the accuracy bonus awarded per trial. Though we do not know how the accuracy bonus is calculated (Infinity Ward has not disclosed this information), we know that it is based on the amount of misses, and that the minimum bonus is 0 and the maximum bonus is 3. The results for the average accuracy bonus are shown in Figure 9. As with other dependant variables, the best performance and highest accuracy bonus was achieved using the keyboard and mouse, and the worst using the standard controller. The average accuracy bonus was 2.6 using the keyboard and mouse, 1.7 using the trackball controller, and 1.1 using the standard controller. The difference was statistically significant (F2,30 = 41.51, p < .0001), with the Tukey-Kramer test showing significance between all pairwise comparisons.


Figure 9: Modern Warfare, accuracy bonus per trial.

3.3.3 Summary

These results mirror the results from Penguin Hunt, with the keyboard and mouse offering the best performance, the standard controller offering the worst performance, and the trackball controller offering midway performance. As before, the most visible gap between the two console controllers is for participants who are unfamiliar with both, while there is no difference for participants familiar with one but not the other. When all groups are combined, on average, participants were faster by 3.6 s with the trackball controller than the standard controller. This may not seem like a big improvement; however, it is important to remember that participants were largely experienced with one controller but not with the other. It is also worth noting that the Modern Warfare task was complicated, with a lot of randomness and variability, including luck, participant strategy, mistakes, reload times, flashbang blindness, sprinting or walking, aiming down the sights or firing from the hip, etc. The fact that despite all these variables, and by changing only the method of aiming with the controller, participants were able to shave 3.6 s off their time (a 10.4% speed-up), while using an unfamiliar controller indicates a considerable improvement indeed!

3.4 Qualitative Results

At the end of the experiment participants completed a questionnaire. They were asked to rank the input devices in terms of playing preference for each game. For Penguin Hunt, 15 out of 16 participants (93.8%) rated the trackball controller ahead of the standard controller in terms of preference — an overwhelming majority. For Modern Warfare, 11 out of 16 participants (68.8%) rated the trackball controller ahead of the standard controller. It appears that the increase in performance offered by the trackball controller over the standard controller is also reflected in participants' preference of each.

4 CONCLUSION

This paper presents the gameplay evaluation of a prototype trackball controller. The prototype replaces the right analog stick of a standard game controller with a trackball. We compared the trackball controller's in-game performance to that of the keyboard and mouse, and the standard controller in two game tasks. In Penguin Hunt participants were asked to shoot as many targets as possible with each controller, in a 120-second trial. In Modern Warfare participants were asked to complete an obstacle course as quickly as possible with each controller.

Results confirmed our hypothesis. In both tasks, PC controls offered the best performance, while the standard controller offered the worst performance. The trackball controller provided a performance compromise, bridging the gap between the two. Considering just the standard controller and the trackball controller reveals that the trackball controller offered better performance in both games. The gap in performance was widest for participants with no experience with either controller, and narrowest for participants experienced with the standard controller but not the trackball controller. This is not surprising, and we expect that with practice, users in all experience categories will perform better with the trackball controller than the standard controller. Overall, the results conclusively show that our trackball controller offers a significant performance improvement over the current standard of game controllers. Though it does not offer the accuracy and precision of a keyboard and mouse, it is fully usable in a handheld fashion without a surface to operate on (as with a mouse).

Additionally, though our experiment only involved First Person Shooters, we are confident that the results are applicable to any genre of games involving camera control or pointing, including Real Time Strategy (RTS) games, and Role Playing Games (RPGs). Since gaming is competitive and goal-oriented, we believe that the observed improvement in performance is broadly applicable to game playing.

5 ACKNOWLEDGMENTS

We thank Poika Isokoski and Benoît Martin for permission to use the Penguin Hunt software. We also thank Andreea Izdraila for her editorial help. This research is sponsored by the Natural Sciences and Engineering Research Council (NSERC) of Canada.

6 REFERENCES

[1]    Infinity-Ward, Call of Duty 4: Modern Warfare: Activision.

[2] Isokoski, P. and Martin, B., Eye tracker input in first person shooter games, Proceedings of the 2nd conference on Communication by Gaze Interaction. Turin, 2006, 76-79.

[3] Isokoski, P. and Martin, B., Penguin Hunt, 2006.

[4] Isokoski, P. and Martin, B., Performance of input devices in FPS target acquisition, Proceedings of the International Conference on Advances in Computer Entertainment Technology. New York: ACM, 2007, 240-241.

[5] Klochek, C. and MacKenzie, I. S., Performance measures of game controllers in a three-dimensional environment, Proceedings of Graphics Interface 2006. Toronto: Canadian Information Processing Society 2006, 73-79.

[6] MacKenzie, I. S., Fitts' law as a research and design tool in human- computer interaction, Human Computer Interaction 7, 1992, 91- 139.

[7] Natapov, D., Castellucci, S. J., and MacKenzie, I. S., ISO 9241-9 evaluation of video game controllers, Proceedings of Graphics Interface 2009. Toronto: Canadian Information Processing Society, 2009, 223-230.

[8] Natapov, D. and MacKenzie, I. S., The Trackball Controller: Improving the analog stick, Proceedings of FuturePlay 2010, Accepted for publication.

[9] XPAdder, http://xpadder.com. Accessed September 2009.