What Players Do In Usability Tests

There are 17 buttons on the PS3 controller and two analog sticks with variable input. Can you name them all? Do you know what they all do? Can you remember what they do in the middle of an intense combat scenario? Can you remember how to switch weapons, switch grenades, throw grenades, shoot weapons, use alt fire, aim down your sights, crouch behind cover, strafe, melee, bring up your objectives, sprint, reload, pick up items, heal allies, craft molotov cocktails… all while getting shot at? Did you even notice you were getting shot at?!

(That’s not even getting into touch pads, motion sensing, cameras interpreting your every move, voice control, and 3D!)

We expect a lot from players. Games are complex. The more detail in your games, the more buttons, the more things the player can do in your world, and the more people you want to play your game, the harder it is to deliver something that is understandable and non-frustrating, let alone fun to play.

How a Usability Test Works

What happens from a player’s point of view:

    • You are recruited for a usability test (from various sources)
    • You fill out some questionnaire about your game habits to make sure you fit our demographic (i.e.,  you play similar games, play X hours a week, own a console)
    • You arrive at the developer or publisher and are escorted into the room with console and monitor.
    • There’s sometimes a cheat sheet on controls or a bit of backstory, depending on what part of the game you will be playing.
    • There’s a camera watching you and a speaker for someone to talk to you from a different room.
    • You play the game for 1-4 hours. You might return the next day.
    • At the end they’ll interview you a bit, and often have you fill out another survey, on what you thought, fun and frustration levels, games you’d compare it to, etc.

What happens from a developer’s point of view:

    • If the usability test is done at the studio, there’s usually a room with various leads, project managers, QA, designers, and other misc. people.
    • If the usability test is done offsite, you sometimes get a video feed you can watch from your desk while a couple leads go to the location to be there in person.
    • You watch two screens – one of the playtester in the room so you can see their expressions, and one that’s a video feed of their gameplay.
    • There’s usually a usability person there to handle most of the details and talk with the player (they’re better about not accidentally leading the player when asking questions).
    • Sometimes there’s donuts or pizza.
    • You write down notes that will turn into future tasks or bugs to fix.
    • You cry as the player takes your carefully crafted experience you spent the last two years working on and shows you all its ugly flaws.

So note, these are players familiar with the genre conventions and control schemes. They are gamers, many of whom identify as “hardcore” players, who play our games and games like ours and read about games on popular gaming websites. They represent a cross-section of regular players, not the overly competitive players, not the most expert users, not the journalists whose job it is to analyze games.

What Players Actually Do

Here’s the stuff I’ve seen in usability tests. I’ve narrowed it down to shooters specifically for this post, else the list would be way too long:

    • Find a new gun. Never use it.
    • Find a new gun. Use it and never switch back to a previous gun.
    • Play through a five-minute tutorial on alt-fire abilities on your weapons. Never use alt-fire again.
    • Pick up a grenade. Never use it.
    • Attempt to pick up a grenade, ammo drop, or extra health even though you have full ammo, full health, and full grenades. Use all your buttons to try to pick them up.
    • Attempt to pick up a piece of intel. Throw all your grenades instead.
    • Carry a heavy weapon that has low ammo and slows your character down considerably. Carry it up three flights of stairs with no combat. Spend two minutes trying to get it onto the elevator. Do this while your other three teammates wait.
    • Try to follow an objective marker, but there’s a wall between you and it. Jump at the wall. Melee the wall.
    • Jump at an indistinguishable rock for at least two minutes trying to get on top of it.
    • Immediately after starting the game, turn around and jump at the wall for at least a minute.
    • Melee doors that can’t open. Shoot doors that can’t open. Throw grenades at doors that can’t open.
    • Shoot your allies with every type of gun, grenade, and use every melee attack you can. Do this while they are delivering a passionate speech about rallying the troops, or when they are sharing a personal loss with you.
    • Stare at the floor. Especially during cinematic moments or while getting shot.
    • If you’re not staring at the floor, make sure to point your camera so all interesting events like boss intros, cave-ins, helicopter crashes, volcanoes transforming into secret military bases, ally deaths, and nuclear explosions are juuuuuust out of view.*
    • Attempt to interact with every single computer screen, box, briefcase, wire, piece of paper, light switch, and coffee mug in the room while your ally tells you to hurry up we’re under attack! Do this in the next four rooms, too.
    • Melee everything. EVERYTHING. Don’t press any other buttons than the melee button.
    • Melee bosses.
    • Melee flying bosses equipped with rocket launchers.
    • Attempt to jump on top of the flying boss enemy. Note: there is a chasm below you.
    • Melee an explosive barrel.
    • When attacking a boss, ignore the bright glowing pulsating orange weakpoint. Instead, shoot everywhere except at the weakpoint.
    • If your ally tells you, “Shoot at its eyes! Its EYES! Shoot the eyes!” shoot everywhere except the eyes.
    • If a boss has a shield, ignore the shield and just shoot the boss with your gun. Don’t try to take the shield down. (What shield? The one that’s a big blue sphere around the enemy and lights up every time you shoot it. The one your allies keep telling you about.)
    • Die a lot because the game is really hard, but don’t use the rocket launcher in your pocket or the mannable turret next to you.
    • In a multiplayer campaign, run forward as fast as you can. Don’t let anyone else to trigger those cinematic events but you. Definitely don’t wait for any of your teammates to catch up.
    • In a multiplayer campaign, walk in the opposite direction than the rest of the players.
    • In a multiplayer campaign, if the objective requires all players to pick up their new guns, do not take the gun. Ignore the other players communicating to you that the door won’t open unless you pick up the gun. Ignore the objective marker, objective text, allies pointing at the gun, dialogue telling you to take the gun, and hint text pop-ups that tell you that you cannot progress until you take the gun. Instead, wait at the door.
    • In a multiplayer campaign, when you go into a downed state, crawl away from your allies.
    • In a multiplayer campaign, when your downed ally crawls next to you and asks you to revive them and a giant red indicator tells you they are dying, run away.
    • Run at enemies. Especially if it’s a cover shooter. It’s okay if you die because you can just try running at the enemies again.**
    • Run past all the enemies. Do not engage with them. Just avoid them and try to get to the exit. Completely bypass six minutes of intense, carefully crafted gameplay with the sprint button.
    • Run at enemy turrets.
    • Run into laser beams.
    • Attempt to melee or push your allies into laser beams.
    • If a crate doesn’t break when you melee it, then unload all your ammo into it.
    • The objective tells you to “Climb to the top of the sewers.” You walk around in circles at the bottom.
    • Normally play games with inverted X-axis controls. Not Y-axis – X. Where you swap left and right. Attempt to play a first-person shooter where this is not an option.
    • Use the analog sticks as though you can only move one of them at a time. Need to move the camera? Better stop moving. Need to move sideways or forward? Don’t touch that camera.
    • Don’t move the camera at all. Do not use the left analog stick. Attempt to navigate through the level facing the same direction the entire time and rely on your teammates to describe to you how to get around cover that is in your way as you walk backwards. (This was not an isolated incident…)
    • Critique the graphics. Whenever someone asks you about gameplay, bring it back to the visuals. If the developer explains the game is a year from being shipped, continue to comment on the gray textures and placeholder cinematics.

Mind you, I am not calling players stupid. What you learn from sitting in on many usability tests is just how much you take for granted about what players know. You realize how little you’ve taught them. You realize all the different ways in which players have a bad experience or just give up in frustration.  You’ve spent two or three years working on a game and know all the mechanics and how it’s supposed to be played. A new player doesn’t.

On the flip side, usability tests can also lie. Players tend to tell you how much they enjoyed it and go blank when you ask them if there was anything they didn’t like, even though they died constantly and nearly threw the controller on the ground. Sometimes they do things they wouldn’t normally do just to see what happens because in a way they have been put on a stage to perform. Often they don’t spend as much time in exploring and completing secondary objectives because they feel pressured by time and unspoken expectations of the people in the other room.


For something a bit more useful, in my experience usability issues mostly fall into the following categories:

    • Player does not understand or internalize a secondary mechanic: “I spend all my time shooting enemies, but I didn’t realize or forgot there were grenades, alt fire, decoys, turrets, vehicles, or crafting.”
    • Player does not remember how to do an action or they find it too complex, so they skip it: “I know I have four guns but I forgot how to switch between them so I guess I will just stick to this one for the rest of the game.”
    • Player does an action but it is too difficult, complex, or results are inconsistent: “I dropped the grenade at my feet. While driving a vehicle to the exit, I constantly crashed it. It takes me too long to use a special gun and I die when trying to aim it.”
    • Player needs to do a specific action and does not know it: “I need to kill the boss. I am shooting it. Why isn’t it dying? You mean I need to shoot its eyes first? How was I supposed to know that?”
    • Player does not comprehend narrative moments and how they affect their current objectives: “I know we’re going to a bridge but I don’t know why. Who are these allies again? Why am I hacking this thing?”
    • Player does not understand what is or is not interactive: “I hacked a computer once before, so I should try to hack every computer I see. My objective tells me to climb but I don’t see anything I can climb on.”
    • Player doesn’t get appropriate feedback for actions or their consequences: “I died. I don’t know why I died. What killed me? Was I low on health? I didn’t even notice.”
    • Player is lost: “I don’t know where I am supposed to go. All these rooms look alike. Was I here before? The objective marker is telling me to go somewhere on the other side of this wall, but I don’t know how to get there.”
    • Player is frustrated and not having fun: “I have given up on the core gameplay so now I will just run at enemies and melee them.”

Some problems are fixed with a variety of clues (audio, visual, lighting, HUD indicators, hint text, dialogue, more granular objectives). Some are fixed by removing an element (cut a secondary ability because it’s just too complex or not useful or not fun, remove extra doors or computer consoles distracting the player from their goal). Others are fixed by adding a tutorial (gun ranges, training modes, situations where your abilities are limited and you’re asked to do very specific things, taking control from the player to show them a cinematic moment). And others we choose not to address because it’s totally okay so long as the player knows or learns the expected result (i.e. it’s okay to melee a boss enemy and die, so long as they don’t think they NEED to melee the boss and the boss reacts appropriately).

* The closer you get to seeing it without actually viewing the event, the more the developers cry.
** This is also me. I run at enemies.