I used to pester the user research team every time I heard a playtest was scheduled. Can you ask if they understood the narrative? Can you ask if they prefer the way this gun sounds or that one? I wanted feedback from people who weren’t on the project team badly enough that I made myself a recurring inconvenience. Getting honest reactions to audio or narrative from other developers is genuinely difficult — the number of people on project teams who have admitted to me that they turn the sound off completely when playtesting is quietly depressing, but that’s a different post.
The user researcher pushed back almost every time. Sometimes the questions were too subjective for the test structure. Sometimes adding them would bloat the questionnaire to the point where playtesters lost interest and compromised the rest of the survey. Sometimes they simply didn’t have the time or tools to build an adequate test around what I was asking. They were right, usually, even when they couldn’t fully articulate why. I didn’t understand their reasoning at the time and even found it frustrating. It wasn’t until I learned what success metrics actually were — and watched them used both well and badly — that I understood what they were protecting.
What A Success Metric Is And Isn’t
You’ll hear the term KPI — key performance indicator — used in game development to describe a few different things. At the business level, KPIs tend to be macro and persistent: monthly active users, retention rates, revenue figures tracked over months or years. What I’m talking about here is something more specific and more useful during development. Some studios will still refer to these as KPIs, and it’s not incorrect to do so — but I think a more appropriate name is success metrics. Feature level measurements used during playtesting and iteration to evaluate whether something is actually working. Not whether it feels like it’s working. Whether it is.
If you’ve read the pillars posts, you already have half the framework. Pillars define what the game should be — the high level rules you test decisions against. Success metrics are the layer below that. They’re how you prove what you’re building is having the effect and providing the experience your pillars shaped.
The distinction between a real success metric and a fake one matters more than it might seem, because a lot of what gets called a metric in game development isn’t one. A measurement that can be interpreted differently by different people isn’t a metric — it’s an opinion with a name attached to it. And opinions, in a room full of people with different relationships to the work, different levels of investment, and different amounts of context about how hard something was to build, tend to resolve in favor of whoever is loudest. (Opinions are also important in their own way, a future post to discuss that is in the works.)
A real success metric removes that variable. It produces a number. The number means what it means regardless of who’s looking at it.
The Subjective Trap
Here’s a question that seems like a reasonable success metric: did the playtest participant indicate that they understood the gameplay?
It’s a valid thing to want to know. Getting feedback from players who aren’t developers is genuinely valuable, and understanding whether your game is communicating its own rules is a real and important question. But “did they indicate they understood” is subjective data. One participant might mark yes because they’re being polite. Another might mark no because they had a bad session. Someone reviewing the results might weight the yeses more heavily because they observed what looked like understanding. Someone else might weight the nos because they don’t understand the feature at all. The data can be argued with, which means it will be argued with, which means it doesn’t actually measure anything.
If you want to know empirically whether a player understood the gameplay, you need a structure that can’t be argued against. How many checkpoints did the player complete? How many times did they die in the first hour? How long did they spend in the inventory screen? These numbers are facts. They don’t care how much work went into the feature. They don’t care whether the reviewer loves it or hates it or just wants to be done. They don’t change based on who’s in the room.
Context can affect what the numbers mean — a player who is deeply familiar with the genre will produce different numbers than one who isn’t, and that’s worth accounting for in how you set targets for different audience segments. But the numbers themselves don’t lie and can’t be interpreted differently. That’s the point.
Not everything has a clean numerical success metric. Audio is genuinely difficult to measure in isolation — it’s one of the harder disciplines to build objective indicators around, and sometimes you need to build telemetry tools that don’t exist yet before you can track what you need to track. But even in cases where a direct number is hard to find, you can often get closer than a broad survey question. If the player is supposed to hear the name of the main antagonist five times during the quest being tested, ask them afterward: do you know who X is? Did you hear the name X during your playthrough? That’s not a feeling. That’s much closer to a fact you can track.
A Line, Not A Gate
One of the most common misuses of success metrics is treating them as pass/fail. Feature either passes or it doesn’t, ship it or cut it, done. That framing misses most of what makes them useful.
A success metric is most valuable when it’s a line you can track across multiple tests over time. You run the playtest. You get a number. You adjust something. You run it again. The line moves — dramatically or barely, in the direction you expected or somewhere surprising. That movement is the information.
If you make a significant design change and the line barely moves, the design change probably didn’t affect the thing you were trying to affect. If you change a sound, or an animation, or a UI element, and the line jumps — that’s the data telling you something. Not suggesting it. Telling you.
This is where success metrics become something more than a measurement tool. They become a way of demonstrating, objectively and without room for argument, which changes actually mattered and can prevent big changes in design that little changes in UX (user experience) might address far better. And that has implications that go beyond any single feature. (It’s also a common scope planning trap we can go into another time.)
What The Number Actually Proves
There is a persistent assumption in game development that design is the primary driver of whether a feature works. Design is important — obviously. But it isn’t everything, and anyone who has watched a feature transform when the right sound was added, or the right animation landed, or the UI finally communicated what was actually happening, knows this intuitively.
Success metrics can prove it. If a feature isn’t working and the design hasn’t changed but a sound did, and the metric moves — the data says the sound mattered. Not the lead on that feature. Not the most senior person in the room. The number. And a number is harder to dismiss than an opinion, even a well-informed one.
In my experience, this is one of the underappreciated values of proper success metric tracking — it shines a light on the contributions of the multiple disciplines that made a feature work. Over time — across a project, or multiple projects — that kind of repeated evaluation can shape a studio’s workflow in ways that make everyone more effective, because everyone is learning from the same data together. Audio, animation, UI, narrative — these aren’t polish. They’re often the difference. Success metrics, used correctly, can make that visible in a way that subjective discussion rarely can.
A Note On Scope
Everything in this post is about feature level success metrics in game development — measuring whether the things you’re building are working, and using that data to make better decisions. That’s a different conversation from KPIs as they apply to developer performance and team metrics, which is its own topic with its own complications. That one’s worth getting into separately, another time.
For now: get the number. Track the line. Let the data speak for itself.