TIGSource Forums

Hi,

has there actually ever been any kind of research on how to mathematically define and measure the difficulty of video games? I guess this could be useful for analysing and procedurally generating games. Although of course you would probably need different definitions depending on the genre of the game(eg turn-based-strategy vs shoot 'em up)
How do you go about realizing a difficulty curve in your game? Do you have any ideas of methods for estimating the difficulty?

You can't really "objectively" measure the difficulty of a game, because it's relative to the player's skill level. Games don't have a preset difficulty which applies for everybody; e.g. I know of people who 1CCed DeathSmiles on literally their first try, but I'm sure quite a few people might never be able to 1CC it.

The only real way I can think of to estimate the difficulty of a game would be to, well, play it. Even if it could be just boiled down into a number, there are a ridiculous number of factors to take into account even for something like a regular 2D action game (physics, how the level designs complement the physics, enemy patterns, speed of the action, etc.) that can only be appreciated (at this point, anyway; the situation will have probably changed in a few decades or so) by a person. If you're developing it, get other people to play it if you want, e.g. your friends or family members.

I beg to differ. There's the complexity of gameplay mechanics, whether they're twitch-based, turn-based, or menu-based; there's the level design and how well it teaches you these mechanics, how often they test your use of them and the severity of punishment for failing said tests (IE: take some damage and redo? drain some timer backtracking? instant death?). There's also extra life/continue scarcity, and how much of the game actually does consist of these tests, and how much of it is "fluff" in between them.

Right, but how can you quantify those things mathematically? It'd have to depend on your own personal preferences for difficulty (or maybe the preferences of your "target audiences.") And things like "how well a game teaches you the mechanics" or "life scarcity/severity of punishment" would be seen differently by players of varying skill levels and tastes (as well as everything integrating into each other, like I said; you can't evaluate a game's punishment mechanism or health scarcity without taking into account the level design, and vice versa.)

The different player views don't matter here.
You surely can measure the difficulty of the game objectively. The fact, that different people will feel it differently doesn't matter, cause its two different things...
Though, for every gametype/genre, the method/formula will be different.

It's safe to say there is no general solution to this problem. People still use playtesting to measure difficulty.

You should do your own research though. But don't focus on general solutions. Pick specific gameplay and try to break it apart. Then, test your theories using Internet/twitter/flash portals.

I would imagine that if one really want numerical data from most games' difficulty, it had to be done with statistic analysis. Meaning lots and lots playing and then extracting results from that data mass.

Then again I suppose that certain genres, like managements and games which already rely heavily on numbers could be analyzed more easily mathematically and possibly result some kind of formula. Then again, I'm not a mathematician so...  :shrug2:

Difficulty in game is a multidimensional scale who rely on many interdependent parameter.

In increasing order of priority (the first lead to the last)

Visibility: how clear information pop out
Readability: how much information is understandable
immediacy: how fast the information is process
complexity: how much information is presented at the same time

recognition: how difficult it is to recognize pattern of information
memorization: how much prior knowledge is needed
depth: how much behavior anticipation is needed (yomi layer)

effort: how many input you have to enter
timing: how much input rely on time (time window/speed)
complication: how different the various input need to be
danger: how close/fast to failure it lead.

For example a one screen straight line require no timing, no complication, no effort, is extremely readable, is entirely visible, might be readable, is immediate, has low complexity, low recognition, no memorization and no depth, no danger.

A maze like corridor that span multiple screen require many input to take corner (takes more effort), no timing, little complication, not entirely visible, has little visibility, is not immediate, is complex, has no recognition, has no memorization, has no depth, no danger.

Toss some branching path and you increase, complexity (need to track various path), recognition (detect the right path), memorization (which path is the good one).

Toss some trap like moving seesaw and you increase danger and timing.

Toss an AI and you increase depth.

Procedural tend to decrease memorization in favor of recognition.

edit:

Couldn't one see difficulty as the probability that the player loses the game? Then the difficulty would be
(the number of all possible sequences of actions that lead to game over)/(the number of all possible sequences of actions that exist)

Of course that's a rather rough estimation as it assumes that all sequences have equal probability. One would probably have to ignore sequences that are so obviously useless that no player would consider them.
However think that it than could be used for example stuff like turn based battles or sokoband puzzles, as they usually require relatively few actions.

EDIT: Sorry, the probability of each path would be the product of the probabilities of its actions. and coming up with a useful probability estemate for each action is probably most difficulty part of this.

It's too thing: trouble to progress, probability of losing.

If a player do not progress he quit, which equate to losing.

So the aspect who interfere with difficulty are the thing in my list. (ability to find a solution, to understand what happen and execute the solution to progress/not lose). Measure them and you have difficulty.

It's easy to estimate, how many variable there is (number of enemy, stat, status, etc...) vs margin of maneuver (degree of freedom, acton possible, health remaining, etc...) considering that interface (readability) is already sort out and constant.

Yes, at least implicitly in research on dynamic difficulty adjustment (dda). But this is typically "difficulty for the current player" not "difficulty for the average player" (or for some more general type of player).  Here's one academic paper on dda for example:
https://www.aaai.org/Papers/Workshops/2004/WS-04-04/WS04-04-019.pdf
(skimming the paper, the metric in this case seems to be "if the player is taking lots of damage they're probably in a difficult spot")

On a similar note I believe valve and others create visualizations of difficulty-related metrics, like a heat map of a level where the color indicates how many players died there.

One could get some play testers and do statistics.

Players skills are normally distributed about a mean, and the game should reflect that.

ie, Consider the number that matters most, be it score in an arcade game, how far a player progresses in an FPS game, or %wins/%losses in PvP games. Then tweak the game until this figure become normally distributed.

-- If the mean is too high, then there is no impetus to improve one's play, because all players can already get far. That can make a game boring and long winded.
-- If the mean is too low, then all players aren't doing well.
-- If the standard deviation is too low then both low and high skilled players will perform similarly. This can make a game feel stupid, as next to nothing is gained by trying to improve at it.
-- If the standard deviation is too high then this can also make a game feel stupid, as a little improvement takes you much further than you'd like.


If you want to quantify difficulty, then the quotient of (standard deviation / mean) may measure that.

If this figure is approximately one, then skilled gamers (top 15%) will get twice as far as average gamers. the top 1% of gamers will get three-four times as far as average gamers. (While the bottom 15% of gamers will find the game too difficult to play)

Those sort of figures sound about right to me.

Jasmine: "The game got too hard, so I gave up"? That's what you want people to take away from your games? Arcade games, sure, but I don't think that approach is generally applicable.

But yeah. Playtesting and analytics. Rather than trying to work out some precise measure of difficulty - honestly, don't bother, it's different for every person and it's certainly not one-dimensional - the two things you're concerned about are when people get stuck and when they get bored. Difficulty, poor signposting and poor affordances are big factors in the former but the latter's as likely to be due to excessive repetition, a lack of mechanical depth, or even just not engaging with the fiction.

This is where the research you're interested in (http://uk.gamespot.com/features/the-science-of-playtesting-6323661/) is being done. Not on dry, academic theories of difficulty but how to run objective playtesting sessions and interpret them, because that's what's useful.

I was mostly thinking of arcade games when I wrote that, but I also said there are other numbers than %progress one could measure. eg, %wins/%losses in PvP experiences.

But I do believe that all games should ramp up difficulty to the point that players won't be able to truely master the game, whether that difficulty comes from the level design, or from PvP competition.

Some of the most celebrated games are like this.
Think of Jet-Set-Willy: After 30 years, still nobody has ever completed it without cheating.
Think of Chess: Players can't hope for 100% wins, because nobody has solved the game.
Think of Starcraft: Players can't hope for 100% wins because there are no absolute best players.

Isn't that because the original Jet Set Willy contains gamebreaking bugs? (http://en.wikipedia.org/wiki/Jet_Set_Willy#Bugs)

If you mean that nobody has completed a version of Jet Set Willy with the bugs that made it unwinnable fixed, well... I've heard a rumor that some people completed I Wanna Be The Guy on Impossible mode, so this couldn't be much more difficult than that, right?

Pretty much just need to watch people play in my experience. People who have never played it before, and don't help them!

you can objectively measure things which vary from person to person. for instance, eyesight varies from person to person. but you can still objectively measure eyesight, and give someone a prescription

difficulty could be measured similarly: you can measure how hard something is for a person. there are various units you could use: average time it takes to beat the game, % of the people who managed to finish a game, % of the population which beats a game in under x hours, % of people who managed to finish the first level without dying the first time they play the game, and lots of other ways to measure it. there's no standardized unit right now, but that doesn't mean one could never become widespread

The problem is that eyesight doesn't get better with practice.

You also can't really use a specific mechanic to objectively measure difficulty unless every game uses that mechanic.

Game difficulty will always be subjective-- not only do players have different skill levels, they also have different skill levels PER GAME. And while say beating Megaman 9 WOULD make someone better at Megaman 10, they still have to practice Megaman 10 to beat it (just the way those games are designed) and it does absolutely nothing for their skill in say, basketball.

And while something like Super Meat Boy is accepted as incredibly difficult, it is still something that you can practice and eventually learn to master. Meanwhile you can master Solitaire in an afternoon, but you will always have that same rate of failure due to random chance.

eyesight actually does get better if you exercise it

You can, in fact, objectively measure the difficulty of a game.¹ Check Metrics #1 and #2 in this Gamasutra article about analytics (http://www.gamasutra.com/blogs/BenChong/20111121/8938/Indie_Game_Analytics_101.php). Players kept dying in the same places, and dying way too often in certain levels, and then gave up more often than not. That's objective proof of difficulty.

This sort of analysis shows you if a level is harder than its neighbours, where it's the most difficult, and gives you some idea of what to tweak to change this. In this case the author hired a company to do this sort of thing, but I'm sure you could program it into the game itself to build a database of information that could be graphed and plotted. The inverse can also be figured out: hardly anyone dies in level X, but level X is near the end of the game, why? It's too easy. Average level completion time is measured in seconds instead of minutes, why? It's too easy.

¹ IMO, it's pointless to say one game is more difficult than another, unless they're virtually the same game. Factor in different genres and why would you even bother comparing them?

measuring very simple statistics like that can be done with steam achievements, indirectly. some games have an achievement for doing different things -- beating the game, beating a particular level, finding a particular thing, or whatever. and steam shows the % of all people who own the game who accomplished that achievement. since most games have a 'beat the game' achievement, you can get a fair sense of the game's difficulty (or length, or boringness, among other factors) by looking at what % of the people who bought the game beat the game

as an example, see this forum's own zaratustra's game eversion; its achievements on steam are interesting: http://steamcommunity.com/stats/Eversion/achievements

92% of people beat level 1
86% beat level 2
84.8% beat level 3
64.6% beat level 4
63.7% beat level 5
61.9% beat level 6
54.6% beat the game in the normal way
29.9% got the best ending

just from looking at that, it's obvious that level 4 is the hardest level, with a drop-off of 20.2% of players who beat level 3 but not level 4. this is followed by level 1 (8% drop-off) and the final level (7.3% drop-off). for the first level the effect may have to do with some people who own the game not even bothering to play it, however, so that can probably be discounted

I think you can measure it for some aspects. But there is no mathematical way to do it in general, like the cone-response-function for how we see graphics was actually just measured and is simply statistic. So you have simply to refer to statistic. If you represent the statistic fairly well then you can judge the difficulty well by your own:) 

Oh wow. That explains why so many games give me dumb achievements for completing a level...probably also just designer laziness (I've been guilty of this myself), but that is a very good use for achievements. But then again, on some platforms (iOS) you could just have it send you data every time someone beats a level.

I didn't know this about the achievements paradigm. But I think you need more than one dimension to form this sort of conclusion. If you don't factor in deaths, the drop-off (at any point) can be explained by people getting bored and not continuing – a low death rate followed by a sharp drop-off would also show you that – which is just as likely when the drop-off occurs at the end of level 1.


I would hope once the game leaves beta testing this sort of thing would be unnecessary.

If beta testing was sufficient for everything, Blizzard/Valve wouldn't be constantly tweaking their games after release :) You should definitely beta test, but you can't catch everything until it's released in the world.