next »

[Kinimod]

Hi,

has there actually ever been any kind of research on how to mathematically define and measure the difficulty of video games? I guess this could be useful for analysing and procedurally generating games. Although of course you would probably need different definitions depending on the genre of the game(eg turn-based-strategy vs shoot 'em up)
How do you go about realizing a difficulty curve in your game? Do you have any ideas of methods for estimating the difficulty?

[DavidCaruso]

You can't really "objectively" measure the difficulty of a game, because it's relative to the player's skill level. Games don't have a preset difficulty which applies for everybody; e.g. I know of people who 1CCed DeathSmiles on literally their first try, but I'm sure quite a few people might never be able to 1CC it.

The only real way I can think of to estimate the difficulty of a game would be to, well, play it. Even if it could be just boiled down into a number, there are a ridiculous number of factors to take into account even for something like a regular 2D action game (physics, how the level designs complement the physics, enemy patterns, speed of the action, etc.) that can only be appreciated (at this point, anyway; the situation will have probably changed in a few decades or so) by a person. If you're developing it, get other people to play it if you want, e.g. your friends or family members.

[baconman]

I beg to differ. There's the complexity of gameplay mechanics, whether they're twitch-based, turn-based, or menu-based; there's the level design and how well it teaches you these mechanics, how often they test your use of them and the severity of punishment for failing said tests (IE: take some damage and redo? drain some timer backtracking? instant death?). There's also extra life/continue scarcity, and how much of the game actually does consist of these tests, and how much of it is "fluff" in between them.

[DavidCaruso]

Right, but how can you quantify those things mathematically? It'd have to depend on your own personal preferences for difficulty (or maybe the preferences of your "target audiences.") And things like "how well a game teaches you the mechanics" or "life scarcity/severity of punishment" would be seen differently by players of varying skill levels and tastes (as well as everything integrating into each other, like I said; you can't evaluate a game's punishment mechanism or health scarcity without taking into account the level design, and vice versa.)

[rivon]

The different player views don't matter here.
You surely can measure the difficulty of the game objectively. The fact, that different people will feel it differently doesn't matter, cause its two different things...
Though, for every gametype/genre, the method/formula will be different.

[mirosurabu]

It's safe to say there is no general solution to this problem. People still use playtesting to measure difficulty.

You should do your own research though. But don't focus on general solutions. Pick specific gameplay and try to break it apart. Then, test your theories using Internet/twitter/flash portals.

[Tumetsu]

I would imagine that if one really want numerical data from most games' difficulty, it had to be done with statistic analysis. Meaning lots and lots playing and then extracting results from that data mass.

Then again I suppose that certain genres, like managements and games which already rely heavily on numbers could be analyzed more easily mathematically and possibly result some kind of formula. Then again, I'm not a mathematician so... 

[gimymblert]

Difficulty in game is a multidimensional scale who rely on many interdependent parameter.

In increasing order of priority (the first lead to the last)

Visibility: how clear information pop out
Readability: how much information is understandable
immediacy: how fast the information is process
complexity: how much information is presented at the same time

recognition: how difficult it is to recognize pattern of information
memorization: how much prior knowledge is needed
depth: how much behavior anticipation is needed (yomi layer)

effort: how many input you have to enter
timing: how much input rely on time (time window/speed)
complication: how different the various input need to be
danger: how close/fast to failure it lead.

For example a one screen straight line require no timing, no complication, no effort, is extremely readable, is entirely visible, might be readable, is immediate, has low complexity, low recognition, no memorization and no depth, no danger.

A maze like corridor that span multiple screen require many input to take corner (takes more effort), no timing, little complication, not entirely visible, has little visibility, is not immediate, is complex, has no recognition, has no memorization, has no depth, no danger.

Toss some branching path and you increase, complexity (need to track various path), recognition (detect the right path), memorization (which path is the good one).

Toss some trap like moving seesaw and you increase danger and timing.

Toss an AI and you increase depth.

Procedural tend to decrease memorization in favor of recognition.

edit:

I usually use a 5 dimensional heuristics to match gameplay and scenario.

For each area, I set before jumping into mechanics:

The degree of spectacular (fancy things that is shown or that happen)
The target tempo
The feeling of danger (how much harm i will get)
The feeling of difficulty (how much faster i will solve it)
The feeling of complexity (how much effort i need)

Any "feeling dimension" is also set with two variable: real and imply.
Real is the actual value of the pattern we would use.
Imply is what the setting actually suggest to the player.

For exemple a level set in a volcano with lava and fire all around, with fireball that fly just one tiles higher (out o reach thus non dangerous) than the character jump distance, boulder that fall just right after the main character or just right in front while he run at full speed are example of IMPLY Danger without any realities behind, just smoke and mirror. And even with the exact same stat that one winged chimeric angel would still provide a different experience that a lonely slime.

By combining that you can quickly create interesting situations and feelings, interdependently of the nature of system.

[Kinimod]

Couldn't one see difficulty as the probability that the player loses the game? Then the difficulty would be
(the number of all possible sequences of actions that lead to game over)/(the number of all possible sequences of actions that exist)

Of course that's a rather rough estimation as it assumes that all sequences have equal probability. One would probably have to ignore sequences that are so obviously useless that no player would consider them.
However think that it than could be used for example stuff like turn based battles or sokoband puzzles, as they usually require relatively few actions.

EDIT: Sorry, the probability of each path would be the product of the probabilities of its actions. and coming up with a useful probability estemate for each action is probably most difficulty part of this.

[gimymblert]

It's too thing: trouble to progress, probability of losing.

If a player do not progress he quit, which equate to losing.

So the aspect who interfere with difficulty are the thing in my list. (ability to find a solution, to understand what happen and execute the solution to progress/not lose). Measure them and you have difficulty.

It's easy to estimate, how many variable there is (number of enemy, stat, status, etc...) vs margin of maneuver (degree of freedom, acton possible, health remaining, etc...) considering that interface (readability) is already sort out and constant.

[Zaphos]

has there actually ever been any kind of research on how to mathematically define and measure the difficulty of video games?

Yes, at least implicitly in research on dynamic difficulty adjustment (dda). But this is typically "difficulty for the current player" not "difficulty for the average player" (or for some more general type of player).  Here's one academic paper on dda for example:
https://www.aaai.org/Papers/Workshops/2004/WS-04-04/WS04-04-019.pdf
(skimming the paper, the metric in this case seems to be "if the player is taking lots of damage they're probably in a difficult spot")

On a similar note I believe valve and others create visualizations of difficulty-related metrics, like a heat map of a level where the color indicates how many players died there.

[randomnine]

Jasmine: "The game got too hard, so I gave up"? That's what you want people to take away from your games? Arcade games, sure, but I don't think that approach is generally applicable.

But yeah. Playtesting and analytics. Rather than trying to work out some precise measure of difficulty - honestly, don't bother, it's different for every person and it's certainly not one-dimensional - the two things you're concerned about are when people get stuck and when they get bored. Difficulty, poor signposting and poor affordances are big factors in the former but the latter's as likely to be due to excessive repetition, a lack of mechanical depth, or even just not engaging with the fiction.

This is where the research you're interested in is being done. Not on dry, academic theories of difficulty but how to run objective playtesting sessions and interpret them, because that's what's useful.

[Nillo]

Jet-Set-Willy: After 30 years, still nobody has ever completed it without cheating.

Isn't that because the original Jet Set Willy contains gamebreaking bugs?

If you mean that nobody has completed a version of Jet Set Willy with the bugs that made it unwinnable fixed, well... I've heard a rumor that some people completed I Wanna Be The Guy on Impossible mode, so this couldn't be much more difficult than that, right?

[jquave]

Pretty much just need to watch people play in my experience. People who have never played it before, and don't help them!

[ஒழுக்கின்மை (Paul Eres)]

You can't really "objectively" measure the difficulty of a game, because it's relative to the player's skill level.

you can objectively measure things which vary from person to person. for instance, eyesight varies from person to person. but you can still objectively measure eyesight, and give someone a prescription

difficulty could be measured similarly: you can measure how hard something is for a person. there are various units you could use: average time it takes to beat the game, % of the people who managed to finish a game, % of the population which beats a game in under x hours, % of people who managed to finish the first level without dying the first time they play the game, and lots of other ways to measure it. there's no standardized unit right now, but that doesn't mean one could never become widespread

[justinfic]

You can't really "objectively" measure the difficulty of a game, because it's relative to the player's skill level.

you can objectively measure things which vary from person to person. for instance, eyesight varies from person to person. but you can still objectively measure eyesight, and give someone a prescription

The problem is that eyesight doesn't get better with practice.

You also can't really use a specific mechanic to objectively measure difficulty unless every game uses that mechanic.

Game difficulty will always be subjective-- not only do players have different skill levels, they also have different skill levels PER GAME. And while say beating Megaman 9 WOULD make someone better at Megaman 10, they still have to practice Megaman 10 to beat it (just the way those games are designed) and it does absolutely nothing for their skill in say, basketball.

And while something like Super Meat Boy is accepted as incredibly difficult, it is still something that you can practice and eventually learn to master. Meanwhile you can master Solitaire in an afternoon, but you will always have that same rate of failure due to random chance.

[ஒழுக்கின்மை (Paul Eres)]

eyesight actually does get better if you exercise it

[rek]

You can, in fact, objectively measure the difficulty of a game.¹ Check Metrics #1 and #2 in this Gamasutra article about analytics. Players kept dying in the same places, and dying way too often in certain levels, and then gave up more often than not. That's objective proof of difficulty.

This sort of analysis shows you if a level is harder than its neighbours, where it's the most difficult, and gives you some idea of what to tweak to change this. In this case the author hired a company to do this sort of thing, but I'm sure you could program it into the game itself to build a database of information that could be graphed and plotted. The inverse can also be figured out: hardly anyone dies in level X, but level X is near the end of the game, why? It's too easy. Average level completion time is measured in seconds instead of minutes, why? It's too easy.

¹ IMO, it's pointless to say one game is more difficult than another, unless they're virtually the same game. Factor in different genres and why would you even bother comparing them?

[ஒழுக்கின்மை (Paul Eres)]

measuring very simple statistics like that can be done with steam achievements, indirectly. some games have an achievement for doing different things -- beating the game, beating a particular level, finding a particular thing, or whatever. and steam shows the % of all people who own the game who accomplished that achievement. since most games have a 'beat the game' achievement, you can get a fair sense of the game's difficulty (or length, or boringness, among other factors) by looking at what % of the people who bought the game beat the game

as an example, see this forum's own zaratustra's game eversion; its achievements on steam are interesting: http://steamcommunity.com/stats/Eversion/achievements

92% of people beat level 1
86% beat level 2
84.8% beat level 3
64.6% beat level 4
63.7% beat level 5
61.9% beat level 6
54.6% beat the game in the normal way
29.9% got the best ending

just from looking at that, it's obvious that level 4 is the hardest level, with a drop-off of 20.2% of players who beat level 3 but not level 4. this is followed by level 1 (8% drop-off) and the final level (7.3% drop-off). for the first level the effect may have to do with some people who own the game not even bothering to play it, however, so that can probably be discounted

[J-Snake]

has there actually ever been any kind of research on how to mathematically define and measure the difficulty of video games? I guess this could be useful for analysing and procedurally generating games.

I think you can measure it for some aspects. But there is no mathematical way to do it in general, like the cone-response-function for how we see graphics was actually just measured and is simply statistic. So you have simply to refer to statistic. If you represent the statistic fairly well then you can judge the difficulty well by your own:)