next »

[Matthew]

It just seemed a bit as if Matthew completely ignored the larger issue of judges not playing their assigned game (and the article that started this new round of discussion) and instead responded to an older topic that wasn't currently the center of discussion.

I just don't feel like there's much to say about it.

Games are "over-assigned" to judges to make sure there's enough scores.  Sometimes there aren't because judges flake out or because the game has compatibility issues.  If a game has a low number of scores it's brought to someone's attention to see what the issue is (I'm not sure if it's still in use in the jury system, because I haven't touched the code this year, but there was a mechanism to see under-scored games so active judges could help out where needed).

There's an admin-only report of all judges by activity level (comments and scores).  The tail end of the judging pool is chopping off every year to make room for new blood and to cull inactive judges.

Yes, things can be improved.  They can always be improved.  Organizationally, judges that are assigned a game they never actually score aren't an issue, because the data reflects their behavior.  The actual potential problem is a judge who does score a game they didn't actually play--or didn't play enough to form an opinion--because that's really hard to suss out.

[Paul Eres]

Couldn't judges be required to provide a public motivation for rating a game as they did? It would make them put more though and effort into the judging, and it would be easy to pick out bad judges who should not be allowed back.

that's actually a pretty interesting idea; being able to articulate the reasoning process behind your nominations / scores is something that judges should be expected to do. it wouldn't even have to be public, it could just be something the other judges or the jury could read

e.g. 'i nominated game x for audio because it has an extremely nice soundtrack; i found myself listening to its soundtrack for days, and hummed one in the shower. its sound effects are competent but not exceptional, it's the music tracks, particularly the boss theme and the title theme, that impressed me'

this would also help with the problem, which mattheww mentioned above, of scoring games someone didn't even play

[Christian Knudsen]

It just seemed a bit as if Matthew completely ignored the larger issue of judges not playing their assigned game (and the article that started this new round of discussion) and instead responded to an older topic that wasn't currently the center of discussion.

I just don't feel like there's much to say about it.

Games are "over-assigned" to judges to make sure there's enough scores.  Sometimes there aren't because judges flake out or because the game has compatibility issues.  If a game has a low number of scores it's brought to someone's attention to see what the issue is (I'm not sure if it's still in use in the jury system, because I haven't touched the code this year, but there was a mechanism to see under-scored games so active judges could help out where needed).

There's an admin-only report of all judges by activity level (comments and scores).  The tail end of the judging pool is chopping off every year to make room for new blood and to cull inactive judges.

Yes, things can be improved.  They can always be improved.  Organizationally, judges that are assigned a game they never actually score aren't an issue, because the data reflects their behavior.  The actual potential problem is a judge who does score a game they didn't actually play--or didn't play enough to form an opinion--because that's really hard to suss out.

That was actually quite informative. Thanks.

[Paul Eres]

as an aside, since judges are evaluating games in multiple categories, i kind of question that they can competently do that in 4 minutes. they have to be evaluating not only how *fun* the game is to play, but also its visuals, its audio, its novelty (nuovo award), and so on

can someone really confidently say that any game, even superman 64, doesn't deserve a nomination anywhere (not visuals, not audio, not technical, not overall, not nuovo) based on a few minutes of play?

i don't know about others, but i can only really focus on one thing at a time. i have to be specifically paying attention to the audio (sound effects and music) to judge it, and then pay attention to the visuals to judge those, and then think about what the game does that's new to judge its innovation, and then think about how it's programmed to judge its technical excellence, and so on -- those categories each require separate evaluation, i don't know if someone can really do all that, simultaneously, so quickly

4 minutes sometimes isn't even enough time to listen through a single song. so they are judging the entirety of a game's audio based on whatever song they happen to hear first, and they can't even pay attention to that song because they're also playing the game to see how fun it is and evaluating the game's graphics and programming as well

[Matthew]

as an aside, since judges are evaluating games in multiple categories, i kind of question that they can competently do that in 4 minutes. they have to be evaluating not only how *fun* the game is to play, but also its visuals, its audio, its novelty (nuovo award), and so on

You do realize the catalyst for the current discussion is a game that so completely invokes Gameboy aesthetics that it presents itself as a fictional port of a 1992 title, right?

[Glaiel-Gamer]

as an aside, since judges are evaluating games in multiple categories, i kind of question that they can competently do that in 4 minutes. they have to be evaluating not only how *fun* the game is to play, but also its visuals, its audio, its novelty (nuovo award), and so on

You do realize the catalyst for the current discussion is a game that so completely invokes Gameboy aesthetics that it presents itself as a fictional port of a 1992 title, right?

the judge comments for this game show 3 judges putting an effort into trying that game (one of which sounds like the 50 minute playtime judge from the article). Hes lucky a judge played it for 50 minutes, if it really did get good later on, that judge would have made a comment about it alerting other judges to give it a chance, but thats not what happened.

[Fallsburg]

This is not about Kale in Dinoland, and to dismiss all concerns because Kale wouldn't get nominated in the best of judging circumstance is to miss the point.

The point is that it is wrong that judges dismiss a game in 3 minutes. It is wrong that judges don't play games assigned to them.  Kale is just the one that happened to draw attention to the fact that this is going on, and as they stated themselves, they didn't expect to win.  They just wanted to be given a fair shake.

Andy wanted us to draw attention to games that should have won but didn't.  I mean, I can go through the list of the 500+ games, play them all, and let you know, but hey that's what the IGF is supposed to be fucking doing and apparently isn't.

[AndySchatz]

No, I don't want you to go through and play them all.  But it's a good sign that this argument is about a bunch of strawmen since people can't seem to argue what's wrong with the actual, specific results.  Surely if the IGF is so broken you can come up with a few specifics that we can argue about on the merits of the games.

Paul, thank you for bringing up Bit.Trip.  But disagreeing with Bit.Trip winning is based on personal preference, not how much time was given to the games.  I wasn't on the art jury, so I can't say what went into the selection process.  I do think there's a very valid argument over whether the new IGF jury system creates a bias for games that are well known (I believe it does), though I can't say whether that helped this particular game -- I haven't played it.

That said, I get the feeling that Paul is basing his opinion on THE SCREENSHOTS OF THE GAME.  Surely this is more than a bit hypocritical...  Correct me if I'm wrong.

[AndySchatz]

And by the way, just to reiterate: EVERYONE agrees (including and especially the people that are close to the IGF) that judges that don't put in the required time to evaluate games suck.  It's a problem that the IGF organizers constantly and actively battle.

[Matthew]

This is not about Kale in Dinoland, and to dismiss all concerns because Kale wouldn't get nominated in the best of judging circumstance is to miss the point.

The point is that it is wrong that judges dismiss a game in 3 minutes. It is wrong that judges don't play games assigned to them.  Kale is just the one that happened to draw attention to the fact that this is going on, and as they stated themselves, they didn't expect to win.  They just wanted to be given a fair shake.

Andy wanted us to draw attention to games that should have won but didn't.  I mean, I can go through the list of the 500+ games, play them all, and let you know, but hey that's what the IGF is supposed to be fucking doing and apparently isn't.

I think it's important to draw out specifics here.  Main competition judges are asked to nominate games for jury scrutiny; there is a single checkbox for each prize category (and a checkbox for "I played this and don't want to check anything").

I think there are plenty of games where I can read the description, watch their video, and play the game for 5 minutes and feel entirely comfortable checking a box that says "I think there might be something special here".  This is especially true for games that present a down-the-line invocation of a genre, like Kale, which is why I brought it up specifically.

I don't think this true for all games, certainly, and it's an easy statement that more time should be spent on judging.  This is true!  I'd rather see more accurate judging than more numbers, though.  Draconian "YOU MUST JUDGE YOUR LIST" measures will just get judges phoning in scores.  I would much, much rather judges ignore their work if they get busy, because it's easy to spot games with a low number of scores and make sure they're looked at.

[Glaiel-Gamer]

I just brought up a specific example because everyone else seems to be bringing up the hypothetical "amazing overlooked game with a less than stellar intro" scenario, without ever providing an example of an actual game entered into the IGF that fits this description.

[allen]

http://www.unwinnable.com/2012/02/23/the-igf-is-just-fine-youre-the-problem/

this is the kind of judges IGF have?

a concerned independent developer posts about his experience with judges not playing his game. this angry prick of a judge then goes off an some rant about technical issues and then ends the post with a nice "fuck you".

~~~~~~indie games~~~~~~

[peanutbuttershoes]

oh you guys. I mean really. You guys aren't very independent.


I BOUGHT INTO A SYSTEM THAT I THOUGHT WOULD PROTECT ME AND PROVIDE WEALTH FOR MY FAMILY AND YOU F'D ME OVER...

I BOUGHT INTO A SYSTEM

I BOUGHT

INTO

A SYSTEM



y'all should have seen it coming.


 signed,



(i'm kind of kidding)

[Christian Knudsen]

Wow. With friends like that, the IGF truly doesn't need enemies.

[bento_smile]

Oh god, that's just shocking :/