next »

[Rob Lach]

Guys guys...

if that was a reference to the closure discussion, i don't think i was unfairly trashing it

Ah, I didn't mean anyone was. I meant I see it coming.

[Blademasterbobo]

the honorable mentions are the best awards

[Oddball]

Picture the scene. It's the winter Olympics. New skater 'Kale in Dinoland' takes to the ice rink. The judge for igfland stands up and says "Bah! I can tell I'm not going to like this routine, because of the amateurish costume she's wearing.". As he walks to the exit he turns back and says "I'm off to the pub. Call me back when that Fez I've heard so much about is ready to skate. She looks hot.".  Kale, now distraught, asks "But shouldn't I be given a fair chance to impress you." The igfland judge replies "Nah! I only volunteer for this so I'll decide what deserves my time.". Door slams shut!

[Manuel Magalhães]

My view of the time used: Like someone mentioned I think it depends on the game itself (a game like Canabalt needs less time than a JRPG) and the judge. That said not playing the game in the first place (which was at happened the most with Kate in Dinoland) is despicable. I simply can't believe that the judges don't have that amount of time to play the games. Plus I think it should be obligatory to give feedback, that way they can explain why they liked/disliked the game. 
But anyway I'll do like phubans: try my luck on other competitions and try to 

[Fallsburg]

I think one of the biggest problems in this thread is that the IGF Defense Force is unwilling to suggest that things might be wrong.  I'm not saying that Kale would have won, but I would hope that the fact that it didn't get installed by a judge assigned to it would ring a warning bell. 
Every argument defending the IGF falls into one of these categories:
1) The results wouldn't change, so the process doesn't matter.

2) Judging all these games is a lot of work. I'd like to see you try it.

3) Things like this will happen.

To which I think the proper responses are.
1) That's a horrible attitude, and I find it hard to believe you live the rest of your life using this principle.  If I can invest my money and I'm looking at 2 investment firms, Firm A and B. Firm A reads tea leaves to make investments. Firm B does detailed analysis on all levels. Up until now Firm A and Firm B have made the exact same investments.  That doesn't mean I'm not putting my money with Firm B.

2) No one said slogging through lots of bad games was easy.  But allen's "If you don't have the time for it, don't do it" stands true.

3) Perhaps. But that doesn't make it right. And it doesn't mean that efforts shouldn't be made to fix it.


Note: I'm not really defending Rotten Cartridge.  I don't think airing dirty laundry on the internet ever leads to anything good. And their declining the phone call offer was bad.  But to turtle up and try to defend what happened just makes the IGF look worse.

[Christian Knudsen]

The only solution I see to this (if you want to keep the current system of judges) is to implement some kind of API that tracks how long a judge has played a game (but still keeping judges anonymous to the entrants). But that would likely be both costly and cumbersome.

The big problem seems to be that the number of games submitted to the IGF has just exploded in recent years to the point where it's no longer possible to secure enough judges to give each game a fair play. I think you should either crowd source this process somehow or have smaller regional qualifiers to the main IGF.

[Paul Eres]

The only solution I see to this (if you want to keep the current system of judges) is to implement some kind of API that tracks how long a judge has played a game (but still keeping judges anonymous to the entrants). But that would likely be both costly and cumbersome.

no, that's not the only solution. the solution is much simpler and much easier: neglect to re-invite for the next year judges who don't play all the games they are assigned (barring technical hurdles), or judges who only play all their game assignments at the last minute / just before the deadline

yet even this simple thing, the igf is against doing

it's like you're saying the only thing that will prevent burglary is putting a videocamera in every room in the world, therefore we should be okay with burglary

also the bit about there not being enough judges is ridiculous. i gather that there are far more people recommended to be judges than there are actual judges. there seem to be almost as many judges as entrants now. a single person playing 18 games in 3 months is not an extreme number at all, i don't get why it's treated as this superhuman feat to do

a judge who spends 5 minutes on each of their 18 games spends less than two hours judging games. i don't think you should volunteer to be a judge if you can't spend more than two hours on it. there are thousands of people who would die for the opportunity to judge the igf who are willing to take the place of someone who isn't willing to spend 2 hours on it

[Christian Knudsen]

But how will you know that a judge hasn't played a game if there isn't some API in place to track this?

it's like you're saying the only thing that will prevent burglary is putting a videocamera in every room in the world, therefore we should be okay with burglary

It boggles my mind that you could get this from my post...

[Paul Eres]

I think one of the biggest problems in this thread is that the IGF Defense Force is unwilling to suggest that things might be wrong.

that was my primary thought when reading brandonnn's comment to the article on the rotting cartridge -- it's like he's defending to the death a judge's right to spend as little time as possible (even none) on a game, and doing everything he can to refuse to admit it's a problem. i don't have the comment in front of me but my impression was that he did not seem apologetic in any way in that comment, it was just a self-justification thing. would it kill him to say 'yeah, we have lazy judges, and i'm sorry you got caught by a few of them' instead of talking about how it's a good thing for your game that judges don't look at your game?

[Paul Eres]

But how will you know that a judge hasn't played a game if there isn't some API in place to track this?

in the past (i'm not sure if this is still the case, i presume not?) judges gave point values for the games they were assigned -- giving each of their games a 1 to 5 score in the different igf categories. the games with the highest score became nominated. this was changed with the jury process. i feel that this system should have been retained, even if it didn't determine the nominee anymore, because it's important to see a cumulative score of what people on average thought of the game in each category

secondly, there's feedback; judges can give feedback to developers about their games. judges that never provide feedback at all to any developer, or judges whose feedback is just generic one-line stuff could easily just be removed for future igf judging panels

also, many judges publicly blog about playing through all the igf games at the last minute, or publicly complain about their assignments being too long and they can't get through them all in an afternoon; those problem judges could be specifically removed

you can also easily track whether a judge *downloads* each of the games they are assigned from the igf file servers. many judges don't even do that

[jotapeh]

But how will you know that a judge hasn't played a game if there isn't some API in place to track this?

I pondered this earlier, but there's really nothing stopping a judge from launching your game and going to watch TV for an hour just to not get 'caught' (if they really care anyway.)

[Paul Eres]

@jotapeh - yeah but if a) not getting caught would require just as much effort as actually giving the games more than 3 minutes of your time, and b) if there were actual consequences for getting caught instead of the attitude of "yo pal, don't worry about putting in effort, you'll be a judge no matter what", then that would go a long way

[Christian Knudsen]

You're right that if a judge hasn't even, well, judged the assigned games, he shouldn't be allowed on as a judge again. But from what I understood from the article, we don't know that the judge that didn't play the assigned game didn't rate it. From what I understood, the point of the article was that the developers were able to track whether or not a game was actually played, not judged.

It would be interesting to hear if the judge that didn't play the game actually rated it (to appear to have played it).

EDIT: This was a reply to Paul, by the way.

[Paul Eres]

@christian

well, you can read one judge of that game's own comment in the comment thread there. from what i remember, he defended only playing it for a few minutes, saying something like 'sorry but the game was clearly bad so i decided it wasn't worth my time'

even if you can't entirely fix the judging problem, what you *can* do is take clear isolated problem cases like that and fix them individually (by not re-inviting that particular judge, and making clear to other judges that that type of behavior isn't going to be tolerated)

but the igf is unwilling to do even that, instead they defend judges like that and claim they are great judges because players would also give up in 3 minutes (even though there's evidence showing that actual players of the game do not give up that quickly on average)

so i'm not saying you have to catch everyone, or make some complex superhuman AI system which makes sure people played their games the right amount of time, i'm just saying the igf should *discourage* this type of behavior from judges rather than defend it and justify it

[Christian Knudsen]

Oh definitely. I just have a sneaking suspicion that it's a much more widespread problem than these few instances of judges being "caught", and while the IGF should definitely handle these instances in a proper manner to set an example for the other judges, I doubt it would fix the underlying problem.

And "tracking API" = "complex superhuman AI system"? Really? The article was written precisely because these developers had access to a tracking API, so there's no need to make it sound harder than it is.