next »

[Christian Knudsen]

The only solution I see to this (if you want to keep the current system of judges) is to implement some kind of API that tracks how long a judge has played a game (but still keeping judges anonymous to the entrants). But that would likely be both costly and cumbersome.

The big problem seems to be that the number of games submitted to the IGF has just exploded in recent years to the point where it's no longer possible to secure enough judges to give each game a fair play. I think you should either crowd source this process somehow or have smaller regional qualifiers to the main IGF.

[ஒழுக்கின்மை (Paul Eres)]

The only solution I see to this (if you want to keep the current system of judges) is to implement some kind of API that tracks how long a judge has played a game (but still keeping judges anonymous to the entrants). But that would likely be both costly and cumbersome.

no, that's not the only solution. the solution is much simpler and much easier: neglect to re-invite for the next year judges who don't play all the games they are assigned (barring technical hurdles), or judges who only play all their game assignments at the last minute / just before the deadline

yet even this simple thing, the igf is against doing

it's like you're saying the only thing that will prevent burglary is putting a videocamera in every room in the world, therefore we should be okay with burglary

also the bit about there not being enough judges is ridiculous. i gather that there are far more people recommended to be judges than there are actual judges. there seem to be almost as many judges as entrants now. a single person playing 18 games in 3 months is not an extreme number at all, i don't get why it's treated as this superhuman feat to do

a judge who spends 5 minutes on each of their 18 games spends less than two hours judging games. i don't think you should volunteer to be a judge if you can't spend more than two hours on it. there are thousands of people who would die for the opportunity to judge the igf who are willing to take the place of someone who isn't willing to spend 2 hours on it

[Christian Knudsen]

But how will you know that a judge hasn't played a game if there isn't some API in place to track this?

it's like you're saying the only thing that will prevent burglary is putting a videocamera in every room in the world, therefore we should be okay with burglary

It boggles my mind that you could get this from my post...

[ஒழுக்கின்மை (Paul Eres)]

I think one of the biggest problems in this thread is that the IGF Defense Force is unwilling to suggest that things might be wrong.

that was my primary thought when reading brandonnn's comment to the article on the rotting cartridge -- it's like he's defending to the death a judge's right to spend as little time as possible (even none) on a game, and doing everything he can to refuse to admit it's a problem. i don't have the comment in front of me but my impression was that he did not seem apologetic in any way in that comment, it was just a self-justification thing. would it kill him to say 'yeah, we have lazy judges, and i'm sorry you got caught by a few of them' instead of talking about how it's a good thing for your game that judges don't look at your game?

[ஒழுக்கின்மை (Paul Eres)]

But how will you know that a judge hasn't played a game if there isn't some API in place to track this?

in the past (i'm not sure if this is still the case, i presume not?) judges gave point values for the games they were assigned -- giving each of their games a 1 to 5 score in the different igf categories. the games with the highest score became nominated. this was changed with the jury process. i feel that this system should have been retained, even if it didn't determine the nominee anymore, because it's important to see a cumulative score of what people on average thought of the game in each category

secondly, there's feedback; judges can give feedback to developers about their games. judges that never provide feedback at all to any developer, or judges whose feedback is just generic one-line stuff could easily just be removed for future igf judging panels

also, many judges publicly blog about playing through all the igf games at the last minute, or publicly complain about their assignments being too long and they can't get through them all in an afternoon; those problem judges could be specifically removed

you can also easily track whether a judge *downloads* each of the games they are assigned from the igf file servers. many judges don't even do that

[jotapeh]

But how will you know that a judge hasn't played a game if there isn't some API in place to track this?

I pondered this earlier, but there's really nothing stopping a judge from launching your game and going to watch TV for an hour just to not get 'caught' (if they really care anyway.)

[ஒழுக்கின்மை (Paul Eres)]

@jotapeh - yeah but if a) not getting caught would require just as much effort as actually giving the games more than 3 minutes of your time, and b) if there were actual consequences for getting caught instead of the attitude of "yo pal, don't worry about putting in effort, you'll be a judge no matter what", then that would go a long way

[Christian Knudsen]

You're right that if a judge hasn't even, well, judged the assigned games, he shouldn't be allowed on as a judge again. But from what I understood from the article, we don't know that the judge that didn't play the assigned game didn't rate it. From what I understood, the point of the article was that the developers were able to track whether or not a game was actually played, not judged.

It would be interesting to hear if the judge that didn't play the game actually rated it (to appear to have played it).

EDIT: This was a reply to Paul, by the way.

[ஒழுக்கின்மை (Paul Eres)]

@christian

well, you can read one judge of that game's own comment in the comment thread there. from what i remember, he defended only playing it for a few minutes, saying something like 'sorry but the game was clearly bad so i decided it wasn't worth my time'

even if you can't entirely fix the judging problem, what you *can* do is take clear isolated problem cases like that and fix them individually (by not re-inviting that particular judge, and making clear to other judges that that type of behavior isn't going to be tolerated)

but the igf is unwilling to do even that, instead they defend judges like that and claim they are great judges because players would also give up in 3 minutes (even though there's evidence showing that actual players of the game do not give up that quickly on average)

so i'm not saying you have to catch everyone, or make some complex superhuman AI system which makes sure people played their games the right amount of time, i'm just saying the igf should *discourage* this type of behavior from judges rather than defend it and justify it

[Christian Knudsen]

Oh definitely. I just have a sneaking suspicion that it's a much more widespread problem than these few instances of judges being "caught", and while the IGF should definitely handle these instances in a proper manner to set an example for the other judges, I doubt it would fix the underlying problem.

And "tracking API" = "complex superhuman AI system"? Really? The article was written precisely because these developers had access to a tracking API, so there's no need to make it sound harder than it is.

[Chris Pavia]

I find it funny how hard people rail on the igf, yet when I see Matt, Brandon and others posting (here and other places) they seem more than willing to hear out and discuss peoples complaints. Hell, in that article he offered to talk to them directly over the phone and the devs didn't take them up on it. Bitching in your blog is easy, but when they get a chance to do something about it, it's suddenly not worth thier time?

Not playing your assigned games is definitely unacceptable, but most here seem to want a shadowy strawman to blame thier problems / lack of recognition on.

[Soulliard]

I always wonder why people get so upset over problems with the IGF. It's just a contest, and while it does have some flaws, it still does a pretty good job of identifying the best indie games out there.

I suspect that the business plan of some indies looks like this:
Step 1: Make Game
Step 2: Win IGF
Step 3: Profit!

Honestly, if you make a good game and you market it well, people will buy it regardless of whether it won the IGF.

[Christian Knudsen]

Sure, indie developers shouldn't bet on their game winning the IGF -- or even have the IGF as part of their business plan. But the IGF is still a great opportunity for indie developers, and I think it's understandable that they'd want it to be as "fair" as possible (a viewpoint I'm sure is shared by the organizers). And when entrants pay $100, the least they can expect is that their game is actually played by the judges.

[ஒழுக்கின்மை (Paul Eres)]

Oh definitely. I just have a sneaking suspicion that it's a much more widespread problem than these few instances of judges being "caught", and while the IGF should definitely handle these instances in a proper manner to set an example for the other judges, I doubt it would fix the underlying problem.

And "tracking API" = "complex superhuman AI system"? Really? The article was written precisely because these developers had access to a tracking API, so there's no need to make it sound harder than it is.

if you read the comments to that blog post, other people mention similar problems yes, so it's very common. one person said that that guy was lucky, that there were cases when *no judges at all* played a game. here's a few quotes:

This happened to me too a few years back when IGF was doing mod submissions. It was networked and found out that there wasn’t a single judge that even logged on to play. I got an apology from Simon and a refund for my submission but that was it. I’m disappointed to hear that the IGF is still continuing this practice while claiming to represent a fair process.

I entered my iOS game “Foozle” into the student competition portion so it was free. My experience was far worse. After pestering the contacts I had at IGF a couple times, about a week before the *extended* student judging deadline I finally got an email asking for promo-codes(mind you, without the extended deadline I wouldn’t have even been asked for promo-codes for the judges prior to the announcement of winners). Of the 10 promo codes they asked for, exactly 1 was actually redeemed. So, in the student portion I got a single play on an iOS game that has over 200 ratings averaging 5 stars in the app store. I didn’t want to be a whiner, but if it’s widespread that’s just not acceptable.

This happened to me way back as well. We submitted a download link to our PC game, which was hosted on our own server. The link was unique to IGF, and it never got a single hit.
Obviously, we brought this up, only to be completely ignored. Only when this started getting a fair bit of publicity via slashdot did they contact us about it with about the same reply you had. We talked to them, and got an apology but not much else – they offered to send us some stuff. We accepted and gave them a shipping address, deciding it wasn’t worth more of our time (we had a game to work on after all).
So that was the end of the story. And I mean the end of the story, since the part where the stuff they promised actually arrives never happened either.

there are many more comments in this vein; so yeah, at least subjectively it seems to be extremely common

and it was an exaggeration of the complexity and intrusiveness of such a thing. i don't see how it'd work unless every entrant were required to install that api in their game. it's a good idea for people to decide to do individually, and i'd like time-tracking to become more common in igf entrants because it's the only way we actually have proof about this issue, but i would not be for *requiring* that entrants all use some time-tracking api, because of privacy issues and because of (as you mentioned) it being cumbersome -- so as i mentioned, it feels like saying 'well, we can't force everyone to put the right trash in the right recycle bin without a camera watching their every move, so let's not do anything about it at all'

[Christian Knudsen]

Agreed. The tracking API isn't an optimal solution. But it was the only solution I could think of that would pretty much ensure that all judges play their assigned games. The IGF making an example of judges that haven't done this (when they're for example caught by developers that implemented their own tracking API -- or the IGF not being aware of tracking capacity in the selected systems) is definitely the easiest and most obvious first step.

And I did offer other suggestions, so one of my suggestions not being easily applicable certainly isn't the same as me saying not to do anything about it at all -- I don't even know why I have to say this.

[ஒழுக்கின்மை (Paul Eres)]

I always wonder why people get so upset over problems with the IGF.

this was mentioned earlier in the thread, but it's largely because of steam and publishing contracts

steam seems to have an informal policy of games that are igf finalists being the only indie games that get on steam. there are counter-examples aplenty, but it's generally the case that igf-finalists get on steam, and non igf-finalists don't

similarly, igf finalists/winners tend to get publishing contracts on consoles, other indie games do not. the publishers of those consoles sometimes specifically go through igf winners to see which games they can publish on their consoles

i do agree that it's not a good idea to make getting an igf nomination part of one's marketing plan obviously, but at the same time let's not underplay how important the igf is by saying it's just a contest. it's similar in importance to indie game developers as sundance is to indie film makers, it makes and breaks careers. even fish has said something about fez never would have been where it is now if it weren't for the igf. when something makes and breaks careers you're going to have a lot of drama around it

[ஒழுக்கின்மை (Paul Eres)]

And I did offer other suggestions, so one of my suggestions not being easily applicable certainly isn't the same as me saying not to do anything about it at all -- I don't even know why I have to say this.

i think i had that impression because you said "only" -- as in the only thing we could do would be the api tracking (which you then said was impractical)

so i was objecting to the word "only", since i think there are other easier things that the igf can do, like to simply stop rationalizing / being defensive and to admit it's a bad thing for indie developers when they pay 95$ and get nobody playing their game or 5 people playing it for an average of 3 minutes, to stop trying to hide the issue by discouraging people to talk about it (which is what that phone call offer was really about), and so on

[Christian Knudsen]

Not to get all semantic on you, but I didn't say it was the only thing we could do, I said it was the only solution. There's no guarantee that the IGF not inviting judges back that didn't play all their games will solve the problem -- mainly because we don't know that these judges aren't just judging the games anyhow, so the IGF won't know that the judges didn't play them.

[hanako]

wow, so much nonsense.

here's MY bias - Only entered once, lost miserably, probably never going to enter again because there's not a prayer that my kind of game will ever get noticed in the IGF. That said, I'm still standing up for the judging because a lot of what's said here is just cuckoo.

a single person playing 18 games in 3 months is not an extreme number at all, i don't get why it's treated as this superhuman feat to do

The judges do NOT get three months. They get less than one. Considering how many devs don't bother getting their final builds in on time and the number of technical problems getting some games available, quite often they have one week. Admittedly a lot of judges do seem to wait until the last minute to get started and that's sort of annoying.

But still - do your fucking job. What's the point in ponying up so much money if people aren't gonna do their job.

If you look at it as a financial transaction, it's pretty unfair to yell at the judges to do their fucking 'jobs' when they're not being paid at all. It sucks for an entrant to give their money and get nothing more than the minor publicity boost of having their game listed on the IGF site, but the judges don't get even that much.

People volunteer because they love games and they want to help discover and recognise good ones. They're not hired playtesters. If they look at your game for <5 minutes and think it's not winning material and drop it, they HAVE done the job they signed on for.

(That said, have they still not released the feedback to you guys? Not much point in us writing it if you're not going to receive it in a reasonable amount of time. )

and if somehow you have all these life's little things that prevent you from playing a few video games why don't you just tell someone and ask for someone to replace you?? "oh hey I won't be able to test this GAME I'M ASSIGNED can you get someone to do it for me? THANKS I AM BEING RESPONISBLE")

Many judges DID do this sort of thing.

---

Also, the interactive fiction competition did NOT require you to play for two hours before judging - it required you to play for NO MORE THAN two hours. The idea was that if someone's game was long, they shouldn't expect people to play the whole thing.

[Manuel Magalhães]

since i think there are other easier things that the igf can do, like to simply stop rationalizing / being defensive and to admit it's a bad thing for indie developers when they pay 95$ and get nobody playing their game or 5 people playing it for an average of 3 minutes, to stop trying to hide the issue by discouraging people to talk about it (which is what that phone call offer was really about), and so on

This. This type of defense sickens me:
https://twitter.com/#!/psychicteeth/status/172702000444751873
The judging system isn't the problem, the problem are the judges. I'm not saying that all of them are like that, but like Paul said we should make something about that.

But the judges say that everything is ok and we just don't like losing, so who am I to say otherwise? If your game wasn't judged and you wasted $100 on it, Alex May tells you to that you should have spent it on food.