next »

[Paul Eres]

yeah, i expect that too, and i also heard of judges saying they chose monaco over super meat boy for that reason. that was just said in reaction to the idea (expressed a few pages ago) that we shouldn't limit games by budget.

(i still find it weird that limbo won anything at all, with its 2-3 million dollar budget and team of 12+. that wasn't a case like minecraft where the game was made with little money and then got money, it started with a huge advantage, already had a publishing deal, and would have succeeded whether or not it won igf prizes. that said, it's a really great game, it's just not really a game that most indies here could have possibly afforded to make.)

[brandonnn]

btw, is there anything this year in the new system that says a judge has to play the game they're assigned to play? i ask because i received an email from one of the judges (one of the guys at molleindustria) who said he couldn't open the zip: i re-uploaded the game and asked if the new zip worked, and he never replied. i'd hate to think he just didn't play the game because he couldn't open the zip or something.

The general rule of thumb -- until somebody (read: me) can design way more feedback into the system -- is that if you're not hearing from anyone, everything is going fine. Your re-uploads did work!

[Paul Eres]

thanks for the info, good to hear. any idea on when the feedback (if there is any) will be sent out? i know it wasn't required this year, but i'm still hoping for some anyway.

[adev]

if this is a solution then please, do go on. Any examples? How will the public see the concept of having non-game related people judging a game? How different are the jury members of this year compared to the guidelines you'd like to see? Can you name critics you'd feel would be a good addition? How about academics? I'd just like to know more about your point of view so we can understand what you feel and think. You already told us how you came to this answer; now, explain to us how the IGF could actualy do this in reality.

Maybe I should start working for the IGF too? No, I don't know anything about who's who in video game academia. I don't follow any mainstream game media outlets either. Nor am I the least familiar with the inner workings of IGF. I don't understand why you would expect of me to be able to explain how the IGF could successfully adopt it. Is this a condition for criticism to be relevant? When people complain, you ask them to suggest a solution, but when they suggest a solution, you ask them to provide a detailed blueprint? Why don't we discuss whether the idea has any merit first?

[adev]

For those suggesting that the jury be made up of (or partly made up of) people who do not develop games or study game design in some form seem to fail to realise how much worse and more biased the judging would become.

One example being "technical excellence", and how people who are not into the development process of a game could never judge that category correctly. There's a good example of that in the comments on Indigames Blog about Minecraft where they say it didn't deserve the nomination because "the graphics aren't complex" or "the game still has bugs".

Technical excellence would probably not be the best award to bring in judges outside of game development for (then again I never said I thought it should be done to all awards but specifically mentioned muisc, graphics and nuovo), but IF you were to do it for this award, it seems you'd choose people from non-game programming, who might have some competence in the field.

[adev]

I think the organizers are probably understandably hesitant to engage on the tig forums because of the overall animosity, lack of humility, and general internetiness of people's attitudes towards to igf on tig.  I'm gonna give you the benefit of the doubt and respond in good faith, though.  I'm not an organizer myself, but I am on two juries.  I have been a judge in the past.  I've also entered four times, been a finalist twice, and won once.

I don't agree with what you're doing here. If you really do think I'm a troll, please feel free to not respond to my criticism. If you do chose to reply, I think it's very rude to start out by saying that I might not be worthy of a reply but through your kindness, you're giving me the "benefit of a doubt" despite my "lack of humility" or the "general internetiness of people's attitudes" or whatever it is you're subtly accusing me of. Just for the record, I'm doing my best to be reasonably humble while at the same time not pretending to "sort of agree" with people just to avoid conflict (debate is conflict!).

(that said, I AM happy that you replied sharing your perspective and participating in the debate)

While I don't disagree with the idea that more diversity would be good, what I do disagree with is the idea that finding entire juries of qualified, non-dev judges is feasible at all.

It's all well and good to say what the ideal jury should be, but assuming you are a rational, humble, smart guy or girl, and were you closer to the actual organization of the festival, you would agree that the jury selection is pretty close to as good as it gets.

I really think you're barking up the wrong tree.

So I'm not close to the festival at all and maybe it is really really hard to improve the juries in the way I suggested. But what are the actual counter-arguments? All I hear is realpolitik and the validity of it is impossible to assess from the outside (at least in the way you phrased it - you basically say that if I were closer to the festival, I'd get it). I think it's valuable to discuss what the ideal solution would be, even if it turns out to be a solution that's really hard (even nigh impossible) to implement.

[adev]

To address adev here, mainly, I will say that "circle-jerk" syndrome was very much not a part of the juries this year, and was part of the reason we wanted to have a round of long-form discussion to dig further into the nominations. Everyone that DID feel super close to an entry/developer disclosed it up front, and I got more than a handful of private emails asking if they should even be allowed to express an opinion on a game because they lived down the street from/had dinner with someone last night.

More importantly, though, my selections for the juries were based on an implicit trust that those people COULD be objective about a creation from someone that they had on their gmail chat "buddy list", particularly when final jury nominations were based on a private ballot (in other words, people were free to NOT nominate a game from a dear friend without having to stand up and say "I like this person, but this game stinks").

On getting a wider breadth of jurists, I can only say that I tried to strike as fine a balance as I could in finding people that: had time or were willing to make time to play a ton of games, had time or were willing to make time to participate in a discussion about their strengths and weaknesses, and were immersed enough in the world to appreciate those strengths and weaknesses, especially in more specialized fields like Technical Excellence.

I would LOVE to have more mainstream developers participate in this process. I can list more studios than I have fingers and toes for that have smart, thoughtful and talented people who I think could contribute a huge amount to the discussion. Unfortunately, most of them were or are in active dev 10-12 hours a day, and/or didn't respond or politely bowed out. Again, if you know someone, [email protected]! There are more than several handfuls of academics and critics that were part of the judging process, but I specifically/intentionally limited juries to active indie developers to create a "jury of peers" (the Nuovo Jury had more than a few, however), following on from (paraphrased) complaints in prior IGFs that "some nonsense 'journalist' at X website doesn't understand what makes my design/engine/sound unique".

I can totally understand how any large organization that has a semi-opaque process can seem like a shadowy cabal. I did what I could this year to make open the participants in the jury dialogue and pulled actual snippets of the dialogue out of the process to give a flavor of what it was like, but also gave those jurists enough discretion to be open and honest in that dialogue.

There is not a single bone in my body that believes that the jury process should wholly be made up of people not in active development, and I guess you're just going to have to trust me that there is no back-slapping/glad-handing/cigar-smoking mafioso trying to actively suppress people from outside the "family" while stuffing the ballot for their drinking buddies. Everyone-- EVERYone-- EVERYONE! involved in this, from top to bottom, is looking for that diamond in the rough to be surprised and delighted by, especially from someone none of us have heard of before, from some remote pocket of the world. That's why we do this.

Thank you for responding, Brandon.

Let me start off by saying that I don't think the juries consist of back-slapping/glad-handing/cigar-smoking Mafiosos suppressing outsiders, not at all. It's not even that I think that the judges are necessarily responsible for explicitly "circle-jerking". I'm sure you're right when you say that everyone (or at least mostly everyone) involved in the IGF really wants to find that diamond in the rough, etc. The problem is, though, if there's a dearth of variety in the jury, there's going to be a dearth of variety in what people think makes a rough diamond. If you bring together a bunch of like-minded people with similar backgrounds stemming mostly from the same tightly-knit community, can we really expect them to make the best and most varied selection of nominees? If they by chance managed to do that this year, I'd argue it's in spite of the system, rather than because of it.

The case I tried to make isn't that the individual developer is necessarily a problem, or that the individual developer has some sort of an agenda. I mean, OK, I do think that it is a problem that for examples Derek Yu and AdamAtomic worked on entries that got nominated while also being involved in the juries, it's just that that problem is so slight when compared to what I think is the bigger, systemic problem with this year's process. The bias of each developer is fine on it's own, but when you take 20 young, mostly male, Western indie developers and put them together, you've created a system which is inherent to produce a certain outcome, you've compounded that one bias. I mean, honestly, bias - may it be towards "somebody you have on a Google buddy list", a person you've worked with or your own spouse - is not something you can just switch on or off at the turn of a button. The problem is only exacerbated by the fact that what the IGF has ended up with is developers taking turns in handing out and receiving the awards (again, not accusing any single judge or nominee, I'm just talking about what I think are the failures of the system).

If not a single bone in your body believes it'd be the right choice to bar active indie devs from jury duty, I can only hope that you will at least consider decreasing the presence of them next year. I also wish you the best of luck in finding more mainstream developers, etc!

[AndySchatz]

So I'm not close to the festival at all and maybe it is really really hard to improve the juries in the way I suggested. But what are the actual counter-arguments? All I hear is realpolitik and the validity of it is impossible to assess from the outside (at least in the way you phrased it - you basically say that if I were closer to the festival, I'd get it). I think it's valuable to discuss what the ideal solution would be, even if it turns out to be a solution that's really hard (even nigh impossible) to implement.

Sorry if I offended, I was hesitant at first to dip my toe into what I feared would devolve into a flame war rather than a reasonable discussion.  It's clearly not that.

At any rate, the ideal?  Well I suppose the juries would be composed of well-informed people from a diverse set of professions, backgrounds, and personalities.  The juries should inspire trust in the people that care about these awards - that they actually understand what is important about the games that they are judging.  And they should all be extremely committed to doing the job well.

I agree that from an ethnic and gender perspective, the juries are not at all diverse.  My biggest problem with the new system this year is that it ended up picking predictable games... the system did little to expose diamonds in the rough.  But I also think this year is much better than previous years, mainly because the judges, for the most part, really cared about doing their jobs well.

[Melly]

adev, I'm sure you can write more concise posts (no need to quote full posts either). This is a forum, not a novel.

[adev]

adev, I'm sure you can write more concise posts (no need to quote full posts either). This is a forum, not a novel.

This is such a useless post I have to question how you can be a mod, if you think this is a good contribution to this debate (it's as if you want to derail the discussion). Even if I overlook how aggressively anti-intellectual your response is, you still miss the fact that I responded to Brandon's 7 paragraph reply with 3 paragraphs, and Andy Schatz responded to my 3 paragraphs with another 3 paragraphs.

Edit: but feel free to remove part of Brandon's (almost) full quote from my post if you must, I included it on purpose for the benefit of people who want to follow the discussion, not for my own gain.

[Melly]

Whoa, somebody is upset. I made a light-hearted comment, no need to get all riled-up about it.

Edit: I honestly meant nothing bad with that. I was just making a small joke and a giving a little friendly advice in the process. I'm sorry if I hit some sore spot, but you can't take stuff like that so seriously.

[eva_]

melly shut up

[PsySal]

i agree glum buster is one of the best platformers ever made, and probably my favorite indie game of all time, but i don't really see how it's innovative at all; there's nothing it did that other platformers didn't do before it, it just does it better. i feel like i'm in some bizarro world where 'good' means 'innovative'. 

Hmn. I think innovation can occur on different levels, though.

On the one hand, Glum Buster doesn't break any much ground in terms of game mechanics. In the same way it's not got a lot going on in terms of new visuals (like for example Wind Waker)

Where I feel it is genuinely new is in it's ability to communicate ideas through puzzle set-up, or at least, the nature of what some of those ideas are that are communicated. Like for instance the tree level, or the bird level. The actions you can take are intimately tied into the themes and the way the space itself is structured. Also the structuring/ordering of the levels themselves is highly original.

I know what you are saying though, for instance I would say that An Untitled Story is not as innovative but more just Darn Good. However it's definitely not like any other game I've played, either.

Tchus!

[PsySal]

This is something I've been saying for years, which is contrary to the masses of complaints from a few loud entrants: it's a competition, not a forum for feedback.  You can get that a lot cheaper than 95 dollars or whatever the entry fee is.  And the amount of *volunteer* work that the judges put in is massive.  It's better that the judges spend time playing the games rather than sending feedback.  The feedback is also unfortunately just an opportunity for people that weren't selected to cry "its not fair!"

Hmn. I do appreciate the amount of effort spent evaluating games by the judges, and they shouldn't have to worry overmuch about justifying their choices because the responsibility for fairness rests with them. That's what it means to be a judge and you have to trust that as an entrant!

But: feedback is genuinely helpful to a developer, and the "smaller" or less experienced, the more helpful it can be to have an experienced developer offer critique, even a few lines. Multiply this by a few judges and it is actually quite valuable.

For a judge to even record their impressions would be hugely valuable, particularly because they are in the midst of reviewing so many other excellent games.

Finally, it's not strictly a competition-- it's a festival. Isn't it more in the spirit of a festival to provide a forum for improvement for the participants?

[AndySchatz]

There is certainly value in the feedback, but it's just not the point of the competition.  Judges are still encouraged to provide it, it's just not stressed as an expectation of the entrant.  Plenty of people will still get lots of great feedback, I hope.  It was just a mistake to ever make the feedback be some sort of consolation prize.