next »

[ஒழுக்கின்மை (Paul Eres)]

i believe one becomes a judge by being selected by the igf to be a judge. judges are paid in a sense, i believe they get free gdc passes, worth many hundreds of dollars.

i believe they had 150 judges. i imagine that's more than enough for 400 games. perhaps some see judging the igf as a free way to the gdc, involving little work?

also, guert, are you seriously proposing vvvvvv wasn't good enough for the igf? that's a strange proposition if so. how do you think it could have been made better, out of curiosity?

[Alex May]

i think this is true. however, it seems a lot of the judges were pretty awful when it came to rating games, i've heard some waited until the last day or so before the deadline to play any of them (and then complained when they were too long to play in one day!). if they had 6 months to judge games instead of 6 weeks, they might just wait until the last day before the deadline again. i think higher quality judges is as important as more judges, or more time to judge.

I would fix this by holding the judging in rounds. Give a judge ten games on which to feed back, in two weeks. Then in the following fortnight, give them ten more. And so on until they'd done forty. That's two months, so the new deadline for submitting games to the IGF would be (allowing one month for admin as it appears there is now) October I think. I am guessing this would come with increased administration costs for the festival and is one of the reasons the judging process is short now.

Also, I am currently unaware/ignorant of any rewards for IGF judges in terms of GDC passes.

EDIT: It appears we are offered a discount (unspecified) on GDC tickets and seats in the front row of main audience seats just behind the nominees.

[ஒழுக்கின்மை (Paul Eres)]

that rounds idea sounds good, i'd support that. this is all theoretical though, as the people in charge of the igf said they are perfectly pleased with how judging works and don't intend to change it.

[Glaiel-Gamer]

I think they did a pretty good job judging this year, after hearing all the complaints from last year (it was a pretty lackluster selection IMO. I was walking around the booth and there were only 2 or 3 games that actually looked interesting. This year there's a ton of them, and I'm super curious about Limbo and others). Also, not too many misfit selections IMO (except in the technical category which is ridiculous to try and judge anyway), and no "joke" games like Dont Shit Your Pants or YHTBTR got in either (some might call it a travesty, but IMO as hilarious as DSYP was, having it take a spot in the igf would have been a big middle finger to the people that tried.

About VVVVVV (i havent played it btw): There will always be games that a lot of people think deserve it that don't get into the festival. Look on the comments for the announcement on various sites, there's like 10 games that keep getting mentioned, "What? No ______? WTF" This is a problem when you have 300 games and 30 slots and a lot of games that take up more than one slot (sry bout that), and they all deserve the spots they got nominated in. That doesn't mean other's don't deserve it, it just means there's a ton of quality games out there, and not enough space to show them all. There's a number of ways to remedy that problem, but I don't see how revamping the judging would do anything. To see how difficult it is, try to make a list of nominations (5 for each category) that doesn't leave someone out who deserves it.

I know what it's like to be on the short end of the stick, was there last year. Don't let it get to you, and just take it as motivation to work even harder this year for a shot in the 2011 IGF. Preparing for IGF was a HUGE motivation for me, it's almost a shame that there aren't any more big festivals for me to submit too, cause now I have to impose my own deadlines.

[Crackerblocks]

Pretty surprised that Enviro-Bear made it. Me and 4 art games in the category 

I can't find find any explanation on how the judging exactly works, but it seems they erred on the side of being democratic and accountable, and didn't worry so much about being statistically strong.

[brog]

I think they did a pretty good job judging this year

Heh, well you would, right?  (congratulations on all the nominations btw)

My feeling is that with the number of entries they're getting, they could do with expanding the number of finalists.  I suspect that several of the games mentioned were very near misses, and that it was quite crowded at the top, making it semi-random who got in.  But I'm happy, because I've received really positive feedback from one of the judges.

[Guert]

Paul:

(of course I'm aware that some games I would have loved to see picked as finalist but that's an opinion and opinion are quite personal. but as Fish mentioned somewhere, there are some close calls and perhpas 30 or 50 other games barely made it, as non-judges we don't have all the scores that were given to all the games.)

Maybe I didn't phrase it correctly (actualy I think I forgot a word or two in the sentence  ) but VVVVV was on my mind when I wrote this. I'm thinking that it was a close call and unfortunately didn't make it to the finals. 

[Guert]

Brog, how and when did you get feedback? I,m wondering if I'm gonna get some for my entry but have no idea how the judges proceed.

[aeiowu]

i believe one becomes a judge by being selected by the igf to be a judge. judges are paid in a sense, i believe they get free gdc passes, worth many hundreds of dollars.

i'm pretty sure judges don't get free gdc passes. though i i'd like to know for sure...

[Matthew]

Judges do not get free GDC passes.  If I recall correctly there's a 25% discount code.

There are actually two judging processes, one for Nuovo and one for everything else.  Nuovo is more towards the democratic side of things, where the Nuovo jury deliberates on a private mailing list before making their selections.  The main competition is very statistically sound, though (150+ judges and enough scores per game to be meaningful).  This isn't to say that there wasn't discussion in the main competition--one game had 40 judge comments--but it's much more compartmentalized.

More information on the Nuovo process is here:  http://www.igf.com/2010/01/2010_igf_nuovo_jury_releases_f.html

I'm going to paste my comment from the TIGSource front page, because I think it's quite relevant:

There are 20 IGF finalists this year. There were more than 20 great games entered into IGF 2010.

If we have X finalists slots, and Y great games, we’re always going to be at X<Y, and that’s always going to trigger this conversation. Just because Game A, B, or C isn’t a finalist isn’t a flawless indictment that the IGF judging process is broken…

By the way, nor is it any indication that Game A, B, or C wasn’t ranked #6 or #7 in any particular category. Not being a finalist isn’t the same thing as being last place.

I feel like the IGF judging this year was the best it's ever been.  We're always open to suggestions on how to improve it, but these suggestions need to incorporate resource constraints in order to be meaningful.  Saying "judges should spend twice as long on each game", or "the judging period should be twice as long" is not very useful.  By that measure entrants should simply spend twice as long developing their game if they want to be a finalist.  It's just not actionable advice

[Farbs]

My feeling is that with the number of entries they're getting, they could do with expanding the number of finalists.

I'm not sure I agree. What we're all competing for is an exclusive mark of recognition that gets a lot of media attention, but if that mark were shared by more games the value of that mark would be diluted. What's more, that mark would garner less attention because it'd give journalists a more difficult and less interesting story. If all 300ish games made finalist it'd be worthless. It makes sense to me to scale the number of finalists based on player interest, not submission count.

---

I found my server logs from the Captain Forever IGF2010 judging account kinda interesting. I don't have any point to make with them, but I've shared them below in case anyone would like to have a look.

First access, only four days after submission. Have the games been distributed to judges already? This person accessed two pages, one only 5s after the other. Unfortunately I can't tell how long they spent on the second page or what that page might be.
2009-11-05 14:32:39 | IGF2010                   | AL_SUCC  | FAKEPASSWORD
2009-11-05 14:32:50 | igf2010                   | TSK_SUCC | SESSION BHJWFJBZTFHL1T9
2009-11-05 14:32:55 | igf2010                   | TSK_SUCC | SESSION BHJWFJBZTFHL1T9

Probably a judge. Spent < 2m on one page and an unknown time on the next.
2009-11-18 14:12:08 | IGF2010                   | AL_SUCC  | FAKEPASSWORD
2009-11-18 14:12:24 | igf2010                   | TSK_SUCC | SESSION VEO6MHBWOBSOOTW
2009-11-18 14:14:04 | igf2010                   | TSK_SUCC | SESSION VEO6MHBWOBSOOTW

This user hits logout button before viewing any pages or playing the game. Weird. Might have been me testing the login system.
2009-11-19 23:42:09 | IGF2010                   | AL_SUCC  | FAKEPASSWORD
2009-11-19 23:42:31 | igf2010                   | LOGOUT   | XRCETEMRH70O5SR
2009-11-19 23:42:31 | igf2010                   | TSK_FAIL | COOKIE XRCETEMRH70O5SR

One page visited after login, unknown time spent on it.
2009-11-21 13:05:57 | IGF2010                   | AL_SUCC  | FAKEPASSWORD
2009-11-21 13:06:11 | igf2010                   | TSK_SUCC | SESSION 0FVUP0VGTVJTNY8

By this point I'd added more logging so I could see what pages people were visiting. The initial visitor from four days after submission returns, plays CF for ~30s, rummages around in the screenshots page, plays CS for ~5m, then checks out more screenshots and the site's README file.
2009-11-29 23:26:54 | igf2010                   | TSK_SUCC | COOKIE BHJWFJBZTFHL1T9
2009-11-29 23:26:55 | igf2010                   | PAGERQST | CAPTFOREVER.BAT
2009-11-29 23:53:26 | igf2010                   | TSK_SUCC | SESSION BHJWFJBZTFHL1T9
2009-11-29 23:53:26 | igf2010                   | PAGERQST | MEDIA.IFF
2009-11-29 23:53:43 | igf2010                   | TSK_SUCC | SESSION BHJWFJBZTFHL1T9
2009-11-29 23:53:43 | igf2010                   | PAGERQST | MEDIA.IFF
2009-11-29 23:53:50 | igf2010                   | TSK_SUCC | SESSION BHJWFJBZTFHL1T9
2009-11-29 23:53:50 | igf2010                   | PAGERQST | MEDIA.IFF
2009-11-29 23:53:54 | igf2010                   | TSK_SUCC | SESSION BHJWFJBZTFHL1T9
2009-11-29 23:53:54 | igf2010                   | PAGERQST | MEDIA.IFF
2009-11-29 23:53:57 | igf2010                   | TSK_SUCC | SESSION BHJWFJBZTFHL1T9
2009-11-29 23:53:57 | igf2010                   | PAGERQST | CAPTFOREVER.BAT
2009-11-29 23:53:57 | igf2010                   | TSK_SUCC | SESSION BHJWFJBZTFHL1T9
2009-11-29 23:53:57 | igf2010                   | PAGERQST | CAPTSCCSSOR.BAT
2009-11-29 23:58:34 | igf2010                   | TSK_SUCC | SESSION BHJWFJBZTFHL1T9
2009-11-29 23:58:34 | igf2010                   | PAGERQST | MEDIA.IFF
2009-11-29 23:58:38 | igf2010                   | TSK_SUCC | SESSION BHJWFJBZTFHL1T9
2009-11-29 23:58:38 | igf2010                   | PAGERQST | README.TXT

Nearly a month later, someone drilled straight through to CS.
2009-12-22 07:24:16 | IGF2010                   | AL_SUCC  | FAKEPASSWORD
2009-12-22 07:24:16 | igf2010                   | PAGERQST | LOGIN.BAT
2009-12-22 07:24:24 | igf2010                   | TSK_SUCC | SESSION 2ADHMXRXS0AU0M72
2009-12-22 07:24:24 | igf2010                   | PAGERQST | CAPTSCCSSOR.BAT

[brog]

Guert: Just by email.  From the site: "The Festival is particularly keen to give constructive, written feedback to Main Competition entrants -- even if they did not place as a finalist. As a result, over 1500 written, anonymized judge comments will be passed along to entrants in the next few days, an important part of deriving value and takeaway from entering the IGF." - I guess you'll get something soon enough.

Farbs: Maybe you're right.  Maybe if they were to release a list of "honourable mentions" instead (as they did for the nuovo award) then it would help - give the near-misses some recognition without diluting the exclusivity of the true finalists.

[Bennett]

It's inevitable that if you have a transparent and democratic voting process, that some of the best games won't get nominated. Democratic systems only produce a good representation of what people like, and this is quite independent of what is good.

If you wanted IGF nominations that were reliably good, you would need to choose a small, elite panel of people with good taste and an impeccable eye. But that would be open to corruption,  because people on the panel had personal enemies or friends among the entrants, and so on. It would also tend to favour an unbalanced, one-eyed view of what's good. For example, Jon Blow (who would almost certainly wind up on such a panel) really dislikes games which are focused on narrative or world-building rather than gameplay mechanics.

I think the many-judges system is fine, but we shouldn't be too surprised or offended when good games don't get rewarded. Good films miss out on Oscars all the time, using a similar voting system. If you like VVVVVV (and I LOVE it) then you should get online and help promote it... after all, publicity is the main value of being nominated at IGF.

[Derek]

I agree with what Bennett said.  And with what Farbs said about diluting the award with too many finalists.

I'd just add a few more stages to the whole thing.

1. Technical disqualifications - not indie, broken, buggy, tech demo, unplayable
2. Weeder round - obviously bad/mediocre/uninteresting games, try to get it down to < 100 games
3. Democratic vote - pick 10 games for each category
4. Elite panel - pick 5 finalists for each category
5. Final vote - all judges pick their winner for each category

Also, I'd post the process on the results page.  Why not?  It would save Simon/Matthew having to repost it over and over again in each discussion...

[Bennett]

If you were prepared to refund the entry fees for games which were disqualified under round 1. or 2., then you could get away with having just one or two people do all the weeding-out on their own. Otherwise, having five phases will add a lot of bureaucratic inefficiency.

[hatu]

What's up with the video entries? Don't you need anything playable to take part in this?

[TheDustin]

I agree with what Bennett said.  And with what Farbs said about diluting the award with too many finalists.

I'd just add a few more stages to the whole thing.

1. Technical disqualifications - not indie, broken, buggy, tech demo, unplayable
2. Weeder round - obviously bad/mediocre/uninteresting games, try to get it down to < 100 games
3. Democratic vote - pick 10 games for each category
4. Elite panel - pick 5 finalists for each category
5. Final vote - all judges pick their winner for each category

Also, I'd post the process on the results page.  Why not?  It would save Simon/Matthew having to repost it over and over again in each discussion...

This. I really commend how the Nuovo applicants are treated, and this places the 'regular' games in the same regard. This has the judges focus on the titles that rightfully deserve attention.

[MorganRamsay]

Monaco looks like a fun little roguelike actually, kind of like spelunky, but with 4 player co-op, seems like a winning formula to me!

Woah. If your description's accurate I'm in love.

I played an early XBLA build of Monaco with Francois Bertrand, who provided his voice for some characters. The description is accurate and the game is great fun. I helped the developer, Andy Schatz, with pro audio equipment and minor engineering support for voice sessions. Sadly, there was no need of my guitar expertise. Check out the

! (EDIT: I guess Andy is registered here!)

[AndySchatz]

Haha.  The game is set in Monaco where they speak french, and I happened to know two french canadians in san diego (both in game development) so I called them up and had them do the voices for the guards.  Twas a fun day

[Guillaume]

So I'm kind of sad Mon&Bot didn't make it, but I'm really excited by Super Meat Boy, Monaco and as everyone else, Limbo. And all the rest as well 

Congrats and good lucks to all the finalists on TIGS!