next »

[ஒழுக்கின்மை (Paul Eres)]

from what i remember, it was removed because, even though it was required, nobody was doing it anyway. but yeah i'd prefer something like that be required too

failing that, it might be interesting just to have the scores of games publicly available, and the discussions the judges have about the games publicly available. that ties in with the whole transparency thing. they couldn't be immediately available obviously, just after the igf ceremony, they could be released to entrants. i imagine there's lots of useful info in the discussions about games that developers could use, but have never had access to

[phubans]

Wait... What? They removed the feedback feature? So now I'm not even getting feedback for my entry fee?

[Manuel Magalhães]

They didn't. It's just isn't obligatory anymore.

[Matthew]

Paul:  You said banning or reprimanding judges for 5-minute play sessions would be one change that would fix the IGF.  That's a pretty hyperbolic stance.

Zooming in on the details like this is frustrating and pointless, especially because both of us seem to have an inexhaustible energy to go on and on forever.  Zooming out produces such better--and fruitful!--exchange of opinions.  For instance:

I absolute agree that judge interaction quality is hugely important!  In my mind, the jury system solves this problem completely, in so far as the games the jury ends up discussing are discussed in great length.  From this perspective, the IGF at large actually becomes a discovery problem; how do you get the right games in front of the jury?  How do you filter 500+ entries down to potentials?  The main pool of judges has already been crudely adapted to fit this purpose.  Two years ago, the finalists were the top-ranked games as averaged by all judge scores.  The system today is run in nearly the same way, except numerical scores have been replaced by checkboxes.  

Inefficiencies are forming with these processes because their initial design and the structure of today's IGF have diverged.  But the current controversies are all focusing on the wrong things:  100% judge assignment completion is totally irrelevant, and judge playtime is just a tiny piece of the puzzle.  If I sat down to work on IGF for a month, my energy would not go into addressing either of these (sure, I might crunch some numbers to determine how much to increase the judging pool, but ideas like APIs like time tracking are honestly crazy talk).

Think about it like a content discovery problem!  Some random thoughts in that direction:

- Discussion is hugely important to discovery.  In fact, there are already huge comment threads on some 2012 IGF entires (both finalists and non-finalists), but a lot could be done to decrease comment friction and get more dialogue.

- You basically want to build tools for the jury directly (an individual jurist can lobby for any game).  How do you do that; what sorts of views would help different jury personalities discover games?  Keep in mind finalists are chosen by jury approval voting.

- What does an ideal judge interface even look like?  Event wall (Facebook) with all activity?  Why even assign games (as a thought experiment, what's the difference between assigning games and showing every all games with a fixed-per-person random sorting)?  Would an upvote/downvote system make sense, or is it too easy for early votes to submarine games?

Or, you know, everyone could just yell at each other through a closed door about semantic minutia

[ஒழுக்கின்மை (Paul Eres)]

Paul:  You said banning or reprimanding judges for 5-minute play sessions would be one change that would fix the IGF.  That's a pretty hyperbolic stance.

i'm not sure how this is hyperbolic. now, from your perspective you think the igf is fine, and that the drama is just an overreaction. isn't it at least true that that one change would end most of the drama? the reason people are getting so worked up in that rotten cartridge thread, here, and elsewhere is that they feel judges aren't doing their job and that the igf is defending them not doing their job. if a change were made that judges that didn't do their job were not re-invited, or reprimanded, that  one change would end most of the criticism of the igf -- i don't see how it's hyperbole to believe that.

I absolute agree that judge interaction quality is hugely important!  In my mind, the jury system solves this problem completely, in so far as the games the jury ends up discussing are discussed in great length.  

i think it's hyperbole to say that it solves the solution "completely" when you have cases like fader getting an honorable mention for visuals over games like dust: an elysian tail and the iconoclasts? andyschatz says that the proof is in the pudding. the proof of the jury system completely working is pretty weak to me, considering finalist selection. now i don't think that the selections are necessary worse than before there was a jury system -- but i certainly don't think they're clearly better, either

But the current controversies are all focusing on the wrong things:  100% judge assignment completion is totally irrelevant, and judge playtime is just a tiny piece of the puzzle.  If I sat down to work on IGF for a month, my energy would not go into addressing either of these (sure, I might crunch some numbers to determine how much to increase the judging pool, but ideas like APIs like time tracking are honestly crazy talk).

i agree with the api idea being crazy talk (i reacted to it similarly when it was suggested), i think it's a violation of privacy to do something like that. i do think games can individually elect to use time tracking if they want (preferably informing the judges of this ahead of time), but i don't think the igf itself should track it

that said, i find it surprising that if you had a month to spend working on the igf, that you would not spend it on this issue, if only for public relations reasons. i think it's very damaging to the reputation of the igf among indie game developers for the igf to agree more with jenn frank than with rotting cartridge

[Bennett]

But the current controversies are all focusing on the wrong things:  100% judge assignment completion is totally irrelevant, and judge playtime is just a tiny piece of the puzzle.

I want to second this. There surely are games that get shortchanged under the current system (especially on mobile), but Kale in Dinoland wasn't one of them. The reason you get 8 or more judges per game is for the sake of redundancy, not because 8 are needed to make a fair assessment at the judging stage. And I think if the four judges who did play it only played for five minutes, that's totally valid too.

It's true you might miss the odd gem if you tend to quit a game that hasn't grabbed you after five minutes. But drawing attention to this kind of slow-burn game isn't really a goal of the IGF. The point is one of two things, or some mix of the two: 1) to identify and promote the best games, for the sake of the growth and popularization of the medium as a whole, or 2) to help to uncover games that aren't well-known and increase the pool of well-known developers.

We can argue all day over whether the IGF should be more about 1) or more about 2), but it's definitely not about giving press to games that make a weak first impression. Sure, it's nice if good slow-burn games get recognized, but it's not anybody's primary goal. So I think if your judges install your game and only want to play it for five minutes, that's a completely valid judgment of it. With those things in mind, I really struggle to see how the Kale guys have a valid complaint.

As someone pointed out on twitter, there is a general fear out there that your game might not get played by a judge at all. That probably does happen sometimes, and it's pretty bad if it does, but the chances are probably best minimized by 1) requiring judges to log comments (even private, jury's-eyes-only comments) on every game that is assigned to them, and 2) not inviting bad judges back the following year. I'm sure the Oscars, Grammies etc have similar hassles.

As a final point, I think it's completely inappropriate that the Kale guys wrote to the judges to nag them to try the game during the judging process. It's almost threatening behaviour, and if I had been one of the judges I would have recused myself at that point.

[Matthew]

i'm not sure how this is hyperbolic. now, from your perspective you think the igf is fine, and that the drama is just an overreaction. isn't it at least true that that one change would end most of the drama?

My goal is to make the IGF better, not to end "drama".

[Glaiel-Gamer]

isn't it at least true that that one change would end most of the drama?

IGF without drama?

[Matthew]

As someone pointed out on twitter, there is a general fear out there that your game might not get played by a judge at all. That probably does happen sometimes

To my knowledge this has never happened (at least during the time I ran the database side of things more intimately).  We had a list of games sorted by judge scores, and rigorously investigated the zeroes.  A couple of people bowed out--we would contact them about broken builds and they'd basically say "never mind"--but finalists have never been announced with zeroes still on that list.

[ஒழுக்கின்மை (Paul Eres)]

My goal is to make the IGF better, not to end "drama".

that doesn't really answer the question. also, making the igf better includes making indies who enter it feel as if is were a fair process. if a significant number of independent game developers feel that the igf is unfair to entrants, that would be bad for igf. as noted in this thread, there are many prominent indies who don't even enter it because they know that the type of games they make never win, e.g. jeff vogel, cliffski, etc. -- i can only see that increasing

As someone pointed out on twitter, there is a general fear out there that your game might not get played by a judge at all. That probably does happen sometimes

To my knowledge this has never happened (at least during the time I ran the database side of things more intimately).  We had a list of games sorted by judge scores, and rigorously investigated the zeroes.  A couple of people bowed out--we would contact them about broken builds and they'd basically say "never mind"--but finalists have never been announced with zeroes still on that list.

if it has never happened, what about the cases in the comments section of rotting cartridge where people asked for (in some case receiving, in some case being promised and then not receiving) refunds exactly for no judge touching their game? it could have happened before your time, but still that goes against the "never" part

here those cases are, for reference:

This happened to me too a few years back when IGF was doing mod submissions. It was networked and found out that there wasn’t a single judge that even logged on to play. I got an apology from Simon and a refund for my submission but that was it. I’m disappointed to hear that the IGF is still continuing this practice while claiming to represent a fair process.

This happened to me way back as well. We submitted a download link to our PC game, which was hosted on our own server. The link was unique to IGF, and it never got a single hit.

Obviously, we brought this up, only to be completely ignored. Only when this started getting a fair bit of publicity via slashdot did they contact us about it with about the same reply you had. We talked to them, and got an apology but not much else – they offered to send us some stuff. We accepted and gave them a shipping address, deciding it wasn’t worth more of our time (we had a game to work on after all).

So that was the end of the story. And I mean the end of the story, since the part where the stuff they promised actually arrives never happened either.

Very similar to my experience with indiecade in 2010; they took my money but never even logged into the game client (it’s a client-server game; they did log into the website, once). They made some vague excuse that they would still supply useful feedback, but then later claimed their database had been corrupted and all their review files were lost.

[Matthew]

I absolute agree that judge interaction quality is hugely important!  In my mind, the jury system solves this problem completely, in so far as the games the jury ends up discussing are discussed in great length.  

i think it's hyperbole to say that it solves the solution "completely" when you have cases like fader getting an honorable mention for visuals over games like dust: an elysian tail and the iconoclasts? andyschatz says that the proof is in the pudding. the proof of the jury system completely working is pretty weak to me, considering finalist selection. now i don't think that the selections are necessary worse than before there was a jury system -- but i certainly don't think they're clearly better, either

Fucking semantics again, but here we go:

We are talking about two different things.  I am talking about the "quality" of the jury discussion.  It is lengthy, and involved, and hotly debated in various points.  When I am speaking of the quality of a judge's interaction with a game, I am talking about the depth of their understanding of that game.

You seem to be speaking about "accuracy" of the jury system; asking, "how can it be correct, when Game X or Game Y are overlooked?"  There is no objective measure of games.  So yes, a jury--especially something like an art jury--is going to produce a set finalists that may differ from your tastes.  That's kind of how the world works.  I get it; you like pixel art and anime.  Not everybody agrees with you.

But with the IGF, I am satisfied that the jury process is thorough.  I am worried about the content discovery problem as the IGF entry list balloons past 500.  If I were to spend time on the IGF, I would focus my efforts on tools and workflow to improve the discovery process for the main pool of judges.

The jury's bandwidth has an upper bound.  They can only spend time on so many games, and the purpose of the main pool of judges is to help filter the entries to a digestible number.  This is the potential failure point for the mythical deserving-but-unnoticed game, and where "fairness" is most fragile.  If your game makes it to jury discussion it's had its "fair shake".

And, no, I absolutely don't agree with Rotting Cartridge.  Their own summation is focused on a meaningless stat (because judges are redundantly assigned to games exactly because many judges can't squeeze in all assignments during the holidays):

Judges not playing a game they are assigned to judge, for any number of minutes, is simply not acceptable.

Especially in light of their own opinion of their game:

Going in, we didn’t expect to get nominated for anything.

But I'm not actually clear whether or not they were aware of the details of the judge/jury split.  But anyway, I'm repeating myself again

[Glaiel-Gamer]

My goal is to make the IGF better, not to end "drama".

that doesn't really answer the question.

yes it did?

[Matthew]

Judging mods were before my time, so I can't speak to that (and I believe it was before any serious backend was even in place).

I would need details on the other one.  The fact that they're talking about a download link and not the IGF FTP site is weird.

[ortoslon]

Short Accessible Indie Games Festival

[Glaiel-Gamer]

Short Accessible Indie Games Festival

game can be accessible and engaging from the start without being short or lacking depth and complexity, hell every game can and should be engaging from the start.

I asked on twitter about games with terrible intros that make the game better in the end, the only one that came up that I could agree with was Upgrade Complete.

[Radix]

Short Accessible Indie Games Festival

game can be accessible and engaging from the start without being short or lacking depth and complexity, hell every game can and should be engaging from the start.

I asked on twitter about games with terrible intros that make the game better in the end, the only one that came up that I could agree with was Upgrade Complete.

Are you saying that's not a subjective thing?

[DavidCaruso]

I think that the IGF judges have pretty awful taste in art if they decided that a barely passable "minimalist" aesthetic, a ball on a racetrack surrounded by spheres, or a photo-ripped blurred forest background with random eyes looks better than a "maximalist" pixel or digital art aesthetic. Then again these people also probably think that Knytt Stories has good atmosphere, and they also appear to be under the impression that Johann Sebastian Joust is a videogame. Wait I forgot I'm not supposed to post about "semantics" whoops.

[Matthew]

It's a valid concern!  Content discovery, in general, is hard to solve in ways that preserves depth and variety.  There's a reason Reddit just a bunch of cat images.  I do think "don't be boring" is the best mantra you can hold to as a developer, even if your game is longer-burn and deep and twisty.  There needs to be something compelling (and there are as many ways to do this as there are creative people).

At the same time, I do see a lot of judge discussion on games that have more recessed content, so maybe it isn't as big of a concern in reality as you might expect.

Paul:  You frustrate me, because despite your ability to post endlessly, you very rarely enter into a mutual brainstorming mode.  My last big diatribe was about:  "Hey, I think this other way of thinking about the IGF might be more productive", followed by a bunch of "How do we...?", "What does this look like...?", and so on.  You ignored all that, which is a shame, because I think you would have some ideas if you stopped looking at everyone's posts for attack points and focused more on contributing.

P.S.  That database corruption feedback example is actually talking about IndieCade, not IGF.

[Matthew]

I think that the IGF judges have pretty awful taste in art if they decided that a barely passable "minimalist" aesthetic, a ball on a racetrack surrounded by spheres, or a photo-ripped blurred forest background with random eyes looks better than a "maximalist" pixel or digital art aesthetic. Then again these people also probably think that Knytt Stories has good atmosphere, and they also appear to be under the impression that Johann Sebastian Joust is a videogame. Wait I forgot I'm not supposed to post about "semantics" whoops.

There are emails very much like this inside the juries!

[Bennett]

Short Accessible Indie Games Festival

If the game is accessible it won't need to be short.

Accessible Indie Games Festival: what's wrong with that? Why do inaccessible games need to have an equal chance? Suppose we give all the prizes to games that only serious videogame fans could enjoy - how does that help the scene, or the artform as a whole?



/edit there are lots of ways for games to be accessible. The Minecraft wiki model is one example or a non-traditional way.