next »

[Manuel Magalhães]

i wonder if there's an iphone game emulator that lets you play iphone games on windows -- that might work for igf judges who don't own an iphone who are assigned iphone games

There's only a simulator for OSX, and it doesn't allow to do certain actions like swiping.

[st33d]

I specifically had an issue with increpare's ECT on the Mac. It had righty-only controls (due to a bug) and I couldn't translate the action of the flapper to the keys in my head.

I tried it on iPad and swiping to move it made much more sense. I could translate a hand movement to the flapper's direction, but not a key press.

(Though I still think I've come a cropper from not being able to manipulate the camera and move on the iPad.)

Given my own experience I would say that emulation would be a very poor experience of the real thing and would only lead to more controversy. Especially if the game was multi-touch.

[Christian Knudsen]

I don't see a problem with not giving feedback. You can get feedback from a lot of places for free -- this is a competition, not a workshop -- and I don't think the feedback you would get from IGF judges is any more valuable in improving your game than the feedback you'd get from your players or the Feedback threads on this very forum. But there is a problem with a competition if all entries aren't judged fairly.

[ஒழுக்கின்மை (Paul Eres)]

edit: it turns out bittrip runner actually *won* for visual art, it wasn't just a finalist. not much else to say about that. the monaco guy earlier was asking for examples of games which should not have won or games which should have won instead. this is a pretty easy one





some disagreements in winners are just matters of taste, and i can see 'well, i can see how some people would think that's good, even if it's not for me'. but not in this case, i can't even see how anyone would think the visuals of that game are even competent, let alone the best of all indie games in 2010

[Matthew]

@mattheww - yeah but you also have to realize that people won't be as encouraged to work as hard to win a competition if they perceive the process of judging as non-objective or random

it's one thing to try to do 100 pushups in 30 days and fail and reach 80 instead, it's another thing to try to win a game-making contest, make the best game you can, fail to get mentioned, and not know why you failed. feedback gives the "feedback" to players so they can have some idea what they did wrong, which they would otherwise be clueless about

Well sure.  You should be entitled to a fair shake in any competition.  But the IGF is north of 500 games.  Everyone does the best they can, but keep in mind at that scale a tiny fraction of issues can create a dozen bad experiences.  That isn't to say things are perfect!  Of course things can and should be be improved.  A lot of suggestions in threads like this ignore resource limitations though ("add another judging round!", etc).

But I disagree on the nature of feedback.  There are two problems:

1) Goldilocks.  What is useful feedback?  "Too hard"?  "Cooldown time on upgraded weapon made boss fight harder than it could be"?  Feedback is a hard problem to solve in multi-page, interactive threads where the author is involved in the discussion.  Blind, one-time feedback is going to help?

2) The competition is among 500 games.  Most games aren't finalists because some other game edged them out.  That's hard to give feedback on.  This is often true in competitions in general (you may have perfect squat form, with excellent training/nutrition, but the guy next to you is just bigger or has been training for longer).

[Christian Knudsen]

Why are we discussing feedback? Isn't the issue judges not playing their assigned games?

[Chris Pavia]

edit: it turns out bittrip runner actually *won* for visual art, it wasn't just a finalist. not much else to say about that. the monaco guy earlier was asking for examples of games which should not have won or games which should have won instead. this is a pretty easy one





some disagreements in winners are just matters of taste, and i can see 'well, i can see how some people would think that's good, even if it's not for me'. but not in this case, i can't even see how anyone would think the visuals of that game are even competent, let alone the best of all indie games in 2010

I quite enjoy Bit.Trip's visuals. It's not technically impressive, but they do an outstanding job of creating an immersive style that enhances the audio/visual experience. Subjectivity is a hell of a drug.

[Christian Knudsen]

Also, it's retro!

[ஒழுக்கின்மை (Paul Eres)]

@mattheww - useful feedback would just name what specifically a judge didn't like about a game, and why they thought other games in the igf were superior to it. it wouldn't take the form of a playtesting report

here's a made-up example:

'hi. your game, kingdom hearts, while fun and entertaining for kids, was just not as compelling to me as chrono trigger. i think it's because it was a mash-up of a bunch of different characters from different disney movies and square games, rather than its own entity, so it didn't feel self-contained. i had a hard time taking a game seriously, it felt like a fanfiction crossover rather than an original story. sorry, but i had to give the vote for 'best story' to chrono trigger instead.'

[ANtY]

i repeat: a finalist for visual art

what are the judges seeing that i'm not?

I have same questions regarding many things, including Minecraft and thousands of Minecraft-clones that still sell.

[ஒழுக்கின்மை (Paul Eres)]

Why are we discussing feedback? Isn't the issue judges not playing their assigned games?

separate topic; this is an all-purpose igf thread after all, multiple subtopics can be discussed at once in it. the criticism of the lack of feedback is an entirely different issue than judges not playing games, and much less serious of a problem, but still worth talking about, because if someone doesn't know why their game lost, or even why the other games won, they don't know what to improve or what to try harder at next time

i used bittrip runner as an example of a game that just looks plain ugly to me, as if they didn't try at all for the visuals, but which igf judges felt was the best game visually last year. without feedback explaining exactly why they thought that game had better visuals than all the other games, it's hard to know what lessons can be taken away from that game winning and so many other games that (to me) look visually far superior not even being nominated. i feel that feedback (or some write-up about why they chose that game for best visuals) could have clarified that issue

i mean, what's the lesson people are expected to take from that: make your game use symbolic low-poly 3d in a retro style with clashing colors? use squares for particles? make the screen confusing and hard to read?

here's an example of a game that i thought was very visually appealing but was not nominated for visual art in the igf:

-- i can easily see that game winning for visuals, rather than bittrip runner. so it's not that its competition was all even worse

[Christian Knudsen]

It just seemed a bit as if Matthew completely ignored the larger issue of judges not playing their assigned game (and the article that started this new round of discussion) and instead responded to an older topic that wasn't currently the center of discussion.

[Schoq]

Couldn't judges be required to provide a public motivation for rating a game as they did? It would make them put more though and effort into the judging, and it would be easy to pick out bad judges who should not be allowed back.

[Hangedman]

I think the core issue is that people have come to expect good feedback and fair play from the IGF, whether it's realistic or not, and the IGF seems to encourage that belief (but not necessarily validate it in the end.)

Submitters should be outright told that their game may not get any written feedback and has a chance of not getting played if there are impediments to play, and then they can make an educated decision about whether to enter or not.

I think the whole process could do with a bit more openness, but extenuating circumstances will always occur.

[Christian Knudsen]

I don't understand why entrants would expect to receive feedback? Because it was given up to 2010?

[Matthew]

It just seemed a bit as if Matthew completely ignored the larger issue of judges not playing their assigned game (and the article that started this new round of discussion) and instead responded to an older topic that wasn't currently the center of discussion.

I just don't feel like there's much to say about it.

Games are "over-assigned" to judges to make sure there's enough scores.  Sometimes there aren't because judges flake out or because the game has compatibility issues.  If a game has a low number of scores it's brought to someone's attention to see what the issue is (I'm not sure if it's still in use in the jury system, because I haven't touched the code this year, but there was a mechanism to see under-scored games so active judges could help out where needed).

There's an admin-only report of all judges by activity level (comments and scores).  The tail end of the judging pool is chopping off every year to make room for new blood and to cull inactive judges.

Yes, things can be improved.  They can always be improved.  Organizationally, judges that are assigned a game they never actually score aren't an issue, because the data reflects their behavior.  The actual potential problem is a judge who does score a game they didn't actually play--or didn't play enough to form an opinion--because that's really hard to suss out.

[ஒழுக்கின்மை (Paul Eres)]

Couldn't judges be required to provide a public motivation for rating a game as they did? It would make them put more though and effort into the judging, and it would be easy to pick out bad judges who should not be allowed back.

that's actually a pretty interesting idea; being able to articulate the reasoning process behind your nominations / scores is something that judges should be expected to do. it wouldn't even have to be public, it could just be something the other judges or the jury could read

e.g. 'i nominated game x for audio because it has an extremely nice soundtrack; i found myself listening to its soundtrack for days, and hummed one in the shower. its sound effects are competent but not exceptional, it's the music tracks, particularly the boss theme and the title theme, that impressed me'

this would also help with the problem, which mattheww mentioned above, of scoring games someone didn't even play

[Christian Knudsen]

It just seemed a bit as if Matthew completely ignored the larger issue of judges not playing their assigned game (and the article that started this new round of discussion) and instead responded to an older topic that wasn't currently the center of discussion.

I just don't feel like there's much to say about it.

Games are "over-assigned" to judges to make sure there's enough scores.  Sometimes there aren't because judges flake out or because the game has compatibility issues.  If a game has a low number of scores it's brought to someone's attention to see what the issue is (I'm not sure if it's still in use in the jury system, because I haven't touched the code this year, but there was a mechanism to see under-scored games so active judges could help out where needed).

There's an admin-only report of all judges by activity level (comments and scores).  The tail end of the judging pool is chopping off every year to make room for new blood and to cull inactive judges.

Yes, things can be improved.  They can always be improved.  Organizationally, judges that are assigned a game they never actually score aren't an issue, because the data reflects their behavior.  The actual potential problem is a judge who does score a game they didn't actually play--or didn't play enough to form an opinion--because that's really hard to suss out.

That was actually quite informative. Thanks.

[ஒழுக்கின்மை (Paul Eres)]

as an aside, since judges are evaluating games in multiple categories, i kind of question that they can competently do that in 4 minutes. they have to be evaluating not only how *fun* the game is to play, but also its visuals, its audio, its novelty (nuovo award), and so on

can someone really confidently say that any game, even superman 64, doesn't deserve a nomination anywhere (not visuals, not audio, not technical, not overall, not nuovo) based on a few minutes of play?

i don't know about others, but i can only really focus on one thing at a time. i have to be specifically paying attention to the audio (sound effects and music) to judge it, and then pay attention to the visuals to judge those, and then think about what the game does that's new to judge its innovation, and then think about how it's programmed to judge its technical excellence, and so on -- those categories each require separate evaluation, i don't know if someone can really do all that, simultaneously, so quickly

4 minutes sometimes isn't even enough time to listen through a single song. so they are judging the entirety of a game's audio based on whatever song they happen to hear first, and they can't even pay attention to that song because they're also playing the game to see how fun it is and evaluating the game's graphics and programming as well

[Matthew]

as an aside, since judges are evaluating games in multiple categories, i kind of question that they can competently do that in 4 minutes. they have to be evaluating not only how *fun* the game is to play, but also its visuals, its audio, its novelty (nuovo award), and so on

You do realize the catalyst for the current discussion is a game that so completely invokes Gameboy aesthetics that it presents itself as a fictional port of a 1992 title, right?