You pay 95$ to enter a compo where your item is completely ignored by those in charge to review it.
This is that hyperbole you were called out on (emphasis mine).Missing Judge Scores
I'm going to try one more time to explain how the system works, and then I give up.
- All games are scored by multiple judges
- Some percentage of judge/game combos are never scored (compatibility, scheduling, judge gets busy, whatever)
- Games are over-assigned to judges to account for this
- Multiple admin-only scripts exist to show games sorted by # of scores. If a game is holding steady at zero scores, someone looks into it (usually a broken build or upload--although in recent years judges are encouraged to email directly about such issues).
So yes, not all judges score all of their assigned games. BREAKING NEWS: Thankless volunteer workers sometimes inconsistent!
But this is just a statistic, and it's an easily viewed, easily controlled statistic.
To my knowledge, no game has ever
gone unscored without the explicit permission of the entrant (a couple of people have been contacted about broken builds and replied with "never mind", which is what I mean by that).Judge Score Quality
This is a separate
issue, and a harder one to quantify. Is 5 minutes too little? 60 minutes? 23 minutes? It's obviously a spectrum (and if 60 minutes is fine, what about 59 minutes)? Focusing on time spent playing is a distraction, in my opinion. Many, many games couch themselves in the trappings of very particular styles and genres which, for better or worse, speeds up the process.
There are a lot of divergent opinions here. I encourage anyone with the passion to write up their thoughts to also contribute that passion to the judging pool! The IGF needs more people like you.Moving On
Weirdly, Rotting Cartridge has actually focused on the first issue. From their update: "Judges not playing a game they are assigned to judge, for any number of minutes, is simply not acceptable.
That's the wrong approach to strengthening the IGF judging process. If, out of 150+ judges, 20% always flake out, then you need 20% more judges. This is already built in to the system.
In fact, if you push judges to 100% their assignments, you're going to degrade score quality as more and more people phone in scores. This is dangerous, because I (as a backend coder) can't just pull queries showing Brandon et. al the status of things. I would be massively suspicious of the results if 100% of judge assignments were scored.
As much as I think entrant feedback is a red herring for the IGF, having it there probably increases score quality (by lowering
the total number of scores, actually--it prevents casual scoring).For Developers
BTW, there are a couple things you can do as an entrant to ensure your game has more judge scores:
- Be proactive! The single biggest reason for low scores on a game is compatibility. Check your installer on a fresh system (keep a clean Windows 7 install on VirtualBox or something).
- Be responsive. If a judge emails you about a broken build, don't wait to fix it.
- Don't be boring. Watch this: http://www.youtube.com/watch?v=YCCbRBPWiZ8#t=9m52s
P.S. To Greg: Yes! I definitely feel like some tipping point is reached in these conversations where it goes back curious questions and probing for more data/answers, and ends up looking like a malicious "damn the man" insistence of conspiracy. I almost just fried this whole reply because I feel like all it will do is provide new "openings" for people to pick at.