next »

[lowpoly]

So as my game starts to resemble less and less a lump of crap, I'm thinking that eventually I'm going to need some people to test it out with me.

Just wanted to get people's opinions on how to go about this. What kind of options have you guys explored to get your games tested and how was the experience overall? Do you find it better to go with other game devs or does that lead to additional problems? I'm hoping to maybe one day sell the game so I'm a bit hesitant to offer any sort of public download right now (even tho that's probably the best option).

would love to hear your experiences, be it freeware, commercial, or whatever.

[ravuya]

Maybe you can do it in phases; first a batch of peer game developers to catch the technical flaws, and then a sampling of game players to make sure you can catch gameplay flaws.

[PoV]

If you feel your game has no bugs, then your game has no bugs.


Free testing methods:

Peer testing has worked well for me, as developers tend to have bizarre PC configurations (multiple monitors, multiple gamepads, etc).  There's also a level of critique a developer can give that goes above and beyond, but you need to be clear and capable of asking for the harshest of critiques.

One I'm looking forward to giving a go, bringing several friends over for a testing weekend.  It's maybe not totally free, as you should feed 'em.  With Pizza and Pop/Beer, you can't go wrong.

Another I've considered but haven't tried yet is school testing.  In theory, this should be a really great user test environment, but perhaps not the best bug test environment.  Your choice of High School vs. Grade School should reflect your desired ESRB rating (E vs T).  Visit a local school and speak to the principal about arranging such an event.  Borrowing a room, any TV's or computers, etc.  Be prepared to supply equipment though.  You may need signed permission forms too, especially if you want to videotape aspects.  Recording features, weather integrated in to the game, or via a VCR are a great help for tracking down bugs.  And finally, be prepared to be asked back to talk about game development in a programming class.

And of course, releasing the product after some free testing on your website can work too, and wait for bugs to come in by e-mail/forum.  You should do your best to be sure she runs on a number of configurations first.  Then approach your distribution channels, and media coverage after a few weeks of user feedback.


Paid testing:

Testing has to be the worst job in game development.  Playing the same game over and over again is awful.

Having an in house tester has worked nicely at places I've worked, but it's no replacement for a harsh round of QA at Beta.  The only issue has been down time if they're full time.  But hey, as a small developer, you can always find some task for them to do.  You can always think of them as an assistant.  Converting files, managing forums, doing tedious tasks like building maps for casual/puzzle games.

And as for outsourced testing, I think it's essential for any retail or console downloadable game.  Some publishers and portals have testing departments.  The testing company will send you a bug list, or update a bug database for you.  Some of the better ones record their testing sessions, and send you screen shots or video of the bugs they're trying to describe.  This can be a pricey service though, but as I said, essential in some circumstances


And some notes on integrated recording features for crash reproduction, since I seem to be in a blabby mood:

Be sure your input update is written/recorded immediately before the game processes it.  In other words, at the end your control interpretation code/handler.  That way it can "in theory" reproduce a crash, instead of coming up a frame or two short.  So if it doesn't crash, then your crash is likely related to an uninitialized variable.  So you forgot to set it in a function, initialize it to zero in the constructor, or set/handle a pointer.  Hardware/Driver/Compiler problems are a possibility, but there's an incredibly good chance it isn't.  Be wary of GCC the 3.0.x series though (anything newer is much more reliable, 3.1.x+, 4.x, ...).

Compressed input recording is good for attract mode and in game replays, but not so much crash reproduction (especially if you're waiting on changes before you write).

Prefer fixed framerates for internal clocking.  This'll make reproduction/playback of non rendering bugs/crashes easier.

Recording the initial random number seed is helpful too.  And if you don't reseed, you don't need to record reseedings.  It's so very easy to do this too, it can essentially be your file header of an input recording file.  Read and populate the seed first, then playback all the following input commands.  Technically, you shouldn't ever need to reseed, unless you're making a multiplayer game.

[lexaloffle]

The most valuable testing I get is from people who are willing to play the game right in front of me. Even a 5-10 minute session is enough to get some gems of insight into how players react to a game. I don't plan this though - it just happens naturally as a consequence of having an office in the middle of town. For remote beta testers, I've tried doing some recording and playback within the game which has been handy too, but not as good as being able to question someone on the spot about what they are thinking.

As for who to test with - hedge your bets. Everyone can offer something. Developers are good at being picky and articulate about their experiences. Newbies are good at exposing incorrect assumptions about how the game will be played. I find that you don't really need that many people to get a grasp of how a game comes across. Unless the game is particularly complex, perhaps.

I don't usually worry so much about technical testing early on. Even before release, you should only need to test on common configurations unless you are going to burn an unpatchable gold master CD or something. It's much easier just to do a gradual release, and pick up issues before you have a large number of players. This has worked well for me, but my games don't require much hardware support.

[Bezzy]

Devs and professional testers are definately good for technical advice, as well as design - they have real grounding in the practicalities. You can listen to what they say.

I find you get far more worth from non devs watching them play. What they say is not quite as important. Not always, obviously - if they're explaining why they're finding something frustrating, that's good illucidation into why they're struggling. You just have to take a lot of what they say with a pinch of salt. Their actions speak louder than their words.

Have one set of testers who do repeat plays (these are the more technically qualified group), and then get as many "tissue testers" as possible to see how people interact with the game without any prior knowledge of it. For tissue testers, the less technically adept the better: hardcore gamers can handle offensive interfaces with ease, and these people won't highlight problems you may have on that front.

Don't prompt the testers as you watch them play, even when they're stuck. Only prompt them at the point where they look like they're going to give up trying.  By then you should have figured out what they were having a problem with. Explain this to them before hand so that they know you're not there in a "tour guide" sense.

[lowpoly]

incredibly useful responses everyone, I def appreciate your input

Going back to a 'gaming night' idea, has anyone tried this virtually? The majority of the people I'd want to invite live far enough away that it would be a pain to organize but I wonder if a shared session (irc/video chat/etc.) would have similar benefits? I know nothing replaces the fun of playing together and you would lose the dynamic of people observing and feeding off each other, but there might be something gained by getting everyone on the same page at the same time. I assume a lot of indie teams work this way, no?

[Bezzy]

incredibly useful responses everyone, I def appreciate your input

Going back to a 'gaming night' idea, has anyone tried this virtually? The majority of the people I'd want to invite live far enough away that it would be a pain to organize but I wonder if a shared session (irc/video chat/etc.) would have similar benefits? I know nothing replaces the fun of playing together and you would lose the dynamic of people observing and feeding off each other, but there might be something gained by getting everyone on the same page at the same time. I assume a lot of indie teams work this way, no?

Well, interestingly, Microsoft's testing group has rooms set up where they record the user, and record their screen separately. No-one goes in these rooms with the people so that they can glean as natural a user-experience as possible. There's zero poking, or hints and whatnot.

If you have the resources to make this happen (fraps + webcam?) and can get those testers to set it all up for themselves, go for it. Sounds like it'd be a big headache, to be honest.

I think that unless it's experienced devs you're dealing with, adding irc/chat into the mix would probably increase your noise to signal ratio if you've got tissue testers giving feedback.

[lowpoly]

If you have the resources to make this happen (fraps + webcam?) and can get those testers to set it all up for themselves, go for it. Sounds like it'd be a big headache, to be honest.

I think that unless it's experienced devs you're dealing with, adding irc/chat into the mix would probably increase your noise to signal ratio if you've got tissue testers giving feedback.

I was thinking more along the lines of getting maybe 5 or so people I know, and organizing a time where everyone could play. Open up an IRC channel so people could converse in real time and just go from there. The extent of any video chat would be just so they could point a webcam at their screen and say, 'this texture looks weird.'

I think the logistics of trying to record their play session would be a nightmare and to be honest, I don't think my game is complicated enough to really warrant that. Maybe down the line when it's closer to completion, that setup might be valuable. Again, it's an indie game, so the budget of course is nil.  :D

[ஒழுக்கின்மை (Paul Eres)]

I wrote a entry on playtesting which some people liked which you might be interested in: http://rinku.livejournal.com/1200653.html

I don't particularly like that IRC idea because, you know, it's more important to watch someone play the game directly than to have them play it online and give you a report about it. There's a lot you can miss if you're not sitting there behind them watching them play. Online playtesting should only be a supplement for watching people play the game first-hand, not a replacement.

[lowpoly]

thanks for the link, lots of useful information in there!

I don't know, maybe I'm a special case but the possibility of in-person tests is practically zero for me. That's the underlying basis of my original question I suppose, how to go about finding the right people. I guess I was under the impression most people relied heavily on online testing as well...

[ஒழுக்கின்மை (Paul Eres)]

Aw, come on, you don't have siblings or friends or parents or children or roommates or cousins you could force to play your game or anything? Nobody has *no* access to anyone else in real life, except prisoners in solitary confinement or something.

[sega]

I don't get commission, and I'm not rewarded in any way if you go with us, but I just wanted to let you guys know that my day job is with a 3rd party testing company on the East Coast, USA called Absolute Quality.  PoV is right, it IS expensive to outsource testing, but some of us here (and I'm sure other testing companies as well) have almost a decade of experience, and kick out tons of good info that many people wouldn't think to check out when testing.  We've actually done testing for some indie developers in the past, but they're usually well established before they come to us.  If this is your first game, I imagine your money is needed elsewhere more, and friends would probably have to suffice.  I know I can't afford us.  Luckily, I know a lot of testers .

It's tempting for me to think I can handle all of my game's testing, because I'm a lead tester myself, and I have a lot of experience, but man... a different set of eyes can turn everything upside down sometimes.  I know I'm not good enough on my own if I'm too close to the game.  If you can afford it, professional testers are invaluable, even if for a couple of weeks of work.

By the way, if anyone considers this spam, I'll definitely be happy to modify this (I already removed the hyperlink to be safe).  I'm just proud of our work, and thought that since we were helpful to some indies, there might be some here that would be interested.

[frosty]

In-person user testing is pure gold.

You can never do enough of it, but even 2 or 3 tests will give you a lot of information that you can't get any other way.

Most users actually blame themselves when they run into small problems, like when they have trouble starting a new level, so they won't tell you about it in e-mail-based feedback.  You will also see how players actually play the game versus how you intended it to be played.

[TeeGee]

Yup, in-person tests beat them all.

I have (and had) many different playtesters - here are my experiences with them:

- Dedicated beta-testers (recruiting from gamers, amateur and professional devs - I wanted to have a wide spectrum) are great for bug hunting and suggesting improvements (after some time they become very familiar with the game). On the other hand - they are bad for the gameplay/balance/accessibility testing, they are just to used to the game to judge this stuff.

- Forum feedback is a good way to get quick, general opinions. However it's hard to get any good suggestions there. Just a sort of probe...

- Showing the project to random gamers and asking for their opinions is similiar thing. Good way to get the general feeling, bad way to get anything else.

- I have few fellow game journalists that I showed the game to. Not to get a review, but to ask what they would say if they would have to write one. This sort of feedback is similiar to what you get from forums and gamers, but much more precise and overall worthy. Journalists usually know what they like/dislike and what your game possible flaws are.

- Feedback from other professional indies might be very useful, but you have to remember one thing - they are creative, highly intelligent guys with a strong vision. They have lots of great hints, but they will have their own idea on what your game should look like and that idea might not be necessarily better for your plans. Listen to them, but don't follow exactly what they are saying .

- Still, when I was looking for a way to improve the user interface in MAGI (which was critiqued a lot), all of the above was nothing compared to sitting my girlfriend before the PC and watching her play (silently taking notes).
As a person that doesn't play games at all, she was a great, exaggerated example of what the new user is feeling like. 15 minutes of watching her struggle with the game, gave me much more than tons of brainstorming done with the dedicated playtesters. I clearly saw what was difficult for her (ergo: not intuitive enough) and what aspects of the UI she found confusing. I also observed many mistakes and behaviours that I even didn't thought oo.
Update made after taking this 15 minutes of testing into consideration, caused a big increase in sales...

So I agree with frosty - In-person user testing is pure gold.

[frosty]

I would also add that in-person testing can be rough on the ego at first.  It's frustrating to see players struggling with something you don't want to change.  It's tempting to dismiss it as a fluke.  But eventually you have to do the right thing and change it. 

Also, you will clearly see if players like the game or not, regardless of what they say about it.  That's why when I do tests, my focus is the interface.  It's okay for them to not like the game, but they should at least be able to figure out how to play it.

[Ryan]

I don't know if your game is going to be freeware or not, but if so you could always handle testing like the guys at Data Realms do. Seems to be working good for them, the game has come a long way from when it was D.I.R.T. Also, it is super exciting to find out there is a new build to try out, etc. .

If it's commercial, in my opinion, in-person tests are your best bet, especially since you do not have cash on hand to have a company do professional testing. Think about your target audience, and then get your friends/sibling/wife/etc. to play a five minute round of your game. Surely you can find someone.

Keep us posted, based on the screenshots from your blog your game is looking very good

[lowpoly]

 
How embarrassing, I never responded to my own thread.

In-Person testing is the way to go...

I think this is painfully obvious now, as my limited run, distribution to my friends testing scheme so far has been an unmitigated disaster. I'm going to have to figure out a way to either setup a gaming night or just do one-on-one's with people once the game is at a point that I'm ready to receive gameplay feedback.

Keep us posted, based on the screenshots from your blog your game is looking very good

Thanks man, one day I'll actually update the thing with something useful