next »

[Michaël Samyn]

We're finally getting closer to what is beginning to look like a finished game and I'm starting to realize that there is no way we can thoroughly test the game on our own. Between the bug fixing and the polish, we don't have time for that. A minimal speed run of the game takes me about 2 hours. And then of course you skip over most things that should be tested. So seriously long hours of beta testing are in order.

I feel uncomfortable about a public beta because a) if people are like me, they'll just get the beta out of curiosity with no intention of really testing and b) well, I don't like the idea of spreading our game around at this point. So I was thinking of simply inviting some people to come and test the game in our studio.

But I'm curious to hear how other people handle testing. Inspire me! 

[TheMeatyBall]

I think there was a thread for this somewhere, I'm gonna see if I can dig up a link

EDIT:

http://forums.tigsource.com/index.php?topic=125.0

Read through it, I for one found tons of good tips.

[ஒழுக்கின்மை (Paul Eres)]

With ID, I first did a closed beta for most of a game's production, around 20-30 people had access to it while it was in development for most of its development, so it was a continual testing phase, every new version was played by some of this group, and their feedback helped its development. I then had an open beta when it reached 90% done, which meant a few hundred or so playtesters (only a few of which gave feedback). Overall the closed testing was far more useful, but there were still useful things that I got out of the open beta too, but it wasn't essential, whereas the closed testing group really helped in an essential way.

So what I suggest is you just pick the 30 people who spent the most time on Endless Forest forums and ask them, or something like that.

[gummikana]

With Crayon Physics Deluxe I've been doing, what the Valve people refer to as Focus testing. But it's not really focus testing. It has nothing to do with marketing what so ever.

Testing for design
The way that I've been doing it is every week / every other week I've invited a friend to play the game. We go to a pub or to my place and they play the game and I watch them play it. I don't give them hints or tell them how to play the game. This has been extremely useful to me, but I've been doing since beginning of the project. I'm not sure if it's that useful in the end, because you can't do much changes to the game at that stage. Ron Carmel has a good article about this kind of testing: http://2dboy.com/2007/11/12/rons-rules-for-playtesting/

Testing for PC issues and online experience
The other kinda of testing that I'm getting into right now is probably closer to what you're planning of doing. The idea is to invite friends to download and play the game at their own computers. I've done with few friends already, but I continue to do it in small patches. Mostly because that way I can fix the problems more easily as they emerge.

The third stage of this testing process happens after I've started taking in pre-orders for the game. At some point I'll run a beta with the people who have pre-ordered the game and that's done it's release time.

[ChevyRay]

Get someone who doesn't play games much. They ALWAYS find bugs.

[ஒழுக்கின்மை (Paul Eres)]

I agree with that. By far my best playtesters have been people who don't normally play games, or don't normally play games in the genre I'm making the game in. Because all the other playtesters tend to compare it to games they've played or games they've played in that genre, and often (and this is usually unconscious and unavoidable) suggest things to make it like other games, and see other games as the standard to reach for, instead of seeing the game as its own entity.

[ChevyRay]

Yes. Also, they DO things that gamers wouldn't do, sometimes unexpected things. Like, players expect a certain amount out of games... but if you get someone who doesn't expect anything about the game, and just figures it out as they go along, you'll find corridors and niches being exploited like never before.

My girlfriend kicks butt at it :D

[Powergloved Andy]

I know The Path is supposed to be awesome from what I've seen, and I know you're not comfortable releasing a beta to the public, so I guess a closed beta with a few friends? 2 hours isn't so bad for a speed run, just dedicate a weekend to it or something, and just do everything you wouldn't expect to do? 

I mean you could always offer updates after the release to patch over any bugs 

[Hajo]

if you have a bigger group of testers, you can let some play the game as they want, and others get list of things to test - these lists will be made from bugs/problems found or suspected during development and earlier testing stages.

E.g. if you have a game about traffic and traffic lights have been a problem in the past, advice some testers to check the traffic lights for proper operation extra carefully.

Using gaming illiterate people as testers will also test the accessability/usability of your game, but you must expect less precise reports, at least expect that the terminology of such people does not match what you are used from gamers. E.g. gamers have a good idea of what "ping" means and "lag", but non-gamers will use different words for such effects.

[StewartRoonie]

Wow! It is good to see so many people taking initiative and giving us a good info on Beta testing tips... Thanks a lot and I look forward to see more from you all. 

[Michaël Samyn]

We will be starting the testing of our game The Path next week. The idea is to invite volunteers to come to our studio for a day, one at a time.

First we will confront them with the game without any instructions and observe how they interact with it. As a test of how easy the game is to control and how clear the communication is. I imagine this play test will take about 15-30 minutes.
Then we'll tell them more about the game's structure and how to play it efficiently. We'll ask them to write down what happened in which circumstances when they encounter a bug, or something else that they find strange. And then we'll leave them alone to play. For several hours. Then we'll feed them and discuss the game. And then we'll let them play some more.
The first testers, and the less experienced ones, we'll instruct to try and play the game from start to finish (which I doubt they will be able to do in only one day). The more experienced gamers, we'll allow to do the same but pay special attention to specific systems.

If anyone has some more tips, please share them.

And if anything noteworthy happens, I'll post about it in this thread.

[Tobasco Panda]

So I decided to write up the types of testers you will need on just about any project. This may not be what you are looking for or may seem completely obvious, but in case those who have not been through the testing process it seemed worth writing up.

There are more or less three types of testers you will need: Tissue Testers, Focus Testers, and Beta Testers. These will play your game for differing lengths of time and have different uses.

1) Tissue Testers - These are people who you have play your game for maybe an hour tops. They are for testing the strength of your tutorial elements, the ease of your interface, the hook of your game, and how easy your controls are grasped. The reason for only using them for an hour is that after that long they have started to LEARN what you are doing and adjust to it. Once they have reached the point that they are learning how to play, you are done with them and should toss them out (like a used tissue). Try to refrain from telling them anything, just let them play because you are trying to simulate that first hour of a user playing your game with out you being there to hand hold them. If they don't get something from the game itself teaching them, you know what you need to work on. For tissue testers you want a LOT of people, because this is the one area of your game EVERYONE will see, and be sure to get some NON-GAMERS, to make sure you aren't depending too heavily on 'expected vocabulary' (things like, analog stick, d-pad, DoT, etc).

2) Focus Testers - These are your mid length testers, one full play through, or half a play through. These will give you a better idea how your player will progress and if there are any major issues in your difficulty curve. It is also to check if there are any areas of your game (levels if it is that type of game) that they particularly had thought were too easy/hard or interesting/boring. They should also give you an impression of how enthusiastic they are about the game after sinking some time into it. This is usually the same group that a publisher will use for marketing data.

3) Beta Testers - These are the people who are going to be playing your game for hours upon hours looking for issues. Testing your game loop is one of their main functions (mainmenu->loadlevel->quit->mainmenu->newgame->... etc). This is when having a form of bug database software is pretty important so that you can track these issues, mark priority, assign, and all of that good stuff. They usually play your game mostly at the end when you are trying to go from Beta (first submission for final game) to Gold (the actual build you ship).

Think of it as a triangle like the food pyramid where the vertical axis is the time spent playing your game and the horizontal axis is the amount of people playing your game. Tissue testers are the lowest and you should have the most of them. Focus testers are the middle ground and Beta Testers are the fewest, but spend the most time playing it.

Hope it helps!

[Michaël Samyn]

In my -little- experience, most testers find the same problems. If resources are unlimited, I imagine that having many testers is still beneficial. But in indie game development I can also imagine this rarely to be the case. We don't have the time to take weeks or months out of our schedule to test with several dozens or hundreds of people. And we don't have the money to hire a company to do that for us.

"Luckily" for us, there's so many glaring errors in our game that anyone who looks at it for five minutes can give us two days work debugging.  :D

[bateleur]

"Luckily" for us, there's so many glaring errors in our game that anyone who looks at it for five minutes can give us two days work debugging.  :D

Oh, that's not just you! Indeed, if any game goes into testing and isn't full of bugs I'd be tempted to argue that it means testing started too late (you don't want to be in a situation where testers are complaining about gameplay and you can't fix it because you're out of money and have to release in a week).

[Michaël Samyn]

Ok. We have done 12 test sessions by now. We've reported on them on our blog. It was very very interesting!

They turned out to be playtests more than anything else and we ended up changing a lot of interface elements in the game and even adding a few things to play with. It's very interesting to design when you know you will be able to test your idea the next day.

We had originally scheduled an "authoring phase" at the end of the project, to finetune the gameplay, but to some extent, the testing phase has replaced it. And probably for the better because at the end of the project, I find it very hard to decide whether certain ideas are good or not. Simply testing the idea on a random person immediately gives a clear impression of the benefits and disadvantages of the idea.

I wish we could start working with testers much earlier in the project. But, as we have experienced as well, the game does need to be finished to get good results. Some parts of The Path were not finished yet at the time of the tests and this lead to some players not understanding or appreciating certain aspects of the game. Which in turn lead to doubts in us about the design: would they get it if part x or y were finished or is it just a bad idea?

Each session took several hours. And we ended up spending most of that time looking over the shoulder of the player. So all together, we have witnessed some 50 hours of gameplay and gathered many more pages of notes. Next to design improvements, we also found a lot of show-stopping bugs in the process. But we still consider this our alpha-test phase. Next month, we will do beta-testing with remote candidates who will be asked to install the game on their own computers.

[salade]

Any tips on whether it should be a closed beta test vs. an open public build??

[Michaël Samyn]

For playtesting, I'd definitely recommend a closed test in situ where you can watch people play. For bugtesting on our current project, we're going to try and work with a small group of selected people who commit several hours to the job. But we're not going to do an open betatest. That seems more suitable for multiplayer games or for marketig purposes.

[ஒழுக்கின்மை (Paul Eres)]

I think this also depends on the nature of the game, and whether you care about hype. Having an open beta removes the sense of hype about the game, since everyone can go play it anyway, they won't be excited about release day. Imagine if Aquaria had used an open beta, it probably wouldn't have had so much excitement about it and wouldn't have had so many downloads on the first day.