Questions about playtesting

There’s no need to apologize.  I wouldn’t have known that English wasn’t your first language if you hadn’t told me.

Let me begin by saying that there’s only so much I can say due to confidentiality agreements.  I’ll try to answer as best I can within the scope of what I’m allowed to reveal.  I’m going to write this as “generically” as possible, as much of it applies pretty much to all publishers, not just WotC.

First, I’d like to correct a common perception that playtesters are paid by the publisher - they aren’t.  Some publishers use in-house playtesters, but as far as I know they are employees of the company that have other jobs and playtest mostly on their own time.  Many publishers do provide volunteer playtesters with a free copy of the game after it’s released.  For those who don’t, the playtesters get nothing for their efforts other than the satisfaction of having had a part in shaping the games.

Volunteer playtesters are generally everyday gamers like you and me, but they are chosen for the most part because of their dedication, knowledge of the games, skill at playing them, or some combination of those factors.  (Before anyone asks, the best way to be asked to be an A&A playtester is to post on Larry’s site and get yourself noticed by making insightful comments.)  These people generally have a local group that they game with, and they are the liaisons between their groups and the designers.

All of the people involved must sign non-disclosure agreements with the publisher and are not allowed to share any knowledge of the game they’re testing until it is made public.  That means they can say nothing about it before it’s released, unless the publisher makes certain information public through previews, etc.  They are also limited in what we can say even after the game is released, in that they aren’t allowed to discuss the details of the development of the game.

As you would imagine, each group of testers has its own size and its own way of doing things.  Some groups will play as often as once a day, but that’s rare.  Most play about once or twice a week.  Since we generally have day jobs and/or families, it’s tougher to do it more often than that.  My group consists of myself, a friend of 30+ years, and my wife (yes, you read that correctly).

Every group has their own style.  I like to make the most of each session, so for A&A games we use a “medium luck” method of dice rolling that I’ve devised.  Its purpose is to reduce wild swings of luck without eliminating them entirely.  Basically, it tunes the results more to the center of the bell curve.  We also tend to discuss strategy freely during the game, sometimes even going so far as to back up a turn or two and try something different if we don’t like the way things are going.  I feel that we get around two to three games’ worth of testing results from a single game this way.  As you can imagine, this is a far less competitive atmosphere than normal gaming, but we learn a lot about the game from it.  Of course, many groups just go ahead and play the game straight up and let the chips fall where they may.

After each game, the group leader writes up a session report and submits it.  This report details the strategies of both sides, the highlights of the game, and its results.  If there are new rules or setup changes involved, special attention is given to their impact on the game.  Suggestions for changes may also be made.  (Of course, there is no guarantee they will be taken!)

Speaking of suggestions, there is usually a pretty free flow of ideas between the heads of the groups and the designers.  Sometimes the designers like what they hear, and sometimes not.  It’s not uncommon for playtester suggestions (or at least some form of them) to make it into the finished game.  However, the designers always have the (mostly) final say (more on this later).

I also want to say a word about another misconception I’ve heard on the internet.  Some people seem to think that the playtesters get to test with the actual game that’s produced.  This just isn’t true, and is kind of silly if you think about.  If the game was finished, it would be too late to test it.  We receive the map either in the form of a PDF that we have to print out or in some cases it’s mailed to us already printed out.  From there, “patches” can be made to be taped onto it if changes are necessary.  We also receive a list of other components necessary to play the game, and we have to scrounge those pieces up from other games that we own.  The rules are in a rough form in a word document, and are obviously subject to change.  Put all of this together, and you have a prototype game for testing.

That’s pretty much all there is to playtesting.  Of course, that’s just part of the process of producing a game.  There’s also design, research, graphic design, production, marketing, and legal.  Those things are beyond the scope of the question, though.  I bring this up mainly to point out that game designers are not in complete control of their own creations, as these other departments all have varying degrees of input into what the finished game will look like.  The extent of that input varies from publisher to publisher.

I hope that answers most of your questions.  Feel free to ask if it doesn’t.  Of course, there will be some things I can’t talk about (specifics, mostly), but I’ll just say so if that’s the case.

@atease:

How many playtesting groups does a game like 1914 have in general? Just a ballpark figure, is is closer to 2-3 groups or as much as 30? The specifics for 1914 I’m sure is protected by your confidentiality agreement, but I’m talking more in general for a big production from a company as big as WoTC.

I’ve seen as few as three for some games, and as many as ten or twelve for others.

@atease:

How long does the playtesting last in general, again ballpark figures for a big production from a company as big as WoTC?

It depends on the complexity of the game.  I’ve seen periods as short as two months and as long as eight.

What reason is given for the confidentiality?

Is it not possible to copyright the game until it is in its final published form, therefore you cannot reveal details to an on-line community which might be able to help playtest?

From my own experiences, I’ve helped test and develop several games (medieval themsed mostly) on-line without any such agreements. This includes games published by the likes of Columbia and Fantasy Flight, so we’re not talking about the amateur end of the market - so why the contrast?

Of course there’s always a deadline which can cut off late ideas and alternatives; if I’d had one more day then the Earl of Devon in Richard III would have the correct Courtenay arms instead of that silly pig…

@Flashman:

What reason is given for the confidentiality?

I can’t tell you.  :-D

@Flashman:

From my own experiences, I’ve helped test and develop several games (medieval themsed mostly) on-line without any such agreements. This includes games published by the likes of Columbia and Fantasy Flight, so we’re not talking about the amateur end of the market - so why the contrast?

Hmmm.  That’s interesting.  Nearly all of the games I’ve ever tested required NDAs.

Krieghund in your playtesting group did you ever notice an advantage for one side over another. It is quite possible that your group saw the Centrals winning a lot or the allies. I understand that your group of 3 players can only come up with so many strategies however over time the community is under belief the allies win 80+ times out of 100.

Also I still have issues about Prussia (I’m waiting for 2nd if one because I’d rather have a finished product) I don’t think that Prussia needs to touch a nother territory against Russia. I think however that it should have extended itself farther along the baltic sea still only touching Poland. This would not change how the game plays out however would give myself more of a historical feel as the Germans would have roughly its correct boundaries.

@italiansarecoming:

Krieghund in your playtesting group did you ever notice an advantage for one side over another. It is quite possible that your group saw the Centrals winning a lot or the allies. I understand that your group of 3 players can only come up with so many strategies however over time the community is under belief the allies win 80+ times out of 100.

I really wish I could talk about this, but I can’t.

Thanks Krieghund for posting this.  It is very insightful.

Two comments/questions:

1. It is frustrating to not get any insight into what you think of 1914 or any game because you are a respected voice here.  I understand that you would not want to speak of such things if you are party to NDAs, it is still frustrating.

2. Ok, so these groups that playtest the games.  Do you get the ‘battle reports’ from the other groups?  Say, one group tries a new strategy that you hadn’t thought of or something?  Or perhaps you misinterpreted the rules and could see how other people play?

Yes, we see each others’ reports.  There’s lots of discussion between the group leaders on strategy and rules.  Of course, the level of detail in the reports varies from group to group.  Sometimes you need to read between the lines and ask questions to see the whole picture.

@Krieghund:

@italiansarecoming:

Krieghund in your playtesting group did you ever notice an advantage for one side over another. It is quite possible that your group saw the Centrals winning a lot or the allies. I understand that your group of 3 players can only come up with so many strategies however over time the community is under belief the allies win 80+ times out of 100.

I really wish I could talk about this, but I can’t.

Can u Answer about the Prussia thing? Like possibly the decision of making Germany not have historical borders?

I’m pretty sure that ws done to put distance between Berlin and Petrograd, perhaps made when Petrograd was still Russian capital?

Guys, he’s not going to talk about that map specifically.

Perhaps a better question is:

How often are map changes made during playtesting and how does that happen?

@Flashman:

I’m pretty sure that ws done to put distance between Berlin and Petrograd, perhaps made when Petrograd was still Russian capital?

I’m not saying change how Prussia is connected to another territory. I’m suggesting taking up some more of Poland North

@italiansarecoming:

Can u Answer about the Prussia thing? Like possibly the decision of making Germany not have historical borders?

Sorry, I thought that was a comment rather than a question.  BJCard is right - I can’t comment.

@BJCard:

How often are map changes made during playtesting and how does that happen?

Map changes are much more rare than rules or setup changes.  Generally they are very small tweaks that are done to either give one side less of an advantage or avoid an exploitation.

How it happens is that either the designers see a trend in the playtest games that something is either too easy or too hard, or that the game would flow better with certain channels opened or closed.  Of course, one or more playtest groups could also suggest a change, which may or may not be implemented.  In some cases, the designers will take a playtester suggestion and implement it in a way that makes it even better.

Thanks for the interesting behind-the-scenes description of the playtesting process.

@CWO:

Thanks for the interesting behind-the-scenes description of the playtesting process.

Shame the contract stops him from saying a lot.