So what brings us to the topic of beta sign ups? Vanguard's recent announcement of their beta 1! Does everyone see me jumping for joy? My past thoughts on Vanguard:SoH.
Lets start out by pointing out what they did correct with this beta phase. First off they are offering it to community members and not a general sign up for everyone with a net connection. That is the correct way to reward your community. Hopefully they don't just pick Joe Schmoes that signed up just to apply for beta, but I doubt it. Well... that ends the good.
The bad? Where to begin. I guess with what "grinds my gears" the worst. Over complicating the process and giving false hope. Here is a perfect example.
"How will you go about selecting participants?So now every new beta application is going to be over stating what hours they play or the person submitting it will try to *guess* the *magic combination* of inputs to produce the highest % chance of getting into beta. Congratulations Sigil; you just flooded your beta application pool with a bunch of false information.
Once you have submitted your application, your name will be added to a list of potential participants. Each week, we will add the needed number of people from that list. Sometimes, they will be chosen based on our specific needs for testing (for example, we might need more testers that play during a particular time of day) and sometimes, names will be chosen pretty much at random."
But why even have such a process in the first place? If they honestly think they are going to get any sort of actual *testing* (I use the term loosely) from an over-hyped MMORPG community… they obviously failed basic MMORPG sociology. I could link hundreds of beta leaks and broken NDA contracts, but what would be the point?
What you need to know is the fact that betas are infiltrated by those that want sneak peaks at the game. Definitely not by those that truly wish to test the product. Internal testers and paid testers have proved for years to be able to produce very finished products in the single player market. Apply this to the MMORPG market please! I understand some smaller companies would be unable to fund such testing, but this is Sigil and last time I checked they weren’t short on funds.
Sigil is using this beta as a promotion which is a sad fact. Not only that, but they are overcomplicating the process stealing valuable resources from the game itself. How much work is it to review countless beta applications? I have no solid numbers, but there is no way they can convince me that it doesn’t take away from the game development.
The idea of NDAs is also hard for me to understand. World of Warcraft had no problem without one. Blizzard proved exactly what betas are for… a free pass to view the game. This created a knowledgeable community inside and outside of beta. WoW beta only suffered from too much interest, but Blizzard did a remarkable job of eventually getting 500,000 testers online.
Skip the bullshit Sigil and get to stress testing. Nail down game play, get the game launch ready, and hit the stress test hard. Sigil will be balancing this game as any other MMORPG… over time! If you get the game into a state where it is playable, has a sense of balance, and technically stable… you have a beta. Take all the time it takes to review beta applications and put into reviewing and organizing bug reports.
This isn't about me getting into beta at all, because honestly I have no interest in Vanguard. This is about developers wasting time, resources, and energy on overly complicated beta processes. Make it about the damn game already.
The best way to make a beta would be to make it accessible to everybody, but limit it to one weekend or so. That means people who are interested get a chance to see the game in action, thus making them want to buy the game. But they don't get to play enough to make them already bored of the game. Of course that requires that the game is good enough in the first place that playing it for one weekend doesn't already turn you off. :)
ReplyDeleteOpen beta tests are good for server stress testing. Companies don't even listen to bug reports given about released games, so what would make you think they could use bug reports from beta testers?
I don't know... this vague hope that my /bug reports in WoW had some sort of effect in beta :P But alas... I'd rather know they are spending times on other things than sitting in circles looking at beta applications.
ReplyDeleteI wish we could quantify how much WoW saved in development costs with no NDA and such an open beta.
Gameplay balance should be nailed down in closed testing IMHO. The excuse "that 100,000 testers are required to get all the balance bugs" is horse crap in my book.
Guild Wars had the beta weekends, but that isn't the consistent play that is needed to test server bugs. Maybe a system where bug reports directly influenced whether you got to stay in beta.
Stress testing is about the only difference between testing a single player game and an MMO.
This is in response to your original post on Vanguard.
ReplyDeleteHeartless... I think you are being a bit too pessimistic.
Sure you have every reason to be acting that way. With huge failures in the past of almost every hyped MMOG.
I had never looked at Vanguard until I read your post.
From what I can tell in the features you quoted, it doesn't look to be your 'standard' MMOG.
They are honestly trying to create something worth playing. Something that draws people in.
Sure on the face of it, forcing extended downtime to build up 'social' interaction seems like a pretty stupid idea.
However they seem to have a lot of 'interesting' ideas in what you quoted, and perhaps, just perhaps when these ideas work together they might make an engaging game.
Reminds me of a MUD slighty in fact :)
Anyways, I'll try and get into a later beta to try it out, or if that fails I'll wait for the NDA to be lifted and read some posts from those that played before making up my mind.
I'll be glad to eat my words, but as you've guessed I am not hardcore and WoW leveling 1-59 spoiled me on ever going back to where I was in DAoC three years ago.
ReplyDeleteI appreciate your fair comments though.
Beta testing isn't about promotion, although many games in recent memory have used it for exactly that purpose.
ReplyDeleteBeta testing is about finding bugs. You might also remember that WoW had closed beta, with NDA's for several months before the open beta period.
WoW certainly did have NDAs. I was testing it from the start, in Friends and Family Alpha. The only NDA less phase was the last, stress test.
ReplyDeleteWoW also used betas as promotion. So did DAoC, Horizons, Shadowbane, Irth, EQ II, Eve, Planetside, SWG, gmail... with varying effects as you can tell if you are going "What?" to any of the items I listed.
Your as jackass
ReplyDeleteI'm glad someone can spell around here.
ReplyDeleteAnyways... yes betas are promotions. I get that fact and as I have said... the initial testing is best done by profesionals... not random newbs you pick off the internet.
The time and resources used to select 100 or so testers from an application pool in the 100,000 range is a pure waste.
Developers can't afford to pay in-house testers to find every possible bug in an MMO. MMOs have exponentially higher variables of things going wrong and even more dire consequences if something does (X players, X player types, X spells, X attacks, X transactions, NAT setups, etc).
ReplyDeleteThe main thing that developers get out of beta testing is as many people playing as possible, with as many different network and OS setups as possible. Basically, they get most of their hard questions answered in the first week (without any bug forms). And speaking of bug forms, all that the developer is looking for is the technical bugs: server, connection, and crash issues - NOT balance or design bugs. Designers won't adjust balance based on any gamer's first impression of a system, even if everyone clamors in the forums about it. And if they DO change something, it's only because they were going to change it already, not because of a bug report from a beta tester.
This guys is clueless, its amazing that idiots like this get attention.
ReplyDelete"MMOs have exponentially higher variables of things going wrong and even more dire consequences if something does (X players, X player types, X spells, X attacks, X transactions, NAT setups, etc)."
ReplyDeleteWrong there is the same potential in each and every RPG or sports title released.
The only area that is different is the fact that thousands of users concurrently use an MMORPG. That is where stress and load testing becomes important.
"Wrong there is the same potential in each and every RPG or sports title released."
ReplyDeleteThe others are right. You are clueless if you think a single player game is as complex as an MMO. Either you have never played one or you have no common sense.
Also, the beta selection in no way takes time away from development. You think the devs themselves do the beta selection?? Again.. clueless. The community reps handle this going by guidelines from the devs. That is all.
Internal testers cannot test every aspect of an MMO. They will always miss something that a veteran MMO player will find. That is why there are outside testers for MMO's.
I don't know where you get your ideas about MMO's from but maybe you should stop posting about them until you actually learn a bit more about them
"You are clueless if you think a single player game is as complex as an MMO."
ReplyDeleteOK prove it. How many plays, formations, teams, stadiums, weather conditions and so forth are programmed into every Madden title? An in house testing team at EA has no problem getting that game out the door.
Take a standard RPG like Baldurs Gate. How many NPC dialogues are there? Often the combat is far more advanced than any single MMORPG will ever hope to achieve. Yet the games require no 100,000 people to beta test and make it good to go. The software works properly on just as many (if not more in most cases) PC setups.
Your whole arguement is null & void. The only thing an MMO tests that is different than a single player game is putting 1-3,000 people on a server together.
If you think the core mechanics (not game balance) are more advanced than any other game on the market you are sadly mistaken.
Now game balance is something a singleplayer game rarely hears about after launch... well because if there is an overpowered ability that lets someone beat the game... it only effects them. That is why balance in an MMORPG is so much more significant and is a continious process.
And again just because you throw thousands of players into a beta test it doesn't mean you get any useful data out of it.
You're an idiot. This is beta 1. You need to test things in order to get to beta 3 or 4. It's in those later phases when they can open it up to a wider audience. Most features aren't even implemented yet. Do you really think it would be good publicity to open it up to the public when half the game is complete?
ReplyDeleteWoW didn't have a stress test until their last phase of beta. What the hell are you thinking?
I've been a beta tester for various games, both paid and unpaid, for 7-8 years now. I'm not sure of your reasoning here, but what Sigil are doing is nothing new. In fact, this is how beta testing always used to be carried out. It's just the recent waves of games, by the likes of Sony and Blizzard that have used it for promotional purposes, rather than testing.
ReplyDeleteI actually think what Sigil are doing here, is a step (back) in the right direction. All to often these days we're seeing buggy unfinished games, due to the publishers allowing anyone with an email address to participate. It was good to see Sigil actually making an effort to ensure the game gets tested. Good to see they are also considering to interview people over the phone, that should stop alot of those 'random noobs' from getting in the beta.
Not sure about the time and resources being used to select the testers. As far as I can tell MGS User research was dealing with that and not Sigil.
I'm not trying to build Sigil up or anything, but we'll see if their testing strategy works when the game is released next year (lets face it, it couldn't be any worse than EQ2, WoW or SWG).
See we are just on two different sides. You are for pulling testers off the net through a good interviewing process and I am for using profesional testers and letting the stress test be handled by the net people.
ReplyDeleteIf it had to be over the net selections... then yes I would want a strict interview process and application review for them. I just believe that is a waste of time, money, and resources.
Heartless, your lack of research makes you appear ignorant.
ReplyDeleteFirst of all, there are only about 23k people registered on the Vanguard forums at the moment, so your "100k" figure is a little exaggerated.
Secondly, I don't believe that their screening process is nearly as time consuming as you assume it is. Having actually filled out their application, I can honestly tell you that most of it can be sorted by a database for easier screening. For example, if they want to select people who already have experience beta testing games, they can immediatly pull up the applications of those who listed a number of games they've tested on their app. Then they have exponenially fewer applications to work through. They can also filter through applications using many of the other extensive questions they asked on their form.
I will agree with you, however, on the fact that people may bs their way into beta by telling Sigil what they think may give them the best shot of being selected. We can only hope that the CS department has a good enough bullshit detector to weed out many of those who did so, and stick to those who honestly want to help them release a great game.
Any amount of time is more time than needed in my book.
ReplyDeleteI put the 100,000 number up... well because I was covering more than V:SoH. Still 23,000 is still a haughty number to sort through for 100-1000 testers.
I just don't agree with the effort required to provide such a beta... when you can make a simpler process to get stress testers after an outside party has tested your software.
I can understand what you're saying Heartless, and it does seem like a hefty number, but I agree with an above poster, the application allows for easy sorting of applications, couple that with a third party company handling the initial stages of the application process, and its really not that bad of a task. Hefty? Yes, but not unmanageable.
ReplyDeleteAlso, as far as paying testers. The world of Vanguard, and any MMO for that matter, is huge. The cost to hire professional testers would be rediculous, and I imagine Microsoft game testers are already hard at work putting the game through the motions as it is anyways. But to get the kind of content testing a game of this proportion needs, you would need either alot of time, or alot of paid testers, either way thats alot of money.
By opening up the doors to qualified community members, they save time and money, which is a very important aspect of the design process.
While I certainly understand your wish for paid beta testers, it's not realistic in an MMO. Firstly, it would either be far too costly to have the required number of testers or take way too long for a lesser number of testers. Lastly, this isn't going to be a game you play at home by yourself. If there is a bug, it doesn't just affect your gameplay. If you want to shoot for foolproof, you'd best have some fools testing it.
ReplyDeletegetting paid testers to test whole MMO game is impossible. They are good to hunt bugs one by one, to test certain areas on the game. But when you have game to hold 5000+ user on one realm, you need these 5000 users, for long term testing. To see how it all works togthether, how the economy will work. We're not taling about stress testing, as stress test are for a short period of time, certainly not enough time to see if economy in all areas is fine tuned. That require several months of testing. Now consider getting 5000 paid testers to play the game for several months.
ReplyDeleteThe best how to select testers for that task is to wide open signups, and allow people completly random to enter, some will not even begin testing, some will get bored in a week and some will test. When you invite more and more testers in each phase you can keep the server filled with people, will all levels... so simulating normal game run.
Second with "by net people" testing you can have also compatibility tests on almost hardware configurations for free. Consider internal testing with compatility testing of all possible HW combinations, thats what i say is resource wasting, while you can get it for free.
...
Companies don't even listen to bug reports given about released games, so what would make you think they could use bug reports from beta testers?
-> they do, if a bug is discovered, a serious bug they can eighter fix it or let people abuse it, or let the bug ruin the game for people... so fix or loose people, that makes them fix bugs. For companies not lisening to retail games bugs, well, most of them dont listen, and most of them dont have public betas for their games. Mostly SP games, or not too popular games. You brought it, they get your money... they dont care anymore, but those dont do public betas.
NX