Testing happens in every videogame studio and the more testing you can do, the better. The function is so essential to a studio’s day-to-day operations that it’s easy to take it for granted. From designers to developers, engineers to marketing folks, Bungie team members are often invited to playtest sessions. You show up to the playtest lab, put on the headphones, play for a while, and then share your experiences with others. That fundamental process of play, feedback, and communication is part of the lifeblood of a game studio.

It's so essential, in fact, that the alternative – developing games without that rigorous testing – is unthinkable. So what happens when all of these time-tested and finely tuned processes get upended by something like the COVID-19 pandemic? How do you continue to carry on such an important development practice when everyone is working remotely?

As Fox and Duncan will tell you, you get creative.

Before we talk about adapting to new circumstances, let’s paint a picture of a typical test at Bungie. Like most game studios, Bungie has playtest labs where developers can gather on regular basis to test a build of a game. As you might expect, these labs are constantly busy, with lots of teams looking to book time to get a session in and to share feedback with each other about how a particular area of development is going. Maybe it’s the scenario team who is testing out the difficulty of a particular mission, or the audio team making sure that the weapon sounds are where they want them to be.

In general terms, testing can be broken up between traditional QA (quality assurance) testing and playtesting. Traditional testing is about analyzing and validating design and engineering implementations. Are there bugs or glitches that need to logged and fixed? Are things working as intended? On the other hand, playtesting is about the intended experience for players. Does an event have the desired effect on the player? Does it convey the right mood? Is it fun to play?

As playtest coordinator at Bungie, Duncan’s role is focused on organizing playtest sessions across the studio. “[It’s] a very hybrid role,” said Duncan, who has been with the company for almost three years. “It’s more akin to a lab manager. We have to do a lot of similar to IT work, in terms of troubleshooting [things] like hardware and software issues. We have the test background, so we know how to debug through things, how we get audio up and working.” There’s also the organizational aspects, working with teams to decide which content is going to be tested, which team members will attend the tests, and more.

A typical busy week at Bungie HQ will see the playtest labs being used every day, with multiple sessions per week, scheduled and set up in advance by Duncan and his team. It’s no wonder then, that the playtest labs are some of the busiest parts of the studio during a normal week.

Enter the COVID-19 crisis and the idea of “normal” has been thrown out the window. In late February, Bungie began a massive effort underway across all parts of the studio to gear up for an extended period of remote working. That meant everyone would be leaving the studio… including testing. No more playtest labs, no more in-person sessions to play and discuss. In short, things were changing.

Facing the Problem

“It was the whole stages of grief,” said Duncan, when asked about his initial reaction to the news that they weren’t going to be allowed in the studio any longer. “There was definitely some denial at first. My team specifically was like, ‘Oh, we don't need to be home. There's a lot we can do in the studio.’” 

Initially the team talked about using the potential down time as an opportunity to tidy up and make some improvements around the labs.

“[We thought] we could do things like improve the hardware, do a lot of the manual labor that needs to be done to get re-organized,” said Duncan.

At the same time, the team was already aware that a pause in playtesting wasn’t going to be acceptable. “We [were] thinking about how do we help teams playtest from home,” he said. “We were doing an exploratory process [asking], ‘What are our different options? What is viable and not viable?’”

The complications of remote testing while working from home pile up quickly. There’s the basics of making sure that everyone has a powerful enough machine to play on (in the case of PC testing). Then there are the inherent security risks with removing expensive development kits from the studio. And with internet connections being what they are, it’s unreasonable to expect remote testers to use their home bandwidth to download a new build of the game remotely each time they wanted to hold a testing session.

With these restrictions in mind, how do you continue the rigorous testing schedule that is one of the keys to shipping new Destiny experiences on time?

The answer, it turns out, was in the cloud.

Enter Stadia

Jeff Fox has been with Bungie for seven years, working as a test lead. Alongside Duncan and others on the Test team, Fox has helped test a huge variety of Destiny gameplay, including activities, matchmaking, networking, and more. His most recent project was launching Destiny 2 on Google’s then-new Stadia streaming service.

“It was definitely unique because we’d been historically on traditional platforms – Sony PlayStation, Xbox, PC,” Fox said. “Going into this weird new streaming platform in the cloud was a unique challenge. It was also really exciting from that perspective as well; once you’ve been in QA long enough, everything is kind of very similar. So getting a fresh new perspective on a different platform opens up new challenges [and] it’s always great to move into that.”

By the launch of Destiny 2 on Stadia, Bungie developers and testers were becoming more familiar with the platform. That experience, combined with a workflow that was designed for ease of use, and it wasn’t long after the studio-wide “work from home” orders were issued that the idea of shifting a chunk of all-up testing onto the Stadia platform came up.

“Using Stadia in the ‘work from home’ transfer seemed like the easiest thing we could have done, and the fact that we already had our game stood up on that platform made it kind of a no-brainer to start looking into that,” Fox said.

Whereas a traditional test session is preceded by a relatively lengthy process of propping a build onto multiple consoles or PCs in the testing lab and working through any technical snafus that may crop up, testing on Stadia was a relative breeze.

“On Stadia we can publish a build in a way that all of the instances we use automatically get the build distributed to them at the same time,” said Fox. “We're able to very easily get a pool of up to 300 instances or so with the game ready to play at a click of a button, which is fantastic. You can't do that any other way when we're running a big studio playtest like that.”

Testing also places a high level of importance on uniformity of setup – in a playtesting lab, everyone is using the same equipment as much as possible. As Fox pointed out, this is even easier with Stadia. “It's all cloud-based,” he said. “There is no physical hardware in studio. You can use a variety of compatible controllers. The best thing about developing Stadia [is that] there's literally no hardware at all on the desk. It's all in the cloud so we didn't have to worry about that at all.”

While playtesting with Stadia has its distinct advantages – ease of setup for both players and coordinators, hardware uniformity – it’s taken some work to get there. That’s in part because traditional playtesting success is often the result of long-standing rituals and routines. The teams schedule a test session, the coordinators work to get the lab set up with the correct build, and everyone knows the process of where and when they need to be at the lab. It’s a scripted routine that is the result of a lot of learning over time.

Moving to a new system requires new communication paths, and new rituals to form, not to mention the very real changes that have come when an entire studio is learning to function remotely. It’s something that Bungie’s Test team is still focusing on as the weeks go on.

“In terms of stabilizing and having a normal day to day, we've just about got there,” Duncan said. “[The] first thing was, let's get teams playtesting again. Then we started to transition to wrapping our old team rituals or processes back into the Stadia thing.”

While there’s still more work to go, the team is feeling good about the progress made so far and the potential this kind of cloud-based testing has for Bungie. That’s in no small part due to Bungie’s willingness to adapt, said Fox.

“I was surprised at how quickly people were able to pick up on the process,” Fox said. “Generally when you say, ‘Oh yeah, we'll stream it over the network. It will be fine!’ people are [going to be] pretty skeptical. But the overall feedback has been really good. This is working for us now during work from home.”

Duncan is even more bullish on the future.

“This is going to change everything,” said Duncan. “We are in a new tradition. Just because we used to have every team playtest in labs [in the past], it doesn't mean we stop doing that. But the process is evolving and changing. And we need to continue to be flexible.