Out of sync: Adventures with gameplay footage

in , , ,

First of all, it's nice to be back. I didn't realize it would be that long between posts, but here we are.

I'm working on a new research project that will look at the game console as a socialization hub. That's all I really know for now because data collection just started and I'm having a blast. What is less fun is trying to figure out all the devices, setups, cables, wires and systems that are needed for this to work. I thought I had this all figured out, especially after working with some of this equipment for my dissertation, but the technology keeps changing and we keep finding new ways of complicating things.

This was my setup for my dissertation:

  • Nintendo GameCube / Wii
  • VCR (forgot the brand; I think it was a Panasonic)
  • Digital audio recorder (Olympus WS-200S)
  • Stereo mics (Olympus ME-51S)
  • Analog to digital converter (ADS Pyro A/V)
  • Digital cameras (forgot the brand; borrowed from college)

I was recording gameplay and player chat while they were playing Nintendo GameCube and Wii fighting games (Super Smash Brother Melee and Super Smash Brothers Brawl.) I was mostly interested in the audio data so I had a digital audio recorder and microphone positioned next to the players. I had a video cassette recorder hooked up to the TV to record the gameplay. I also had a digital camera set up behind the players to record their offscreen actions.

There were two small problems. The first problem is that, if you are filming players using a CRT television, the screen will flicker because the TV and camera have different scanning frequencies; that was one reason I recorded the gameplay onto a VCR. The other reason is that, even without the flickering, the camera footage of the game isn't really helpful, no matter how you position the camera. If you're trying to do any meaningful analysis of gameplay, you'll need more than a camera. That was why I recorded the gameplay using a VCR.

The second problem, which is a little bit more annoying, is that when you record something, digitally or otherwise, you're not really recording it exactly as it happens. I found this out the hard way when I tried to splice the player chat with the gameplay footage. The Pyro A/V converts the VHS tapes into digital files, which is infinitely more durable (unless your HD crashes; and yes it did crash on me once so backup your files like there's no tomorrow.) The game sessions tend to last three to four hours, so there was a lot of data. In my case, I wanted to look at what the players were talking about as they were playing, so I replaced the gameplay audio with the player chat. What I found out was that, no matter how precisely you align the gameplay video and the player chat audio, at some point they will fall out of sync. You won't notice it in the beginning, but if your data is long, it'll fall out of sync towards the end and you have to keep re-sync-ing them.

Here's my setup now:

My research design this time is slightly different. I'm studying a group of players who are connected to Xbox Live. Setting up cameras and capture devices at all the players' homes is not feasible. Besides I was mostly interested in how players talked to each other through voice chat. It's a variation of my dissertation, except moving it to a different level and studying how people communicate and collaborate at a distance. I'm gathering data at home when I'm connected to my Xbox Live, using the Hauppauge to capture the video in HD. Seems like a simpler set up, right?

Turns out there are still two problems. Game capture devices don't capture your own audio. That's true for any gaming capture device, Hauppauge or otherwise. The way the cables are connected through the device means that your audio is lost. You can capture the talk from the other players, which is great because that's what I'm mostly interested in anyway. Still, I'll need to record my voice because I do talk and it won't make sense to do a linguistic analysis of a one-way conversation. That's where the digital audio recorder comes in again. I have to record that separately and then splice them together, a bit like what I had to do with my dissertation.

But the syncing problem doesn't go away. I thought the syncing problem was only an issue when converting VHS to digital. I thought it wouldn't be an issue if I was doing everything digitally to begin with, especially since Hauppauge promises "no lag" as it streams the footage to your TV and to your computer at the same time. Well, that's not the case. As I found out, and as you can see in this Youtube example, the files will eventually fall out of sync if you record long enough. So...back to the same problem, where I have to break up the files and some how make sure that everything is in sync when I do my analysis.

What's the big deal about the syncing? Well, it's a big deal if you're doing any kind of micro-analysis, like conversation analysis or something even deeper. But if all recordings are out of sync, then that means no device can ever capture vidoe or audio data of any kind as it really happened. All these recordings, no matter how much they seem like they're exact recordings, are always a little bit off.

P.S. I have other issues with the game capture. The audio part of my Hauppauge died and I'm not sure whether it's the device or the third party software or the laptop or the cable or what. Fortunately I can just capture the audio out of my TV. In some ways, it's easier because I can record the players' voice and my voice at the same time, but I have to mute my headset mic when I'm not talking or the players will hear their own voices in the feedback. It's a bit of a pain but it's a tolerable workaround while I figure out how to fix this issue.

Maybe someday, it'll be possible to capture all voice data in one setup, instead of having to splice things together in post-production. That'll make my life a lot easier.