Knowledge is power

Video games are all about having fun. As a developer, you want your players to have a good time. It doesn’t matter whether it’s a crane operator simulator or a fast-paced shooter. You need to create an experience that people will like. The problem is, as the creator you are not a player, at least not a regular one. Your opinion on your game’s features and balance is heavily biased by your experience. You may love a section of the level, or a mechanic just because you’ve created it. You may think an area is easy, because you have already playtested it a thousand times. You cannot impartially rate your own work, but you do need an impartial opinion in order to make it better.

Playtesting

That’s where playtesting comes in. This is one of the most important parts of the iterative process known as video game development. When you can watch someone play your game, you get a lot of important information. It’s even better if you talk to the player afterwards. This is a great oportunity to learn how they understand the mechanics, what are the good and the bad parts of the gameplay.

The thing is, you cannot always stand behind your player’s desk and watch. You cannot always talk to them. More often than not, players are just random people on the other side of the globe who found your game on the Internet. So what can we do about it? How can we get some of their sweet feedback?

The obvious answer is Google Forms. Create a feedback form with all the necessary fields, attach a link to the game. We can even embed the link inside the game, to make it easier for players to find. We did it in the first demo of Bulet Waste. Simple, yet effective.

Except it’s not.

The problem is, no one actually fills this kind of forms. You know it very well. How many times did you left an opinion in an online shop? How about an opinion different than “zero stars, it’s broken”? That’s right. Writing opinions is too much of a hassle, basically for nothing.

If your game is not bad enough to cause allergic reactions, you won’t get a lot of feedback this way. So we are back to square one.

How can we fix that?

Spying on players for fun and profit

As you can guess from the title of this post, the answer is telemetry. Players are for playing, not filling forms. Of course, they need a possibility to provide some written feedback, but it should be optional. We need to gather data about the gameplay automatically. We’ll probably need to ask for consent, but it’s just “you agree, or you don’t play”. It may sound stupid, but that’s how all of the big corporations work. Have you ever read their privacy policies? Do you know what do you agree to? But I digress…

We don’t want to reinvent the wheel. Many games use telemetry, Unity is one of the most popular game engines in the world, so there should be some ready to use solutions, right?
In fact, there are quite a few. The problem is, these are all commercial products. Either pretty expensive or attached to some big cloud providers. We wanted something cheap - possibly free, and flexible - you never know what you will need in the future. In the end we decided to develop it by ourselves, or rather I decided, because I just wanted to build this kind of thing.

Data Waste

I decided the telemetry system has to be called Data Waste. It aligns nicely with the game’s title and I love a bad pun.

To create a useful telemetry setup we need three things:
1) a module in the game that sends data to the 2) server that receives and saves it in a format that can be analyzed by the 3) analitics module.

The game module

Here, we need to create messages, serialize and send them to an endpoint. It’s a simple task. We just have to remember that we should not use the main Unity thread for network communication, as it will block everything for ages.

Implementation goes like this:

1. Create a queue for the messages
2. Create a thread that will send them
3. Put the messages in the queue - every message besides the actual content consists of a timestamp and the player’s id
4. Serialize the data into JSON format with the sender thread
5. Send the message
6. Use C#’s async Tasks and TaskCompletionSources to provide request results to the main thread
7. Smile, you did great

In theory, we could just ignore the responses we get from the server and thus make the code even simpler. But with this two-way communication we can do things like checking if a new version of the game is available, or testing whether the telemetry server still works. The second one is actually pretty useful for debug purposes.

The telemetry server

This one is even simpler. Receive a message, check if it makes sense and if it does, save to an output file.

But why a file? In the begining, I wanted to use a database for storing the results. Mongo or Postgres would be a nice fit for the task. The problem is, we are doing in on the cheap. In fact, we managed to get a server for running our game’s telemetry for free - a Mikrus FROG instance. It’s great, but pretty limited - it has only 256MB of RAM which is not a lot for the OS, HTTP server and a DB. We save some memory by dropping the database. It means we have to do a little bit more complicated analitics setup, as we cannot access the data directly. On the other hand, we don’t need any realtime data processing, so we can get on with that.

Analitics platform

Data you don’t use is only thrash on your hard drive

~probably Sun Tzu

We don’t use a database on the server, but it would be nice to have one in order to handle the data effectively. So for the analisis, we put all of the JSON files downloaded from the server into a Postgres instance (jsonb is great). From there we can read it using Python in a Jupyter Notebook, because this is how all the cool data science kids do nowadays.

What do we collect

Now we have a way to collect and analize the data gathered from the players. But what information do we actually need? “Everything” is not an option, as you will flood your backend with gigabytes of useless stuff. Games really process a lot every frame. We need to focus on important things, but there is no good way to decide what will be useful and what won’t. This is one of the many moments in gamedev when we have to guess.

These are the most important things we collect. Will this list change? Probably yes. Is all of that useful? Probably no. Can we be sure of anything in this universe? Perhaps.

• player deaths - where and why do people die?
• enemies killed - how and how quickly do players kill enemies?
• player movements on the level - do players get lost?
• performance and hardware data - on what setups the game lags?

Results

We have already analized some of the results and started making changes in the game based on that. The next demo of Bullet Waste will hopefuly be much better thanks to all the information we gathered. As for the details, stay tuned for a next post on this blog, where we will describe changes planned for the next release.

If you are curious what results we actually got, I can tell that as for 30th of april (I’m too lazy to check again now), we had around 850 deaths of players in Bullet Waste.

Now go play the game and give us some more deaths to count.

Andrzej “Pojemnik” Gauza