Cookies help us deliver our services. By using our services, you agree to our use of cookies. Learn more

Tech

Flox: Creating a Backend for GameMaker (Pt.3)

Posted by Mark Alexander on 3 July 2015

This weeks Friday tech blog is the third (and final) tech blog written by our guest writer Ryan Loader, and follows on from the one we posted last week.


The trouble with persisting data, is the finality of it. If something goes wrong when you save data to disk then it’s messed up, restarting the game doesn’t fix anything. If it isn’t human readable then it can’t really be fixed by hand either. Worst case you store it somewhere that isn’t cleaned on an uninstall and you’ve managed to completely break that game for that device.

Last time we discussed some of the challenges of transmitting and receiving data with our rest service, this time we’re going to talk about what happens when you try to save that data to the device, cut out needless requests by caching data and why I wrote an entire wrapper library around the ds* functions! Let’s get stuck in!

Why Persist?

It really sucks having to login every time you open a game, it sucks so much that virtually no game will ask you to do it. But how do they know it’s you? Well they saved who you are of course! This was a must-have feature for the GML port of the Flox library, not just for saving the current player but for reducing the amount of network traffic too! Here’s a simple example...

Say we are trying to load the current players saved game, so we send a GET request to “http://www.flox.cc/api/entities/savegame/123456”* and we get the following response:

{“id”:”123456”,”type”:”savegame”,”entity”:{“level”:1,”lives”:3, etc. }}

*Note: I have shortened API endpoints for the sake of simplicity, the actual URL would look more like http://www.flox.cc/api/games/:gameid/entities/savegame/:saveid 

Now just think, every time the player starts the game they must wait to send the request, the server must receive the request, it must fetch that information from a database and then finally encode and send the data back, a whole lot of work that happens before the game even starts! What’s worse is chances are the data hasn’t changed since the last time the player saved it, why do we need to reload?

Here’s the trick, every time we GET information from the server, we will store it with it’s path inside a map.This way, when we next try to get information from the server, we can check the map and simply get a previous response that was received. The map will end up looking something like this:

{
    “/entities/savegame/123456”: “{“id”:”123456”,...”,
    “/entities/savegame/789123”: “{”id”:”789123”...”,
}

So that’s each response stored against the path that it was retrieved from. Nice and simple right? So now if we are going to make a new request, we do a quick check:

var apiPath = “/entities/savegame/”+saveId;
var cachedResult = noone;
if dsmapexists(cacheMap,apiPath) {
    cachedBody = dsmapfindvalue(cacheMap,apiPath);
}

And we can retrieve our previous response. So as you can see the caching is very simple, but now we have one other problem. We could just use our cached version of the response every time, but what happens if the player logs into another device and updates their save? Now there is an updated version of the game save on Flox,  how do we determine if our cached copy of the savegame is up to date?

The answer actually turns out to be simpler than you’d expect. There is a wonderful HTTP header called ETag and this is used as a “fingerprint” for resources on a server, when we get got our savegame loaded from Flox, it gave us one of these fingerprints. If we save the cached version of our savegame with the ETag, then it will allow us to tell the server what “version” of the savegame we have using the If-None-Match header. Sound complicated? Bear with me! I’ll give you an example...

So our entities are loaded with a GET request to the server, each time we pull the ETag out of the headers and save it in our cache like so:

{
    “/entities/savegame/123456”: “{“id”:”123456”,...”,
    “/entities/savegame/123456-etag”: “686897696a7c876b7e”,
    “/entities/savegame/789123”: “{”id”:”789123”...”,
    “/entities/savegame/789123-etag”: “723be23439123d127a”,
}

Now when we make a request, not only do we grab our cached copy, but we also fetch the etag and add it to our headers map:

if dsmapexists(cacheMap,apiPath) {
    cachedBody = dsmapfindvalue(cacheMap,apiPath);
    var etag = dsmapfindvalue(cacheMap,apiPath+”-etag”);
    dsmapaddvalue(headers,”If-None-Match”,etag);
}

So what’s going to happen? Well the If-None-Match header will tell the server which version of the savegame we have (Which fingerprint do we have?). The server will then adjust its response accordingly. If the savegame has been updated, we will get http status 200 and a body much like we did before, containing all the details of the save game. In this case we overwrite the data in the cache with the version from the server as that is the newer version. BUT if the fingerprints match and the version we have has the same ETag as that of the server, then the server will return a HTTP 304 “Not Modified” response, telling us it’s safe to use our cached copy. The 304 response doesn’t include any body and so is far quicker to transfer than its 200 counterpart.

So now if our savegame changes we will always get the latest version, but if it hasn’t changed then there’s no need for us to download all the information EVERY time the game starts. As an added bonus it wasn’t even that hard to implement! But as I alluded to before, there is a very real danger in saving data and that danger is in corruption.

What Danger?

So while that cache above is a pretty simple system, there are many other layers to caching in Flox. For instance, if you use a flox*queued function and the player’s device is offline, then the request will be postponed until the next available moment it can be made. If the player then shuts down the game, Flox still must remember to make those requests the next time the game starts, so they too are persisted to disk. The current player is persisted, an installation id is persisted… all these things add up to quite an intricate system that must be very tightly controlled.

For instance, early in development there were a lot of places where I would free data structures incorrectly or use a dsmapadd instead of a dsmapaddmap and it would result in very messed up data. Every time this happened I would need to perform a manual clean up on the data, very frustrating! What’s worse is sometimes it would go unnoticed (as I said, it’s an intricate system). I needed some way to both notify of errors and prevent Flox from persisting data once those errors occurred. Enter assertions.

Asserting Truth

Programming with assertions is a way of ensuring that the program has all the intended characteristics in each state of its execution. Very simply, you know something must be true, so you throw an error if it isn’t. Sounds pretty basic (and it is!) but the trick is asserting liberally, you want to make sure that you know as soon as any little thing has deviated from your expectation. Consider the following simple script;

/// scraddplayer_data(inst,data)

var player = argument0; var data = argument1; player.data = map;

Say that later we were expecting to use that “data” the player has:

/// scrgetplayer_name(inst)

var player = argument0; return dsmapfindvalue(player.data,”name”);

So now if we get an error that the map player.data doesn’t exist, how do we know where that bug has come from? We could have passed scraddplayerdata an invalid map, we could have since modified the property on player, or we could have destroyed the map after it was set to player.data… there’s too many possibilities. This is where asserts come in handy:

/// scraddplayer_data(inst,data)

var player = argument0; var data = argument1; assert(dsexists(data,dstypemap),”Data must be valid”); player.data = map;

Now we know for sure that the map added to player is valid and so we can narrow down where the bug is in our program. Asserts make your program easier to change too, because again if you make changes that introduce new bugs you find them much faster.

You may wonder about our scrgetplayername script; didn’t that script also depend on the map existing? or what if we pass a player that doesn’t exist to scraddplayerdata? You might be tempted to assert these too, but I’m not going to. Why? Because these cases will throw an error! You can’t access the data property of an instance that doesn’t exist, so we will know as soon as the script is misused that there’s a problem. Asserts help us find errors, we don’t need them if the errors will come to us.

Debugging Data Structures

Now that we can find what problems are occurring, we need to know how the game got to that state. This sounds pretty simple, but can actually get very complicated. Data structures use real values that serve as ids that you provide to the ds* functions. For example, the first map you create is map 0, the second map is map 1 etc. One optimisation that was introduced a while ago now was the reuse of ids. So say you have maps 0,1,2,3 and you destroy map 1, the next map, rather than being map 4, will be map 1. Most users won’t even notice this change, but it was a massive problem for Flox as it uses and reuses a tonne of data structures. Let me illustrate the problem here:

player.data = dsmapcreate();
dsmapadd(player.data,”name”,”RaniSputnik”);
dsmapdestroy(player.data);

var anotherMap = dsmapcreate(); if (dsexists(player.data,dstypemap)) { var name = dsmapfindvalue(player.data,”name”); }

So this simple logic quickly becomes pretty complicated. Rather than the dsexists failing as we’d expect, it instead succeeds but the name isn’t retrieved because now player.data refers to “anotherMap” - not the map it originally pointed to. However! If we have deleted a map earlier, then “anotherMap” may not take the place of player.data and so the dsexists function will return false. See how this gets messy?

This system can be managed so long as you follow one simple rule; every time a data structure is destroyed, every reference to that structure must also be destroyed. I decided I would need extra tools for debugging and that became a wrapper library for data structures.

I worked to replace all the ds* functions I used with new map* and list* equivalents. These functions worked almost identically to their native counterparts however it allowed me to do things like:

  • Give maps a name using mapcreate(“Player Data”). I then used another map that recorded metadata about the map (such as it’s name) next to it’s id. When I was debugging I could print out contents of the maps along with their names to better see exactly what the map was used for.
  • I could turn of mapdestroy by simply commenting out the dsmapdestroy function. Then no maps would reuse ids (because no maps would ever be destroyed) and I could better see how my data structures interacted.
  • I could add a couple of useful helpers like mapdefault for getting a value then using a default if it wasn’t found, or mapset which would automatically pick add or replace depending on whether or not the key existed. Not critical for debugging, but very handy nonetheless.
  • I could ensure that dsmapfindvalue() returned an invalid value (such as -1) rather than 0 (a valid map id). 

The intention with this framework was not to replace the ds* functions but was to help me ensure that the code I was writing did not have any sneaky errors. It’s certainly performance intensive, but once the code is working flawlessly, then I can worry about optimising it. Data structures are simple in principle but can get complicated in practise, I soon hope to release my map* and list* wrapper libraries and my assert* suite so others can also use these useful debugging tools. In the mean time best of luck, I hope this helps you track down some of those nasty errors you’ve been wondering about!

Cheers

So that’s just about it from me, but I just wanted to take a few words to thank YoYoGames for the opportunity to write for their tech blog. I have really enjoyed (and learnt a lot from) the articles that have been published previously and it’s awesome to be a part of the community and pass on a few tips and tricks of my own. Thank you also for GameMaker, it’s a staple of my career and I don’t think I would have realised how fun game making can be without it. 

Until next time, happy game making!

 

FloxGM is on the GameMaker Marketplace and is available for free


About The Author

Ryan is a game maker / designer / programmer from Wellington, New Zealand. He started a company with two friends, One Legged Crab and together they made games. One of these games was Boy Goes to Space. Ryan has a passion for education, teaching kids how to program and students how to pursue their passion; he believes that teaching is the most effective way to learn! He also loves Burger Fuel.

Ranisputnik on the GameMaker Community
<a href="https://twitter.com/RaniSputnik" target="blank">RaniSputnik on Twitter
http://ryanloader.me 

Back to Top