Game Design, Programming and running a one-man games business…

Designing the orders system for ridiculous space battles

Yikes, this has turned into a really complex and confusing beast. I am very happy so far with the improvements, but I need to publicly brainstorm the details. Simply put, the orders system in Gratuitous Space Battles was a mess, and I needed a much better one, PLUS I wanted to add new orders to take into account the new gameplay style of Ridiculous Space Battles.

In GSB there was a whole bunch of orders, and they all apparently pretended to work together to come up with an opinion on which ship a weapon should target. Orders are per SHIP, not per weapon, which frankly is a whole other story, but I think it would be stupid overkill to have the player assigning different orders to every weapon. My whole unique selling point is BIG fleets with LOTS of ships. Anyway, the original game had orders like Attack Fighters and Co-operate, but it was never clear how they all worked together.

RSB introduces three new orders entirely. Raider (race ahead of the fleet up to a given distance to engage enemies) which is used by Fighters, Last Defense, which concentrates fire on the enemy ships that have got the furthest through our lines, and Breakthrough, which is for punching a hole in enemy defenses, and only fires on ships right in front of you (or in that row anyway).

Here is my new current implementation:

So the big things here are categories, color coding and priorities. OH YES. To explain, this ship has one ‘Special’ Order, which is to engage enemies at 2 squares range. Thats a movement order, so independent of all the targeting stuff.

The next order is ‘co-operate’. Its one of a class of (blue) orders I’m tempted to call ‘Discriminators’ or ‘Tie-breakers’. There can only be ONE of these (Co-operate, Vulture, Retaliate, Breakthough or Last Defense), and essentially they are used at the end of the process to choose which out of a series of potential targets is picked.

The Green and yellow orders are Target Criteria. They describe a target size class or a defense status (shields up, or shields down but armor intact, or down to naked hull). These orders are special because they are in priority order. (I realize now I should show the numbers!). So when the ship decides who to shoot at, it first builds a simple list of all enemy ships in range and fire arc, that match any of those criteria. (You can delete one to make the ship NEVER target those ships). Once that list exists, it then goes through the target criteria from top to bottom seeing if any ships meet that criteria…

And this is where I have a decision to make…

I can either stop the processing once it has one or more targets, then pick the best using the discriminator OR I can then keep filtering down through the list as long as I have 1 or more targets and THEN use the discriminator. I cannot decide which. Are you confused? I suspect so:

Example 1: Using the above orders, I scan my target list and find 4 cruisers and 12 frigates in range. All of them have shields. Cool! So I stop there, and evaluate them all using Co-operate, firing upon the one that is the most under fire right now.

Example 2: I scan my target list and find 4 cruisers and 12 frigates in range. All of them have shields. The NEXT criteria is Fighters. There are none in my list so I ignore. The next is armor. All 16 have armor so again I ignore. The next is Frigates. Wait! only 12 are frigates, so I prune the list. The next is Hull, all check. The next is cruisers, none found. I finally use co-operate to evaluate which frigate to fire at…

Is this dilemma even making sense, let alone deciding on a system? Its very involved. My game IS a complex strategy game, but I don’t want to rely on crazily complex tutorials. Now to be fair, Gratuitous Space Battles had a completely opaque system that nobody understood and was poorly explained and it sold a bazillion copies, so maybe I am overthinking it. But I would like to do a MUCH better job… I know SOME players would be happy to literally write code to control this stuff, and might suggest I add conditional clauses and other complexity but I think that makes the game seem more like work and less like fun…

Looking at it now I KNOW I need to add priority numbers to those yellow/green strips. Maybe they should be called Filters now Criteria? Who knows. I may need to think about this a LOT.

By the way add the game to your wishlist :D Or pester a news website to cover it! Or share a screenshot somewhere you hang out. That stuff all helps :D.

Coding a load-balanced multithreaded particle system

Background: I am coding a 2D space-based autobattler with ridiculous levels of effects called ‘Ridiculous Space Battles‘. I code my own engine, for fun, in directx9 using C++ and Visual Studio.

Because I love coding, yesterday I found myself wondering how well the multithreading in my game was holding up. So I fired up the trusty Concurrency Visualizer in Visual Studio. I love this tool and have used it a lot on previous games. One of the biggest demands on the CPU for my game is particle processing. There are a LOT of explosions, and other effects, and a crazy number of particle emitters (thousands of emitters, hundreds of thousands of particles). Obviously this would normally be a code bottleneck, so I have a task-based generic multithreading engine that handles it. The main thread builds up a bunch of tasks, the threads grab the next task on the list, and the main thread will wait until the task list is empty. If there are still tasks in the queue, the main thread will do one as well. So how did things look?

Disastrous!

So what is going wrong here? I have a generic ‘marker’ set up to show the span of the main thread’s drawing of the game (GUI_Game::Draw()). Inside that, a bunch of unlabeled stuff happens, but I added spans to show when the multithreaded task called UPDATE_PARTICLE_LIST is called. There are two massive things going wrong here. Firstly, there are only two other threads joining in to process the particles, and secondly one of those updates seems to take 20x as long as the other. Worse still, its the one the main thread chose… so its a huge bottleneck. Technically this is still a speedup, but its marginal. How have I fucked up?

Some background to my rendering algorithm is needed: The game has 2 blend modes for particles. A ‘Burn’ mode, that saturates color and is used for fire, lasers, sparks etc, and a ‘Normal’ mode for smoke and debris etc. The particle effects are batched as much as possible, but I cannot mix those blend modes in a draw call. Also, some particles are below the action (the ships) and some above, to give a semi-3D look and make it look like the explosions engulf the ships. So this means particle emitters fall into one of 4 lists: NormalBelow, BurnBelow, NormalAbove, BurnAbove. This is all fine and works ok. In action, the game looks like this:

Because you can freeze frame and scroll around, everything has to be properly simulated, including particle effects currently offscreen. Anyway it all works, and I have FOUR particle emitter lists. So naturally, to load-balance everything, I gave one list to each thread and considered the job done.

BUT NO.

It turns out that those 4 groups are not equal in size. They are laughably unequal. The ‘BurnAbove’ list contains all of the fire and spark emitters on all of the pieces of all of the hulks from destroyed ships, plus sparks from fiery plasma torpedoes, expended drone explosions, and missed or intercepted missile explosions. Thats MOST of the particles. When I checked, about 95% of particles are ‘BurnAbove’. I had 4 lists multithreaded, but they were not vaguely really load balanced.

Once I realized that the solution was theoretically easy, but fiddly to implement and debug. I decided I would add a new load-balanced list system on top. I created 8 different lists, and when an emitter was created it was added to the ‘next’ list (the next value circled round through all 8), and told what list it was in. When it was deleted, it was removed from the appropriate list. Note that ‘deleted’ is a vague term. I delete no emitters, they get put into a reusable pool of dead emitters, which complicates matters a lot…

So in theory I now have a nice load-balanced series of 8 lists that contains every particle emitter that is currently live. The original 4 lists are still valid and used for rendering and blend mode data, but this ‘parallel’ list system existed alongside it, purely to handle load-balancing. What this means is, that a load-balanced-list may contains particles from all 4 render groups, but this does not matter as I am running update code on them, not rendering!

It didn’t work.

Crashes and bugs and corrupt data ahoy. I worked on it for ages, then watched a movie to try and forget it. Then this morning, after some digging, it was all fixed. What actually was going wrong was related to smoke plumes. Because there are a lot of smoke plumes, and they always reuse the same particle config data, they exist in a separate system, updated separately. I had forgotten this! And what was happening was my new load-balanced lists stupidly included these emitters when they should have been kept out of it. The emitters would expire and be deleted in the multithreaded code, then later accessed by the plume code. CRASH.

I worked it out this morning before breakfast! I was very pleased. You might be thinking, what about the only 2 threads thing? LOL I had hard coded the game to use maximum of 4 threads, probably as a debug test. Idiot. I just changed it to be 10 and everything worked:

This is more like it. I wasted ages trying to get the dumb concurrency visualiser to show my custom thread names instead of ‘Worker Thread’ but apparently thats the category. Not much help. FFS show us the thread names! (They work in the debugger). But anyway, that image above is a snapshot inside a busy battle for the GUI_Game::Draw() showing how UpdateParticles tasks get spread over 8 threads. I’m still not sure why that sixth thread misses out on a task, which gets nabbed by the main thread…

Anyway, the point is it works now, and in theory updating particles is 8x faster than it would be with single threading. I do need to apply the multithreading to a lot more of the game code to get the best possible results. I am testing this on a fairly beefy GPU and CPU (Ryzen 9 5900X 12 Core @3.7GHZ and RTX 3080) in only 1920×1080 res. I want the game to look awesome at 5120 res or on a five year old cheap laptop, so plenty more to do.

If for some reason this tips you over the edge to wishlist the game, here is the link :D.

Optimizing load times

I recently watched a 2 hour documentary on the ZX spectrum, which means little to people from the USA, but it was a really early computer here in the UK. I am so old I actually had the computer BEFORE that, which was the ZX81, just a year earlier. The ZX81 was laughable by modern standards, and I expect the keyboard I am using has more processing power. It had an amazing 1kb of RAM (yes kb, not MB), no storage, no color, no sound, and no monitor. You needed to tune your TV into it and use that as a black and white monitor. Its this (terrible) PC I used to learn BASIC programming on.

Anyway, one of the features of ZX81/Spectrum days was loading a game from an audio cassette, instead of the alternative, which is copying the source code (line by line) from a gaming magazine and entering the ENTIRE SOURCE CODE of the game if you wanted to play it. Don’t forget, no storage, so if your parents then wanted to watch TV and made you turn it off, you had to type the source code again tomorrow. I can now type very fast… but the documentary also reminded me of another horror of back then, which was the painfully slow process of loading a game.

These days games load…a bit quicker, but frankly not THAT much quicker, especially given the incredible speed of modern hard drives, and massively so when talking about SSDS. Everything is so fats now, from SSD to VRAM bandwidth, to the CPU. Surely games should be able to load almost instantly…and yet they do not. So today I thought I’d stare at some profiling views of loading a large battle in Ridiculous Space Battles to see if I am doing anything dumb…

This is a screengrab from the AMD UProf profiler. My desktop PC has an AMD chip. I’ve started the game, gone to the ‘select mission’ screen, picked one, loaded the deployment screen, then clicked fight, let the game load, and then quit. These are the functions that seem to be taking up most of the time. Rather depressing to see my text engine at the top there… but its a red herring. This is code used to DISPLAY text, nothing to do with loading the actual game. So a better way to look at it is a flame graph:

I love flame graphs. They are so good at presenting visual information about where all the time is going, and also seeing the call-stack depth at various points. This shows everything I did inside WinMain() which is the whole app, but I can focus in on the bit I care about right now which is actual mission loading…

And now its at least relevant. It looks like there are basically 3 big things that happen during the ‘loading battle’ part of the game, and they are “Loading the ships” “Loading the background” “Preloading assets”. The GUI_LoadingBar code is given a big list of textures I know I’ll need in this battle, and it then loads them all in, periodically stopping to update a loading progress bar. Is there anything I can do here?

Well ultimately, although it takes a bit of a call stack to get there, it does look like almost all of the delay here is inside some direct9 functions that load in data. I am very aware of the fact that directx had some super slow functions back in directx9, in its ‘d3dx’ API, which I mostly replaced, but ultimately I am using some of that code still, specifically D3DXCreateTextureFromFileInMemoryEx…

Now I have already tried my best to make stuff fast, because I’ve made sure to first find the texture file (normally a DDS format, which is optimised for directx to use) on disk, and load the whole file into a single buffer in RAM before I even tell directx to do anything. Not only that, but I do have my own ‘pak’ file format, which crunches all of the data together and loads it in one go, which presumably is faster due to less windows O/S file system and antivirus accessing slowdowns. However I’m currently not using that system… so I’ll swap to it (its a 1.8GB pak file with all the graphics in) and see what difference it makes…

Wowzers. It makes almost no difference. I wont even bore you with the graph.

And at this point I start to question how accurate these timings are, so I stick some actual timers in the code. In a test run, the complete run of GUI_Game::Activate() takes 3,831ms and the background initialise is just 0.0099. This is nonsense! I switched from instruction based to time-based sampling in uprof. That doesn’t now give me a flame graph, but it does also flag up that the D3DX png reading code is taking a while. The only png of significance is the background graphic, which my timers suggest is insignificant, but I think this I because it was loaded in the previous screen. I deliberately free textures between screens, but its likely still in RAM… I’ll add timers to the code that loads that file.

Whoah that was cool. I can now put that into excel and pick the slowest loaders…

Loaded [data/gfx/\backgrounds\composite3.png] in 73.0598
Loaded [data/gfx/\scanlines.bmp] in 20.0463
Loaded [data/gfx/\planets\planet6s.dds] in 11.8662
Loaded [data/gfx/\ships\expanse\expanse_stormblade_frigate_damaged.dds] in 10.7132
Loaded [data/gfx/\ships\ascendency\g6battleship.dds] in 9.3622
Loaded [data/gfx/\ships\ascendency\g5frigate.dds] in 6.9765

OMGZ. So yup, that png file is super slow, and my bmp is super slow too. The obvious attempted fix is to convert that png to dds and see if it then loads faster. Its likely larger on disk, but requires virtually no CPU to process compared to png so here goes… That swaps a 2MB png for a 16MB (!!!!) dds file, but is it faster?

NO

Its 208ms compared with 73ms earlier. But frankly this is not an accurate test as some of this stuff may be cached. Also when I compare pngs of the same size, I’m noticing vast differences between how long they take to load:

Loaded [data/gfx/\backgrounds\composite11.png] in 113.9637
Loaded [data/gfx/\backgrounds\composite3.dds] in 208.7471
Loaded [data/gfx/\backgrounds\composite5.png] in 239.3122

So best to do a second run to check…

Loaded [data/gfx/\backgrounds\composite11.png] in 112.8554
Loaded [data/gfx/\backgrounds\composite3.dds] in 84.9467
Loaded [data/gfx/\backgrounds\composite5.png] in 108.4374

WAY too much variation here to be sure of whats going on. To try and be sure my RAM is not flooded with data I’d otherwise be loading, I’ll load Battlefield 2042 to use up some RAM then try again… Interestingly it only takes up 6GB. Trying again anyway…

Loaded [data/gfx/\backgrounds\composite11.png] in 114.0210
Loaded [data/gfx/\backgrounds\composite3.dds] in 85.6767
Loaded [data/gfx/\backgrounds\composite5.png] in 105.8643

Well that IS actually getting a bit more consistent. I’ll do a hard reboot…

Loaded [data/gfx/\backgrounds\composite11.png] in 104.3017
Loaded [data/gfx/\backgrounds\composite3.dds] in 207.8332
Loaded [data/gfx/\backgrounds\composite5.png] in 141.2645

Ok so NO, a hard reboot is the best test, and swapping to DDS files for the huge background graphics is a FAIL. These are 2048 x 2048 images. At least I know that. The total GUI_Game::Activate is 7,847ms. That png is only about 1-2% of this, and it makes me wonder if converting all the dds files to png would in fact be the best solution to speed up load times? The only other option would be to speed up DDS processing somehow. Having done some reading, it IS possible to use multithreading here, but it looks like my actual file-access part of the code is not vaguely the bottleneck, although I’ll split out my code from the directx code to check (and swap back to a png…)

Creating texture [data/gfx/\backgrounds\composite11.png]
PreLoad Code took 1.0205
D3DXCreateTextureFromFileInMemoryEx took 111.4467
PostLoad Code took 0.0001
Creating texture [data/gfx/\backgrounds\composite3.png]
PreLoad Code took 28.4150
D3DXCreateTextureFromFileInMemoryEx took 71.1481
PostLoad Code took 0.0001
Creating texture [data/gfx/\backgrounds\composite5.png]
PreLoad Code took 0.9654
D3DXCreateTextureFromFileInMemoryEx took 105.2158
PostLoad Code took 0.0001

Yeah…so its all the directx code that is the slowdown here. Grok suggests writing my own D3DXCreateTextureFromFileInMemoryEx function, which sounds possible but annoying.

Ok…mad though it sounds, I’ve done that. Lets try again!

Creating texture [data/gfx/\backgrounds\composite11.png]
PreLoad Code took 0.8327
D3DXCreateTextureFromFileInMemoryEx took 103.4365
PostLoad Code took 0.0001
Creating texture [data/gfx/\backgrounds\composite3.png]
PreLoad Code took 0.6053
D3DXCreateTextureFromFileInMemoryEx took 73.9393
PostLoad Code took 0.0002
Creating texture [data/gfx/\backgrounds\composite5.png]
PreLoad Code took 0.9069
D3DXCreateTextureFromFileInMemoryEx took 105.0180
PostLoad Code took 0.0001

Am I just wasting my life? at least I now have the source code to the DDS loader because it is MY code bwahahaha. So I can tryu and get line level profiling of this stuff now… I’ll try the visual studio CPU profiler:

Thanks Microsoft. But there may be more…

The Visual studio flame graph is saying that actually the raw reading from disk of the file IS a major component of all this, and so is a memcpy I do somewhere… Actually its inside the fast DDS loader, so the flame graph is confusing. The DDS loops doing memcpy calls for each line of data. This is very bad. With a big file, there will be 2,048 calls to memcpy just to read it in. Surely we can improve on that? and yet its clear thats what D3DXCreateTextureFromFileInMemoryEx is doing, as seen earlier. Hmmmm. And now people have come to visit and I have to stop work at this vital cliffhanger…

Is this game you designed actually any fun?

When you develop an entire game by yourself, there is a staggering amount of work to do. Coding, business stuff, marketing, testing, balancing, designing. And I think that the majority of people who ‘want to make video games’ tend to over focus on the design bit. The whole ‘I have an idea for a cool game’ bit. It might surprise people to know that this is the bit that I am least fond of. In many ways I am a cross between an AI/Engine coder and an entrepreneur who realizes he has to design games to sell that code inside. The whole ‘working out how the game will play’ side of things has always been hard and frustrating for me.

You might find this an odd thing for me to say for two reasons: Firstly, I’ve made a bunch of (I think) pretty innovative games. Kudos was the first turn-based life-sim game (AFAIK). Democracy was the first commercial game designed around a neural network and based on the aesthetics of infographics. Gratuitous Space Battles was the world’s first auto-battler game. There is no shortage of innovation there. Secondly, not many game developers would ever admit they don’t enjoy the game design bit. Thats the bit we are supposed to excel at right? Admitting you don’t enjoy that bit as much is almost blasphemy.

As is probably obvious, I’m autistic, and one of the ways this manifests is that I like, and even need… data. You can tweak your ad campaign or marketing strategy and see if sales go up or down by 1%. You can re-engineer your code and check that performance has gone up or down by 1%. But game design? How on earth do you know the game is fun? How do you measure if you are making the game BETTER with all those changes… or worse? And in the absence of such data, what the hell are you doing?

I think most fulltime game designers seem insecure, as they are always asking other people if what they are doing is any good! We have to, because its very very hard to tell. In some ways, designing a game is like writing a joke. You can put a lot of effort in, have some skill, lean on prior experience, but by the time you are finished working on the joke, it stopped being funny to you personally ages ago. If you spend your entire day staring at spreadsheets of weapon characteristics until your eyeballs are sore, the question ‘Is this spreadsheet fun?’ feels almost insane. There is a good reason many game designers are NOT avid players of their own games after release. We are too close to it, too aware of the mechanics, too aware of the areas we are not sure about. We saw the sausage being made, and we do not want a sausage sandwich for breakfast.

This might sound a bit depressing, and it would be more so if this was my first rodeo, but I’ve experienced it before as a musician. For probably 20 years, I was unable to just ‘enjoy’ music. I would listen to it from a technical point of view. I might marvel at the clean guitar tone, the incredible timing, the complexity of the arpeggios, but I was listening to it from a teacher and student point of view, not as an audience member. I can now mostly just enjoy music, but I’m still aware of the keys and scales and techniques…

Being ‘too close’ to your own work will always be a problem. You will not be sure your joke is funny, your novel is gripping, your music is cool or your game is fun. Its just impossible for someone so close to the system to evaluate it in the same way a customer would. There are however, ways to get around this!

One is obviously to ask a lot of people. Friends, family, fellow game devs. The trouble is that these people are normally pre-disposed to worrying about hurting your feelings. Not many people will say to me “Cliff, this sounds boring as fuck”, although over the years I’ve managed to find people who know me well enough to be aware they can be more honest with me than other people. Even so, its not disinterested feedback, and if all your friends are game designers too, you are hardly getting a representative slice of the consumer base.

A second technique is time. Take a weekend off, or a week off. Ideally a month off. Some novelists stick their work in a drawer for a YEAR and then come back to it fresh, and can evaluate it with a far better critical eye. Of course the problem here is you need to earn money, but if you can work on multiple games at once and swap them over, this might be an option. Its definitely a system that works.

A third technique is drugs. Yes I went there. I am quite boring in that my narcotic of choice is just good old fashioned alcohol. Its not like I am permanently drunk when designing (am I making this denial too strongly maybe?), but I *do* drink, and I do my best to learn to ‘channel’ the feeling of being drunk when thinking about game design. The reason? when you are uninhibited, you have a different emotional response, and I think that change in emotional response gets you closer to the enthusiasm of someone seeing your work for the first time. Drunk cliff can watch a battle in Ridiculous Space Battles and have no greater design insight than “WHOAH LASERS!”, and if thats the response to my game, then I am totally fine with that.

In fact ‘Whoah Lasers!’ is a good name for a game.

Anyway, I offer this blog post as counterpoint to the idea that game design is something that you can get from a text book and can be quantified and analyzed with ‘player verbs’ and ‘core loops’. Ultimately what you are trying to do is make something FUN and this is no different to making something FUNNY. Its folly to suggest there is an equation for either humor or fun. Making something with either of these attributes is hard, and fuzzy and it doesn’t come easily to everyone. Certainly not me.

But obviously I need to reassure you that Ridiculous Space Battles will be totally fun. Its currently 92.65% fun by I am optimizing it. You can wishlist it now etc. Wouldn’t that be fun! (am I funny?)