Game Design, Programming and running a one-man games business…

Anatomy of a rare, and weird heisenbug in Democracy 4

Heisenbugs are bugs in code which seem to go away, or change when you look at them in a debugger. They are the worst possible bugs, because they actively resist being found, or even observed by the coder responsible. Any kind of bug that can be reproduced on a developer’s machine, in their development environment can easily be analyzed, stepped through and reasoned with. But heisenbugs are a nightmare. I just (I think) fixed one, and certainly fixed A bug, even if not this one. I thought you might enjoy the process…

First some background. Democracy 4 is the fourth and latest in my series of political strategy games. They are text and diagram heavy, with a properly unique user interface, in 2D, and all of the code is custom, with a custom engine originally coded by me, with a unicode text implementation and vector rendering system coded by jeff from stargazy studios. This means all of the UI elements you take for granted in middleware you use, were custom coded by us. In this case, the bug is my fault, in code written by me. Here is a screenshot:

Then key thing that matters there is the text block under ‘food stamps. This is a policy screen in the game, and that text is the description. Depending on the length of the description (partly influenced by the language you play in), and the screen resolution, it might be that all the text fits in the available space, or it does not. That means sometimes there is a scrollbar to the right of that text, and sometimes its not there. As you would imagine, the scrollbar works just like any windows scrollbar. That means it responds to mousewheels, and also clicks, on the top and bottom buttons (very small) for example.


VERY rarely. And ONLY when I was playing the game outside the debugger, and only in very arcane sets of circumstances that seemed totally utterly random… screens like this (but not just this one) would occasionally not let me click on some elements in the interface. They would not respond at all… BUT… that block of text would scroll upwards, like it was being scrolled through. It was like completely unrelated events (clicking somewhere else) activated a scroll function on a user element I was nowhere near.

This was of course, absolutely infuriating, because it was REALLY rare. And the times I tried crazily to reproduce it in the debugbuild, or even in the release build, but launched from Visual C++, it would not happen. And there were NO reliable steps to reproduce. In fact, even quitting a screen when it DID happen, and then returning to that screen would make it go away.

What the hell?

After a LOT of time, I realized two interesting things. Firstly, when it happened, the game DID make the click sound associated with a mouse click (so the game IS realizing I clicked somewhere) and the scrolling only happened with text inside a specific UI element called a GUI_TextContainer. Although I did not realize it at the time, it was also the case that it ONLY happened where that text container did NOT have a visible scroll bar.

The clue to the bug lies in a closeup of the scrollbars used:

Like all such bars, there is a clickable ‘scrollup’ button, a draggable bar, with above/below page up/down regions, and a clickable down scroll button.

When I create a new GUI_TextContainer, I created a new GUI_ScrollingWindow, which is a container with all of those elements in side. Like any child element, I added it to the Text Container object with AddChild() to ensure it responds correctly and is processed blah blah. Thjn, later when the textwindow is initialized with its position, and the relevant translated text, I do a lot of calculating to work out if that text *is* going to fit in the available space. If not, I set the scrollbar in the correct position on the right, and make a note that the text does not fit (sets BFits to false). Then later, when drawing this UI element, I could skip out a lot of nonsense I don’t need if the text fits, and render it simply as a word-wrapped string. If It does NOT fit, I then drew the scrollbar, and the text in a different way, cropping the text render to the space given the scroll position and so on, blah blah.

All of this sounds reasonable. But it was a big mistake.

I was correctly NOT drawing the scrollbar if it was not needed (why bother! the text fits!) but I had pre-emptively added it as a child of the window anyway. Worse still, I had never initialized the position of the scrollbar and its sub-components AT ALL. This means that in debug / release-from-the-development-environment, all of the relevant variables in that window were getting set to zero, But in a live ‘customer’ environment, the initial positions of those subcomponents could be ANYTHING.

So what could happen, really rarely, is that the ‘scroll up’ or ‘page up’ parts of the scrollbar, might have coordinates like -32054,-5128,27140,41543. In this case, those clickable elements filled the whole screen… but were INVISIBLE. Thus, I have a huge UI mess-up, but I cannot see it, because my ‘quick and easy bath’ that checked BFits, doesn’t draw it.

So very rarely, I create a screen with an invisible scroll up button that fills the entire screen, and happily responds to every click with a scroll-up command. It even nicely plays the button click sound. Luckily, these buttons do not handle keys, so the escape key still quits that whole screen, and the next time it gets created, I’m probably lucky and the invisible bar has moved somewhere harmless.

I’ve changed the code now, to be efficient so that I do not even THINK about creating this scrollbar until I need it, and otherwise its NULL. I also make sure to safely delete it before creating it, in case somehow I go mad and initialize the same text container twice with different blocks of text. I could also have fixed it by initializing the position of a scrollbar safely to 0,0,0,0, but I also like the neatness of ensuring I don’t have a pointless orphan UI element at any point.

This was a classic dumb error. I added an element when I shouldn’t have, and its member variables were not safe. My own dumb slackness. I document it clearly to illustrate how the weird unrelatedness of a bug ‘clicking down here, scrolls text up there. sometimes. only in German, on a Wednesday etc…’ eventually makes total sense.

Hundreds of thousands of people have played probably over a million hours of the game, with this bug in it. No code is bug free, but if you do things right, the ones left in are pretty arcane.

Speeding up Democracy 4 simulation processing (atof is slow)

I just made a major speedup to the loading/next turn/new game code in Democracy 4, and thought I may as well share the gains with readers of this blog :D. It kind of makes me look a bit like an idiot to admit this is a big speedup (I should have known), but anyway, knowledge sharing (especially on optimization) is always good.

Fundamentally, the code in Democracy 4 is structured like a neural network. Without going into tons of details, every object in the game (policy, dilemma option, voter, voter group, situation…) is modeled as a neuron, which is basically just a named object connected by a ton of inputs and outputs to other neurons. You can run through the inputs and outputs, process the values and get a current value for a neuron at any time, which is done for every one of them, every turn.

Also… when we start a new game, I need ‘historical’ data for each value, so the game pre-processes the whole simulation about 30 times before you start to give us meaningful background data, and to ensure the current simulation sits as a reasonable equilibrium.

Those connections to neurons should probably be called dendrites or whatever, but I call them SIM_NeuralEffect. They contain basically the names of a host and a target (resolved to actual C++ pointers to objects), and an equation explaining the connection, and some other housekeeping stuff.

At the heart of it all, is an equation processor which lets you write this:


And actually turn it into a value for that effect, given the current situation. The Equation processor runs each turn, on every neural effect, and there are LOTS of them. Thus, if the equation processor is slow, its all slow.

I just installed a new version of the free vtune profiler from intel. Its not recognizing my ultra-amazing new chip, so only doing usermode sampling, but nonetheless it draws pretty flame charts like this:

Before I optimised

This is showing the code inside that 30-turn pre-game processing called PreCalcCoreSimulation. Lots of stuff goes on, but what I immediately noticed was all this atof stuff. Omgz. Thats a low level c runtime function, not one of mine, and it seems to be slowing down everything. This is a HUGE chunk of the whole equation processing code. How is this possible?

Now, you may think ‘dude, atof is pretty standard. No way are you are going to be able to make that code faster’, to which I reply ‘dude, obviously not. But the fastest code is code that never runs.’.

All those calls to atof are absolute nonsense.

Looking back at the equation above (OilPrice,0+(0.22*x)*GDP) there is obviously some stuff in there which is volatile. I do not know what the current value of x or GDP is, so I will need to grab their pointers and query them when I process the equation, but the rest of that stuff is static. That * is going to remain * and that 0 and 0.22 will remain fixed too. This is the key to a roughly 33% speedup of the whole processing in the game.

I actually did know to look into this, and I do not do manual text parsing of the equation each time. I parse them equation on startup, and stick the various values into buckets, so I am not wasting time each turn. But one thing I had not done is store the atof() outcomes. I was still storing variables[0] as ‘0’ instead of just 0.

Now you may think atof is fast. Its fast enough for most cases, but its WAY slower than just accessing the value of a floating point number thats already in RAM, and cached happily in the equation processor itself. Here is the new diagram:


The difference is, (on my superfast PC), for the whole precalc simulation function: 0.85 vs 1.50 seconds. This probably makes me sound pedantic as hell, but I’m rocking some stupidly new and pricey PC, so there are likely people playing D4 on laptops a fifth the speed. I might be knocking a whole 3 seconds off the new game time for some players!

Also, and worth remembering, I just saved doing a ton of processing, which means a ton of CPU time, power and heat. If you can make your game run more quietly, more coolly, and faster on players PCs, you absolutely should do it.

Lessons from 41 years of programming

I don’t blog much about the nuts and bolts of programming, because I know enough to know that others know more than me, but increasingly I’m aware of how other people code, and feel that I probably do have *some* worthwhile advice on the topic. Not many people have been programming for 41 years, especially given how coding seems to skew young, so here we go…

First some background, I started coding in sinclair BASIC on the ZX81 aged 11. I took some time away from code to try and be a musician, and then worked in IT support, but I have worked as a full time programmer for the last 25 or so years. After a while its hard to keep track! I basically code video games in C++. Thats it. I know some php, but not much. I can probably remember some BASIC, but not much. Yes, I’ve been coding 25+ years in a single language.

Lesson #1 Learn a single thing well.

I sometimes do some stuff that would probably be a bit easier in C#, or Rust, or Java, or probably some other language I have not even heard of. I don’t care. I am a massive believer in knowing how to do one thing super well. This is very unusual, because these days, coders are hired as a shopping list of buzzwords. Adverts for coding jobs amuse me no end. They always want someone with experience of 10 different languages. Did the lead programmer have a problem deciding which to use? When you want a coder who knows 10 languages, you get someone who is mediocre in lots of different ways. Thats not ideal.

The ‘familiar with 10 languages’ trope is common because non programmers understand it. A coder with 10 languages must be more skilled than one with just 1 right? To a non coder, its all unintelligible gibberish anyway, they have no way to tell a really GREAT c++ coder from someone who just knows a bit of syntax. Recruitment consultants and idiotic managers without coding experience have created a world where everyone is truly rubbish at 100+ different programming languages.

We do not expect this in other fields.”Doctor wanted. Must be a throat expert / hearing expert / heart specialist / Neurosurgeon”. We also manage to get by in our lives speaking only a single language. Sure, french is more romantic, German more precise, Spanish more passionate, English more subtle, but we don’t change languages constantly in our day to day lives! Tip: If you work for yourself, learn ONE language super well. Only learn others if they are absolutely essential for the task in hand. You can do almost anything in C++.

Also note that being really good at one language does not mean learning every possible feature (see next lesson), but it means using that language so MUCH, in so many different circumstances, that you know the best way to get things done using that language. (by best I mean reliably, efficiently and legibly).

Lesson #2 Language features are optional.

C++ lets you override operators. It also supports templates and the stringizing thingy (## i think?). I don’t use any of them. I use maybe a quarter of C++. Most of the newer stuff seems like solutions looking for a problem. Its FINE to just stick with the feature set you find to be most usable for you. Nobody is going to point at you at parties and mock you in front of the opposite sex because you don’t use the whole range of C++ features in every program.

There are multiple ways to handle files in C++. fopen, iostream, CreateFile() etc. I use the old fashioned fopen() stuff. I know the syntax super well. I can type it as fast as I can speak. This is fine. I’ve never had someone leave me at the altar for not using iostream.

I am constantly told by people I am reducing my productivity by sticking with old, outdated and cumbersome systems when new-fangled ones have apparently done the same but better. Without exception I am more productive than everyone who tells me this.

Lesson #3 Legibility is the goal.

Coders start off as n00bs with no confidence or idea how to do things. They then go through a ‘gunslinger’ phase, filled with the coders equivalent of testosterone, where they write ‘impressive’ code, that uses cool obscure language features or fancy techniques to do whizz-bang stuff that impresses other coders at parties. This phase lasts about a decade, or maybe longer if you are in a big company with bosses that need impressing.

The ultimate stage for a coder is to realize that ‘impressive code’ phase is just so much adolescent bullshit. The goal of code is to be reliable, efficient and legible. That last one really matters. If there is a way to perform a task that is simple, clear, and a way that I have used 100x in the past, then thats the way I do it. There may be a ‘cool’ way to do that task that involves callbacks and threading and whatever the heck microservices are… but why over-complicate things when you can just write simple code that you will understand instantly on reading it in 10 years time. 99.99% of the impact of your code is on the users, not the other coders who might glance at it. My code probably looks simplistic, very literal, very accessible for even non coders to read. I’m happy about that.

Lesson #4 Consistent Formatting

No 2 coders ever agree on how to format code. I have my own opinions which I shall not bore you with, but here are two thoughts: Firstly, be aware that a lot of ‘standard’ ways to format code were dreamed up before we had intellisense and visual assist. If you are still relying on a naming convention that shows you what is a function, what is an enum and what is a variable… then its kind of pointless. We have syntax-coloring now. You don’t need to do that, and can get back some much needed legibility. No more LpSZ_mem_Name.

Secondly…and this is even more important, the goal of code formatting is to minimize the cognitive load on the person reading it. Most programs do complex things, and to debug or add features, you need to maintain a HUGE network of complex stuff in your head as you work. If you are having to mentally translate a lot of different variable-naming conventions and code layouts in your head as you read then you will be amazed at how much








Our brains seem very good and building up mental translation systems. if you have text in ALL CAPITALS then we adjust to it ok. Pretty much *any* coding standard is fine… as long as you have only one. This is another reason why I hate middleware, because obviously its all formatted differently. Having to chop and change mentally between different ways of doing the same thing adds huge mental overload.

If I ran a studio, the question ‘will you agree to 100% without question always follow the code style guide of the company or face immediate dismissal’ would be the first interview question for new hires.

Lesson #5 Be aware of the overhead.

Almost all modern code is absolute trash, total garbage, an embarrassingly badly constructed towering pile of crud that should bring deep shame on everybody who wrote it. People with a lot of coding experience tend to agree on this topic. I think its inevitable, given that the demand for coders vastly outstripped that number of people with experience. Maybe over time it will be redressed, but I worry that these days coders are just used to code being crap and do not understand its even a problem. You don’t need 20 years experience to post a useless reply on stackoverflow. Sadly.

Its REALLY instructive for you to step through every single line of code in the execution of the stuff you do. If you are an app developer relying on middleware that is closed source… i pity you. A simple step through one instruction of yours probably means running 5,000 lines of bullshit layers of code that you will never see. This is why we all have incredible supercomputers, and yet we stare at progress bars…

If you are lucky enough to have full source access, or have written it all yourself (yay!), then do the ‘step-through-everything’ test regularly. It will AMAZE you how much code seems to be run without actually being needed. The inefficiency of most code is mind-boggling, and its because hardware has outrun software so much that we can write code that runs at 1% efficiency and hardly anyone complains.

Learn how to use a profiler, and go through your code looking at where all the time is spent. Its actually REALLY interesting, and kinda fun. I enjoy spending time in stuff like aqtime, or vtune. If you have not profiled your code, then you have not finished it. You are not a software engineer if you have not done the post-coding analysis of what the code actually *does*.

Lesson #6: Avoid massive functions, and massive source files.

There are technical reasons for this, in terms of compile times etc, but it really comes down to the overall structure of code. If you have a function that is 100 lines long or more… then you have probably fucked up. If a single source file for a class is 1,500 lines long…then you have probably fucked up. (yes, I know about function call overhead. Are you working on spacecraft control systems that have to run at 5,000 fps with minimal power? I am guessing no, so don’t worry about that trivial overhead here)

The most common mistake that leads to buggy, hard-to-work-with C++ code is that classes are too big, and functions too long. A function does one thing. The clue is in the name. What tends to happen is people start off with a perfectly reasonable class and functions, then they add functionality (ha!) and the class sprawls, and the functions sprawl. What was once a perfectly reasonable Update(), becomes 100 lines long. What was once a reasonable Entity class suddenly becomes a behemoth.

This is normal. You now need to work out the cleanest, most sensible way to break that class into multiple objects or derived classes, and change the layout so that the code is done in more single-focused functions. This is not some bothersome task that wastes time… this is literally software engineering. This is the difference between a coder and a software engineer. This vastly increases the legibility and debugability of your code.

Getting code to work is just the easy bit. The hard bit is knowing exactly how to layout the relationships between objects and functions so that everything seems sensible, and organised. Knowing how to do this takes a lot of experience, and is normally the outcome of many bleary-eyed 3AM debugging nightmares where you stare at the big giga-function when you are nested 16 deep into curly brackets wondering what the fuck its supposed to do.

Call stacks are your friend! Call stacks are amazing. I have the call stack always open at the bottom of my screen. I offload my mental model of ‘how we got to be here’ into the callstack window. I find this incredibly helpful. Making sense of the flow of code through the callstack window is 100x easier than keeping mental callstacks in mind as you navigate giant functions.


These are my broad thoughts on stuff I wish I knew earlier. I’m sure lots of people disagree, and thats fine, I’m not starting a political movement here, just sharing my experience. Sadly, the awful state of modern coding is largely outside coder’s control, due to incompetent managers, feature-centric marketing and tech-ignorant recruitment practices. I hope its of some interest to solo coders or lead coders with complete control of how a codebase is made.

Democracy 4: Resizable GUI

For the last week or two I have been working on what seemed like it might not be too bad…but actually turns into a lot of work. This is a resizable GUI for Democracy 4. The current version of the game has fixed size UI elements everywhere which works fine for the old school screen res from maybe 1280×768 up to 1920×1080, but starts to get a bit annoyingly small text at 2560 plus, and frankly the UI can be a bit too big and blocky at the lower res too. I finally got around to fixing this.

The first day or so was wasted trying to find an automated solution. Democracy 4 uses SDL2 and OpenGL, and I was hoping some of SDLs scaling functions would handle this simply. I could easily implement a scaling slider in the game, and then SDL could just handle a final stretching up or down at blit/flip time of a fixed resolution image.

This failed to work. partly because of some messy implementation at my end perhaps, but also because screen aspect ratios can change. Even if it DID work, it would basically mean throwing away the core UI upgrade of Democracy 4 over 3, which is super-smooth fonts and pixel-perfect vector-based icon and UI element rendering. If the player set the render scalar to anything but 100%, any sort of stretching would give a slightly blocky look. I couldn’t live with that.

In the end, I did implement all this as a simple percentage slider on the games options screen. Changing this requires a reboot to see the effect:

In order to get all this to work I just had to make a LOT of small code changes. Probably every UI file in the project got changed in the last week. What I needed to do is get rid of what coders called ‘magic numbers’ and replace them with values that could be scaled up or down based on this slider. For example code that said this:

int iconleft = Area.left + 120;

Would be changed to something like this

int iconleft = Area.left + WIDE_SIDE_PADDING;

The all-caps value could then be coded as defaulting to 120, but be scaled up or down by a global value for RenderScaling, which is decided when the game starts. This way I could reuse that value anywhere in the game and know it would always be the right value. Because I’m not entirely useless, luckily I HAD actually defined and used a lot of named values already like this:

int iconleft = Area.left + STYLEGUIDE_BLOCKPAD;

So in that case no code change was needed, and that value (10 as a default) could be easily scaled at app startup. The problem was… I had not stuck to this, and actually used a LOT of magic numbers (ie: actual numeric values) many, many places in the code. I ended up just having most of the common ones defined as pixel constants:

int iconleft = Area.left + STYLEGUIDE_PAD50;

For example. A bit kludgy, but some values really are used in random places and giving them stupid names like STYLEGUIDE_MINISTER_)SCREEN_TOP_WINDOW_HEIGHT would be overkill if you ask me. In any case, after a LOT of typing and also a LOT of testing, I am very close to declaring this done, and letting users play with it. Its not perfect, because a lot of combos don’t work. If you are playing at a small res like 1280×768 and set the slider >100%, things will overlap and look rubbish. But I’m hoping people are sensible. This is a FIX for people who dislike the default layout for high or low resolutions. Its assumed 95% of players will not touch the slider.

Anyway here is the game in the current (unscaled) view in 2560 resolution and default 100% scaling:

(Its reduced by 50% for here). Check out how small the text is for the finances, and how much space I waste on the timeline at the top. Now here is the exact same screen resolution, but with a 133% slider:

To me this looks WAY better, but you are only REALLY going to appreciate the change if you have a high DPI but smallish monitor, or an insanely high res monitor and poor eyesight. At first glance, you might not be able to tell the difference, but then check out that timeline at the top to see just how things are re-arranged. or check on the far left near the top, the ‘POPULARITY’ text.

Anyway this is coming to the next update for the game. it took a while, but it really needed doing!

CPU/GPU concurrency in video games

I’m no graphics programming expert, and not really any kind of programming expert, unless you want a strategy game coded with its own engine, in C++ for the windows platform in which case *cracks knuckles* I’m pretty experienced. (Actually I dont know hot to crack my knuckles).

What I do know,. is what to look out for, when you are worried about performance. One of the things I learned early on, was learned REALLY early on, when I made a game called Kombat Kars (probably in directx5) and was working on particle systems. To make it clear just how many aeons ago this was, lets take a look at an epic image of the rear boxart (yes! retail!)

Kombat Kars (2001) Windows box cover art - MobyGames

Yup, its not the frostbite engine.

Anyway, I was working on optimizing the drawing of vertex buffers full of particles, or asteroids or whatever, and I was depressed to discover after doing some cunning batching of my draw calls, that the performance went DOWN. Yup. Making the game more efficient in how few draw calls it made, made the game run SLOWER.

How can that be?

Actually super-easy, barely an inconvenience, but to understand why, you need to conceptually understand whats going on in the box when you run a PC game under windows. You basically have two CPUS. One of which is on the motherboard and is general purpose, the other is on the video card and specialized for processing vertexes and shaders and so on. It used to be 95% CPU work, and 5% GPU work. These days the GPU is often the most expensive, and powerful component in the box. On a lot of setups, the capabilities are fairly equal.

Its that equality of power that can actually cause problems. The peak performance of the machine is when the CPU is 100% busy (all threads!) and AT THE SAME TIME the GPU is 100% busy (multiple streams at once etc…). This is almost impossible to achieve, but its possible to actually make things worse than they should be, when you get too obsessed with batching.

If you don’t care about performance you code like this:


Then one day you read some articles about the reason your frankly low-poly indie game runs at 20fps is that you have WAY too many draw calls. You read about batching, and your new code looks like this

for(int n = 0; n < lots; n++)
for(int n = 0; n < lots; n++)

And all is good in the world, because suddenly you are not flushing the queues on the video card every nanosecond, and its doing what it likes to do, what it was BORN to do, which is to stream through a whole ton of data like a sieve and throw polygons at the screen fast! But hold on…things can go wrong…

for(int n=0; n < eleventybillion; n++)

This can actually be a REALLY BAD IDEA. Why? surely batches are good right…? well…to an extent. It really depends how you structure the code. It *might* be that during all those bazillion PrepareAMesh() calls, the GPU has run out of things to do. Maybe it hasn’t done ANYTHING yet this frame. It finished the last frame, and now its basically watching netflix waiting to hear from you some day…

…and once the CPU calls the GPU to render all bazillion polygons, depending how you structure the code, the CPU may be doing nothing. Maybe this is the frame end, and the CPU has to sit on its ass waiting for a Flip() or present() call from the GPU to get back to it some time maybe next week after the rendering is finished, when it can start thinking about the next frame?

This is the CPU/GPU concurrency issue. You can be TOO BATCHY. You can inadvertently set things up so that the GPU is always waiting for the CPU and the CPU is always waiting for the GPU. This is BAD for performance.

Luckily, free apps like VTune let you analyze this. FWIW Democracy 4 has no such problems with this at all, but to show you how it looks, here is the output of a very brief snippet of the vtune CPU/GPU concurrency analyzer:

You can see near the bottom how busy the GPU and CPU are. Luckily for me, they both keep pretty busy, even if I zoom in a lot to see the span of individual frames, but if your zoomed in CPU/GPU concurrency stuff shows big empty blocks within a frame, you have some optimizing to do.

The reason this catches out so many experienced coders is that it *sounds wrong*. Surely batching is good right? It is… but you have to remember that if the GPU would otherwise be sat on its ass eating crisps, even doing a bunch of small inefficient batches of 50-100 vert each, is MORE efficient that just letting it sit idle.

Think of the CPU/GPU as a team trying to do the dishes. The CPU is washing em, the GPU is drying them. Don’t let either of them stand idle.