Game Design, Programming and running a one-man games business…

The war on needless draw calls with GPU profilers

Democracy 4, (my latest game) is quite evidently a 2D game. There is an assumption by players and developers alike that 2D games always run fine, that even the oldest, crappest PC will run them, and that you only need to worry about optimisation when you do some hardcore 3D game with a lot of shaders and so forth.

Nothing could be further from the truth! There are a TON on 2D games that run like ass on any laptop not designed for gaming. I think there are a bunch of factors contributing to this phenomena:

  • The growth in sandbox building/strategy games with huge sprawl and map sizes that can result in a lot of objects onscreen
  • An emphasis on mobile-first computing has resulted in a lot of low-power low-spec but affordable CPUs/GPUs in laptops
  • An emphasis on ‘retina’ display aspiration ahs meant high screen resolutions being paired with low-spec GPUs
  • A lot of game developers starting with unity, and never being exposed to decent profilers, or ground-up programming, without understanding how video cards work

So my contention is that there is a bunch of people out there with mom or dads hand-me-down laptop that was bought for surfing the internet and using word, and is now being pressed into service reluctantly as a games machine. These laptops aren’t about to run Battlefield V any time soon, but they can run an optimized 2D game very well indeed… IF the developer pays attention.

Its also worth noting that markets like Africa/India/China er definitely good targets for future market growth, and a lot of these new gamers are not necessarily going to be rocking up to your game with a 2070 RTX video card…

I recently did a lot of optimizations on Democracy 4 and I’d like to talk through the basic principles and the difference it can make. Specifically how to double the frame rate (from 19 fps to 109fps) on a single screen. The laptop spec in question: Intel core i7 6500U @ 2.50GHZ CPU Intel HD Graphics 520 GPU, screen res 1920×1080. Here is the screen in question:

This is the ‘new policy’ screen in Democracy 4. TBH this screen looks a lot more complex to render than it actually is, because of some trickery I’d already done to get that lofty 19 fps. That background image that is all washed out and greyscaled is not actually rendered as a bunch of tiny icons and text. its drawn once, just as we transition to this screen, and saved in an offscreen buffer, so it can be blapped in one go as the non-interactive background to this screen. I use a shader to greyscale and lighten the original image so this is all done in a single call.

So here we are at 19 fps, but why! To answer that I needed to use the excellent (and free) intel GPA performance tools for intel GPUs. Its easy to use and install, and capture a trace of a specific frame. With the settings i chose, I end up with a screen like this:

The top strip shows time in this frame, and how long each draw call took (X axis) and how many pixels were affected (y axis). A total of 158 draw calls happened in this frame, and thats why this puny GPU could only manage things at 19 fps.

Its worth setting up the frame analyzer so that you have GPU elapsed time on the X axis, otherwise all draw calls look as bad as each other, whereas in fact you can see that the first bunch of them took AGES. These are the ones where I draw that big full-screen background image, and where I fill the current dialog with the big white and blue background regions. It seems like even fill-rate can be limiting on this card.

By stepping through each draw call I can see the real problem. I am making a lot of really tiny changes in tiny draw calls rendering hardly anything. Here is draw call 77:

This draw call is basically just the text in that strip in the middle of the screen (highlighted in purple). Now this is not as inefficient as a lot of early-years gamdev methods, because the text has actually been pre-rendered and cached, so that entire string is rendered in a single strip, not character-by-character, but even so its a single draw call, as is each icon, each little circular indicator, and each horizontal line under each strip.

So one of these strips is actually requiring 4 draw calls. And we have 18 of them on the screen. So 72 draw calls. Can we fix this?

Actually yes, very easily. I actually do have code in the game that should handle this automatically, by batching various calls, but it can become problematic as limits in buffers can be reached, and textures can be swapped, requiring an early ‘flush’ of such batches, which if they happen in the wrong order, can mean missing text or icons. As such, the simplest and easiest approach is to change the code that roughly goes like this:

for each strip
  Draw horzizontal line
  Draw Icon
  Draw little circle icon
  Draw Text

…to something more like this…

for each strip
  Batch horizontal line
Draw all lines
for each strip
 Batch Icon
 Batch little circle icon
Draw all Icons
for each strip
 Batch text
Draw All Text

Which on the face of it looks like more hassle. isn’t it bad to iterate through the list of strips 3 times instead of 1? Well kinda…but I’m GPU limited here. the CPU has time to do that stuff, and actually there is likely NO TIME LOST at all, because of CPU/GPU concurrency. Something people forget is you are coding 2 chips at once, the CPU and the GPU. Its very common for one to just be waiting for the other. While the GPU is doing ‘Draw All Lines’ my CPU can keep busy by building up the next batched draw call. Concurrency is great!

I guess we nee to take a moment here to ask WHY does this work? Basically video cards are designed to do things in parallel, and at scale. A GPU has a LOT of settings for each time it does anything, different render states different blend modes, all the various fancy shader stuff, culling, z-buffer and render target settings etc. Think of it as a MASSIVE checklist of settings that the GPU has to consider before it does ANYTHING.

When the GPU has finished its checklist, it can then pump a bunch of data through its system. Ideally everything has been set up with huge great buckets of vertices and texels, and they can all blast through the GPU like a firehose of rendered pixels. GPUs are happiest when they are rendering a huge ton of things, all with the same settings. The last thing it wants is to have to turn off the firehose, and go through the darned checklist again…

But with a new draw call, thats pretty much what you do. Unless you are doing fancy multi-texturing, every time you switch from rendering from texture A to texture B, or change the render states in some other way, then you are ‘stalling’ the video card and asking it to reset. Now a GPU is super-quick, but if you do this 150 times per frame… you need to have a GPU that can handle this 900 times a second to get the coveted 60fps. Thats without the actual rendering, flipping or anything else…

So the less resetting and stalling of the card the better, which means BATCH BABY BATCH!

Now with the re-jig and those strips being done with the new algorithm (plus a few other tweaks), the GPA analysis looks like this:

Let me here you shout ‘109 FPS with only 52 draw calls!‘. Its also worth noting that this means less processing in general, and thus less power-draw, so that laptop battery life will last longer for games that have fewer draw calls per frame. Its a missive win.

There is actually a lot more I can do here. Frankly until I get to that tooltip right at the end, ALL of the text on this screen can be done in a single drawn call near the end. The same is true of a bunch of other icons. Essentially all I have done here is optimize the left hand side of the screen, and there is scope to go further (and when I have time I will).

I thoroughly recommend you keep (or get) an old laptop with a low spec video card, and slap the intel GPA tools on there. Other such suites of profilers exist for GPUs, the AMD one is pretty good, and nvidia have nsight as well. It all depends on your GPU make obviously.

I do wish more people were aware of the basics of stuff like this. Its hugely rewarding in terms of making your game more playable. BTW these tools work on the final .exe, you don’t need a debug build, and the engine is irrelevant, so yes, you can use them on your unity games no problem.

Hope this was interesting :D

Democracy 4 in Early Access (after 2 months)

So…. my political strategy game Democracy 4 has been in Early Access on steam for 2 months exactly today, plus on sale in a soft-launch alpha before then for a few months. How are things looking from the POV of one of those oh-so-predictable stats-dump indie dev blog posts?

Here are the headline numbers:

Net revenue (cash I get) about $500,000.

Gross revenue (headline dick-swinging number) about 800,000.

Before you go OMGZ how successful, I must also make a political strategy game ASAP, lets dig deeper and find out more about what that means in terms of profits. Firstly, the game is not actually even profitable yet, unless I count money that is in sales reports that I have not received. (Stores can take a while, usually 20-30 days to pay you after a month ends).

Everybody likes pie charts, so here is a breakdown showing where the NET money comes from. Note that the gross split is different as epic famously gives a higher rev share to the developer (as does the humble store). Direct sales came through and through the humble widget, both of which net me 95% of the sales revenue (woohoo). You can see how that pays off if you can get early, keen players to buy direct instead of through a store.

Of course, as I KEEP trying to make new devs aware, revenue, even net revenue after currency conversions costs and everything else are NOT profit. You have to spend money to make the game. So far Democracy 4 has cost $357,000 to make, including a reasonable but not exorbitant salary for me. Thats to get to *this point*, and there will be more spending to come because the game will likely need some more translation costs, some more time (quite a bit more), and just existing as positech costs me over $200 a month server fees and $150 month forum fees, and all sorts of other nonsense.

Still… it looks like I have a profitable game, and certainly one that should make a profit by the time it comes out of Early Access. Its a sequel to a very successful indie game so that’s not a surprise, but I didn’t take anything for granted. Its easy to get cocky and arrogant and then lose a fortune on a sequel.

So far I have not spent the majority of my allotted marketing budget for the game. My plan was to have a marketing spend of $150,000. So far I have spent just $35,000 of that. Part of that was to promote the recent autumn sale. Some more will happen between now and early access release, and then there will likely be a bit of a big ad-splurge when we come out of Early Access.

So far the game has been full price apart from the recent autumn sale on steam where it was 20% off. Its not a cheap game ($26.99) so its still likely at a price a lot of the more cost conscious gamers are not going to bite at. Also there are a lot of people who just *do not buy* early access games, so I am guessing there is a fair bit of pent-up-demand waiting for future purchasing opportunities.

In terms of wishlists the stats are like this:

D4 currently has about 60,000 wishlists. Conversion rate is 13%, total additions is 75k. I don’t get that obsessed with monitoring those figures TBH.

Because my strategy has always been maximum independence and resilience as a company, I try to spread my income as much as I can between different stores. I know a lot of indie devs think that PC == steam, and that’s just flat out WRONG. I got Democracy 4 on the epic store slightly later than I hoped for, so it missed the initial launch, but its still a nice source of revenue, and has not been discounted on that store yet, which is interesting.

Feedback on the game has been GREAT and last time I checked the average review score was in the 90s, which is fantastic. Obviously these are keen, early-players so that review score might be skewed high, but I can live with that :D. Its also worth pointing out, when assessing potential sales, that the game is currently PC only, and in English and (90% done) Italian languages. By the start of January I expect to have French, German, Spanish and Portuguese translations, which should open up the games potential market quite a bit.

A final note: My personal opinion, with a LOT of years indie experience (selling since 1997) is that new indies get WAY TOO OBSESSED with comparing their own games to those of others in terms of stats. Unless you have the exact data on 50,000 indie games (including their budgets) and a machine learning AI, its NOT going to predict whether or not your indie game will do well. I think devs should spend less time analyzing sales stats and more time analyzing player feedback and player stats. I spend a LOT of time trying to balance and optimize and improve the GUI for Democracy 4, and thats a far better use of my time.

I hope this was of some interest.

The simple pleasure of innocent drama

My wife has now firmly got me hooked on Korean drama series, or ‘kdramas‘ as we now call them apparently. On an average day, I’ll likely watch 2 or 3 TV shows, and almost certainly one of them will be Schitts Creek, Star Trek:TNG or some Korean drama. I think a lot about the appeal of this show and why I choose them, and not any of the more popular ‘trending’ TV right now.

Frankly, I think its a 2020 thing, or more a ‘the state of the world’ thing. Right now, a LOT of stuff is worrying. fake news, social media hell, the culture wars, the infighting in the USA, the looming catastrophic hell of climate change, coronavirus, and so on. There is, in other words, a lot of ‘real life drama’ and ‘real life misery’ in the world. We are surrounded by it, and can barely escape from it, thanks to twitter, smartphones and media everywhere.

real world stress

I think its no surprise then, that for those of us who feel bombarded by such negativity (especially if you HAVE to keep vaguely in-tune with it due to making a contemporary political strategy game), and who have problems staying calm at the best of times, there is some appeal in the idea of escapist TV drama.

But the problem is, too much of modern TV is NOT escapist. Or its escapism into hell, or conflict, or fear, misery, death, suicide and culture wars. In other words, writers are trying to AMPLIFY the current anxiety in the world as entertainment, instead of offering a much needed temporary escape from it. Its very rare these days to find a TV show that does not have a bunch of content warnings about ‘suicide references, addiction, injury detail, drugs, explicit sex’ and so on.

I started watching an HBO show (industry) about city traders. Kind of my thing, but in 2 episodes there has been one suicide, lots of abuse, and lots of frankly stupidly out of place and gratuitous sex and nudity. Thats fine, I’m not a moral crusader but…really? is it so necessary. I know what naked people look like, I know what sex is. You can easily imply that people had sex without us watching it. Its frankly not shocking, or edgy or artistic. Its just kinda tragic. The same is true in Game Of Thrones, or any other ‘gritty’ HBO/Netflix series that thinks its essential to have violence and sex in every drama.

5 times Jung Hyuk and Se Ri makes my heart skipped a beat in “Crash Landing  on You” – Ahgasewatchtv
This is intimacy, Korean style

For some reason, you get none of that in kdramas. Maybe Koreans are prudish, or the censors are. Who knows, but also who cares. I’ve got REALLY into a bunch of Korean dramas that are apparently romances, but the main characters don’t even kiss for fourteen episodes. One is about the military, but there has been so far, 2 scenes out of 10 episodes where someone fired a gun. One person has died. Just one. It was not explicit at all. And yet weirdly, it was still exciting and dramatic.

If HBO or Netflix made star trek:TNG now, then there would be multiple explicit scenes where we saw Picard and crusher having athletic zero-G sex. If Worf had to fight someone with his bat-leth there would be blood and limbs scattered all over the place. We would all be tired by now of seeing riker stark naked. Wesley would be snorting cocaine in every other scene…

And it wouldnt be any better a drama series at all.

Pin auf Cool, Random, Fantastic, Funny, Brilliant Stuff...
This is the ideal male body. You may not like it but this is what peak performance looks like.

KDramas, and older shows like TNG, or even family-rated shows like schitts creek, show that you don’t have to bombard people with sex and violence to entertain them. We live in an age where even the US president is accused of paying off porn-stars, where politicians are often caught taking hard drugs, where the media is packed with the salacious details of celebrities addictions and sex lives and so on. There is enough of this in our lives.

Its still ok for entertainment to be FUN. It doesn’t have to do ANYTHING else than entertain. You don’t have to be making statements about social-justice in every casting decision, and every scene. You can cast people of color in a role without them having to shoehorn in a race-relations plotline (even in space!). We are fine with just enjoying some escapism, munching some popcorn and watching fun things happens and hear people speaking witty dialog. You can have conflict without spraying blood and intestines at the viewer. People can fall for each other without us watching every sexual position in 4K.

Give me more kdramas and less ‘gritty, brutal re-imaginings’. if I want gritty and brutal I’ll watch CNN.

Modelling cultural differences in Democracy 4

One of the biggest challenges in designing and balancing Democracy 4 is having a single simulation model work over all countries. In theory, if *everything* is modelled, then you have a perfect model of human society and behavior, and feeding in the correct settings to each country will produce the required results.

Because I am neither superhuman or crazy, I don’t think this is really feasible, so we limit the simulation to that subset of variables that I think makes the game playable and fun, and reasonablyt accurate, and then I am left with the task of working out how to fudge that simulation to fit real world experience.

An example of this in the game is productivity. In the game, worker productivity is quite an important simulation value. The player can see its value, a chart of its history and view its inputs and output:

Of course the simulation is assuming that the list of inputs and outputs to this value make sense over all countries, or in other words, in all cultures, but this is a very simplistic view. Whast actually makes one country more productive than another? there can be a million things, many of which we do not model. For example:

  • Extremes of temperature in a country may require a mid-day siesta where less/no work is done.
  • A culture of extreme deference to authority may reduce productivity due to a lack of suggestions/criticism of bad practice
  • Highly variable and extreme weather may mean that work is regularly interrupted as a result
  • A cultural expectation of strong work/life balance may reduce the tendency to do overtime
  • A cultural work ethic may encourage overwork and unpaid overtime by default

We do not explicitly model any of these things as tracked variables. You could argue that it would be possible, but in some cases they have historical roots that are now just cultural norms and cannot easily be explained. For example here in the UK, the British people tend to have a general distrust of, and dislike of politicians and authoritarian figures. ID Cards are NOT a popular idea here. The reasons for this are hard to discern, but its certainly rooted in the past and not something you could easily model with numbers.

Similarly there is a deferential attitude to age and rank in many Asian countries that works in a similar way, and likely has both positive and negative outcomes. I watch a lot of Korean dramas these days and its fascinating the extent to which people seem both obsessed with education and status, and also massively invested in the minutiae of office politics. Why is this? very hard to say.

Review: Misaeng (First Impression) – The Korean Lass

It might be hard to model this stuff, but we certainly cannot ignore it. For example Germany IS a very productive country, especially at manufacturing. It has decent holiday provision for workers, and strong unions, yet its productivity is super high. why? A lot of it can probably be put down to the cultural work ethic. Take this article, for example:

“In German business culture, when an employee is at work, they should not be doing anything other than their work. Facebook, office gossip with co-workers, trolling Reddit for hours, and pulling up a fake spreadsheet when your boss walks by are socially unacceptable behaviors.”

Because I need to model Germany as accurately as I can, I have blatantly included this as a script that runs and generates extra values for Germany to represent German Culture. in this case, a free bonus to worker productivity:

Adding this may seem like a 2 minute job, but NO. It triggered a boost to GDP that resulted in traffic congestion caused by the emergence of a gig economy (uber!). This has high regulation in Germany, so I had to boost labor laws, which then results in a corporate exodus away from germany…

I am currently considering whether the German language should act as impediment to corporate exodus (Companies tend to prefer to relocate to a country with the same language as them, and German is not as popular as English)… and so the balancing continues…

There are similar scripts for each country, and I’m gradually amassing as much feedback as possible and tweaking the values to make each country feel authentic, while keeping the same basic economic and political model. Lots more tweaking to come over the next few months!