Game Design, Programming and running a one-man games business…

Learning to code (with george osborne!)

So there is this new government initiative in the UK where apparently every student in the country will be taught some programming in an effort to increase the technical knowledge of the future workforce. I see this as an extremely good thing. It surprises me that we have gone this long without introducing it as part of the school day anyway. Physics, Biology,Chemistry & programming? They seem like fairly equivalent importance to me, although I think we should add in ‘Engineering’ too. What slightly worries me is that the depth of the courses will not go far enough.

The general theme behind all this is that schools have been teaching people how to use Excel, rather than teaching them programming. This is clearly a bad thing, partly because most kids know how to use excel anyway, but the problem is, the grasp of what ‘programming’ really means seems incredibly weak amongst those who are discussing it. I keep seeing discussions where politicians and campaigners talk about how ‘you too’ can learn the ‘complexities of HTML’, and equate this to an understanding of how computers work.

Don’t get me wrong, I’m all for people understanding how HTML works, but we can surely go further than that right? Shouldn’t we be encouraging kids to go a bit further down the programming rabbit hole? At the very least php and java-script. ideally all the way to C# or even a real-mans language like C++*. I wouldn’t be shocked if everyone understood at least in principle what assembler was. (I know I could do with learning some more there). My point isn’t that we need a million C++ coders, but that understanding ‘a bit’ about how it works means it’s much easier to find the 1% of kids who really do want to study that sort of stuff, and in any case knowing a ‘bit’ about the next-most-complex layer of a technology is always beneficial. I’m not an assembly language programmer, but I’m not horrified to click on the breakdown in aqtime and look at my codes assembler breakdown either. Knowledge is good.

I know you have to start somewhere, and people think that stuff like C++ will scare people off, but hey, lets set some lofty ambitions shall we? At the very least lets not make the mistake of giving an entire generation of kids the idea that Windows 8 was programmed in HTML and that this is as low-level as it gets. There is more to coding than HTML and phone apps. A lot, lot more.

*I’m kidding**

**sort of :D

 

Cleaning the engine

It’s tricky to get the timing right. Some people do nothing but update their engine. they have re-factored it so much and re-implemented everything so often that it is a true work of art, worthy of actually putting in books on how to code. These people never ship a game. The other extreme is people who are still using assembly language routines for loading in ini files because they wrote them in 1996. These people have tech support issues galore, and are probably still using Directx3.

In between there somewhere is common sense, if you *are* going to write your own engine and not use someone else’s. I think, from chatting to devs and surfing a lot on developer forums, that people tend to want to redesign their engine after every game. They consider that quite a major compromise compared to their real hidden urge to do it every morning. The thing is, if you are doing that you aren’t really writing an engine, you are just writing a new game from scratch all the time.

lexus

I tend to go with a system of marginal improvement. I still have some code from about 5 games ago (non critical stuff like ini file loading, text handling, game timers and some math stuff), but a lot of it is fresher. The graphics stuff, as you would imagine gets a major refresh more often, and my engine only contains some pretty ‘raw’ directx wrapper and vertex buffer / text engine stuff. The actual ‘scene management’ for my games is done in the game itself.

Despite the occasional between-project update and maintenance, occasionally you have to step in and clean things up. The last big update was when I went from Directx7 (Democracy 2, Kudos games…) to Directx9 (GSB,GTB,Democracy 3). This time I’m updating almost everything BUT the directx version.

I recently bought Visual Studio 2013, mostly for the concurrency profiler to enable me to experiment with multi-threading more. This was a good opportunity to take a look at some of the flakier things in my engine. I have a lot of warnings in there for data-conversion and other sloppiness. I also have code I never use (I’m culling it), and the worst and most embarrassing thing is that I can’t decide if I like char* or std::string. I Figure that std::string must be at least a bit better, more robust and safe than char*, so I’m trying to purge all that char* from the engine, and eventually, the game. I’m also planning on re-wiring stuff so that the main game code doesn’t have any FILE* or other old fashioned stuff, but uses my file wrapper more.

Why? Mostly because I can see me heading towards cross platform eventually. maybe not with the next game, but baby-steps and all that… Plus it makes life easier if getting my engine ported is a less messy business. I’m sure after a few days of sorting out this stuff I’ll be climbing the walls and wanting to code some explosions again…

Thread Concurrency bug. Any expert advice?

I’m new to multithreading, and the visual studio concurrency visualiser. I have lovely multithreading code working. basically my main thread throws the Threadmanager a bunch of tasks. The threadmanager has a bunch of threads which all spin doing nothing until told to do a task. Once they finished their task, they set an IDLE flag, and the main threads WaitForAllTasks function then assigns them a new one. It works just fine…. but I notice an anomaly.

See below (Click to enlarge enormously)

bug

The highlighted connection shows thread 4828 sat on its fat ass waiting for the main thread, when it clearly idle and ready to do stuff. The things is, the main thread function is just this:

    bool bfinished = false;
    while(!bfinished)
    {
        bfinished = true;
        for(int t = 0; t < MAX_THREADS; t++)
        {
            if(CurrentTasks[t] == IDLE_TASK)
            {
                //maybe get a queued item
                if(!QueuedTasks.empty())
                {
                    THREAD_TASK next_task = QueuedTasks.front();
                    QueuedTasks.pop_front();
                    CurrentTasks[t] = next_task;
                    SetEvent(StartEvent[t]);
                }
            }
            else
            {
                bfinished = false;
            }
        }
    }

Which basically does sod all. How can this take any time? Setting event sets an event which the worker threads function is calling WaitForSingleObject() on. Again…how can this take any time? Is there some polling delay in WaitForSingleObject? Is this the best I can hope for? It’s the same case for all those delays, it’s just this one is the largest. I’m new to this. Any ideas? :D

 

 

What a directx developer does in a steambox world

Elaborating on my last post. Valve have made it pretty clear they are heavily betting on Linux. Microsoft, you missed the ball. You didn’t get the hint. What action did you take to get valve back? Apparently not enough. Regardless… Linux will be big for gaming. It’s 2% of my sales right now. This is going to grow and grow fast.
This is no big deal to me, apart from the fact that Linux also means OpenGL. I currently have my own engine which uses DirectX. What should I do? Here are my options…

steambox1) Do nothing.
Just stick with it. 2%? Even if it goes up to 20% so what? My current system is me doing the directx version, then paying a contractor to handle the OSX & Linux ports by converting it all to OpenGL. This works fine so far. This option is easy, but it seems like I just go further down a dead end. I don’t want to be the last guy using DirectX.

2) Prepare my engine to be more cross-platform friendly
I could re-engineer my code so that it’s more modular and easier (re: cheaper, faster) to port to OSX/Linux. This way I still keep directx, but make things simpler for a future transition. This is appealing because it’s a hedge towards either direction, and frankly engines should be built like this anyway.

3) Learn OpenGL.
This is the hardest because it involves serious work. I know everyone will say ‘it’s really similar’, but being ‘familiar’ with OpenGL isn’t what I want. Being VERY sure I know the BEST way to use OpenGL for my purposes is what I need. I love optimizing. I love pushing to get as rich a 2D experience as the hardware allows. I don’t want to trade a really fast DirectX engine for a slow OpenGL one. This involves the most work, to do it right. There is huge opportunity cost for that time as a programmer. This is also by far the most future-proof.

4) Hire a full time Linux/OpenGL coder.
Not a sensible option for me. I couldn’t keep them busy, I still don’t learn anything, and this is effectively 1) but more expensive and with more admin. All it does is guarantee me availability, but at too high a price.

5) Switch to middleware.
An option for 95% of devs. I hate middleware in general, but I also do something very unusual. The engine for GSB 2 works in a very specific way. At it’s core, it is a heavily optimised, shader & post-processing based, lots of render-target flipping 2d engine that assumes a huge amount of 2D objects. I’m not aware of any engines designed specifically to do this except mine. Plus there are costs in terms of learning that new engine,  which represents investment in something I neither own or control. I don’t like this.

I’m thinking about this a lot. I’m currently favoring 2), Possibly 3). I have discounted 4) and 5). Doing 1) seems lazy. This is something I need to get right at some point this year.
Feel free to add your thoughts :D

Thoughts on multi-threaded 2D game development in directx9

Am I the only person doing this? probably. I often am. Most people have moved on from DX9 (I know it so well there is big opportunity cost to updating) or use OpenGL, and very few people are doing 2D games where performance is an issue. I am taking early steps with Gratuitous Space Battles 2, and my aim is to have it run at 60 FPS on average hardware with 2 1920×1080 monitors. I also intend to get it running ok for bigger setups too. That’s a lot of pixels, and due to all sorts of fancyness I’m adding to GSB 2.0, it means a lot of processing. a REAL lot.

So…multithreading! it’s about time i ventured forth. To date, my only multi-threading efforts have been the asynch server communication in GSB 1.0 for challenge uploads etc, and the loading screen for GTB and Democracy 3. Actual mid-game multihthreading has scared me until now.

I hate middleware so I’m not using any libraries, just raw calls to CreateThread, TerminateThread and so on… This might make it more complex, but means I have complete control over stuff. My first experiments were not exactly encouraging. I attempted to speed up the position calculations of asteroids. Now to cut a long story short, I use D3DTLVERTEX style stuff (not hardware Transform and lighting) and for good reason i won’t bore you with. The upshot is, I have a lot of non-directx transform stuff to do for anything drawn on the screen.

An ideal case for multithreading!

threads--friend--grey-cat--thread_3218279

So I wrote code to split up the asteroids into 8 chunks (test case of an 8 core chip), and gave each processor a list of asteroids to process. Result? SLOWER. Actually quite a bit slower. Some fiddling with AQTime (My profiler) let me analyze cache misses for each thread, and I also profiled it as 1 thread. The cache miss rate went through the roof. Basically, my transform code was relying on some global camera data, and I suspect that either:

a) Referencing he camera data was a bottleneck with each thread blocking each other from getting it or…

b) The memory locations of the asteroid transform data was laid out in such a way that all the different threads kept fighting for the same cache lines and generally getting in each others way.

I spent a lot of time reading and fiddling and decided that it wasn’t working (although did manage a decent few speedups in other ways). I then decided that if lots of threads sharing the same job wasn’t going to help, maybe lots of threads doing different (unrelated) jobs would…?

And this is more of a success. I have a function called ProcessFrame() which does a lot of non-directx stuff, such as the aforementioned asteroid transforming, updating engine glows, updating explosion plumes, particle effects and distortion waves blah blah… Until recently, it just did them one after the other. I then realized that although a lot of them accessed the same data (camera position stuff mostly), none of them altered it, and the tasks were quite discrete.  So I packaged them up and sent them to different threads, and then spun in the main thread waiting for them to finish. result? 21% faster. yay? not bad, but not 800% faster, which would have been theoretically do-able(not really but…)

Of course the missing link was that I am then left waiting for the slowest thread. Plus if I have more than 8 tasks, I run out of CPUs. So I re-coded it to have a queue of tasks, and when a thread finished a task, it checked the queue, and only reported it was done when the queue was empty. This was way more efficient, and easier to scale to available cores. Result? 41% faster!

Now obviously a 41% processing speedup is good (although this is pre-render, not render, so probably only a 20% FPS boost) but I can’t help thinking that if not 800%, a 200% speedup of that bit of code must be possible. Debugging cache-misses is hard, as even aqtime will bluescreen occasionally on windows 7 when profiling it. I’m pretty sure it’s some cache, false-sharing issue going on.

In the meantime, GSB is now faster as a result, even if i spend no more time attempting to multithread it (and i will… I’ve only just got going). Anyone else attempting this sort of thing?