Category Archives: production line

How long does your indie game take to start up? from clicking the icon to actually being able to take input at the main menu? Just for fun, I decided to analyze whats involved in doing so for mine.

Because the aim here is to actually analyze the REAL impact, not the best case, I need to ensure that the game (Production Line) is not just happily sat there all in RAM from a recent run-through, so it seems best to oh…maybe launch battlefield V beforehand (and quit it) just to populate disk/RAM with a load of other stuff and do my best to evict all my games code.

Then…its time to fire-up aqtime and take a look. I decided to do line-level, rather than just function-level analysis, which slows the game massively, taking 17 seconds to start (the reality is dramatically faster), but I can still do relative comparisons.

First thing to notice is that pretty much the entire time is inside Game::InitApp() which makes sense.

Rather worryingly though, the vast majority appears to be inside SIM_Threadmanager::Initialise. That *may* be an artifact of aqtimes approach to thread profiling, but worth taking a look inside anyway… And it turns out that 100% of that time is inside SetThreadName() (which i only need for debugging anyway). This is a rare bit of code that I don’t understand well, and was from the evil interwebs:

#pragma pack(push,8)
typedef struct tagTHREADNAME_INFO
{
	DWORD dwType; // Must be 0x1000.
	LPCSTR szName; // Pointer to name (in user addr space).
	DWORD dwThreadID; // Thread ID (-1=caller thread).
	DWORD dwFlags; // Reserved for future use, must be zero.
} THREADNAME_INFO;
#pragma pack(pop)

void SetThreadName(DWORD dwThreadID, char* threadName)
{
	THREADNAME_INFO info;
	info.dwType = 0x1000;
	info.szName = threadName;
	info.dwThreadID = dwThreadID;
	info.dwFlags = 0;

	__try
	{
		RaiseException(MS_VC_EXCEPTION, 0, sizeof(info) / sizeof(ULONG_PTR), 
(ULONG_PTR*)&info);
	}
	__except (EXCEPTION_EXECUTE_HANDLER)
	{
		volatile int foo = 9;
	}
}

The exception is basically ALL of the time. WTF? Apparently there is a less hacky way outlined here: https://docs.microsoft.com/en-us/visualstudio/debugger/how-to-set-a-thread-name-in-native-code?view=vs-2019 Which I will try later. I suspect the waiting for visual studios debugger is the cause of the problem.

Anyway…onwards and upwards, so whats next? It basically all Init3D() (click to enlarge)

So basically my DirectX initialisation and the shadermanager stuff is most of the problem. I suspect the DirectX initialisdation may be too black-boxed for me to influence further. The first big chunk is this line:

PD3D9 = Direct3DCreate9(D3D_SDK_VERSION);    

Which takes up 34.74% of the start time. The next slow bit is the largest at 41% which is:

hr = PD3D9->CreateDevice( AdapterToUse, DeviceType, WindowHandle,
 D3DCREATE_SOFTWARE_VERTEXPROCESSING, &PresentParameters, &PDevice);    

So…holy crap. how can that line of code even be run? This can only happen if my checkcaps() code suggest the video card does not support hardware transform and lighting. I suspect some of the reporting here must be nonsense? Especially as my own debug logs suggest that the hardware TNL version is the one than ran… FFS :( lets look outside that code then…

Most of the slowdown is in shader manager, which loads 11 shaders:

so it looks like about half the loading time here is actually spent writing out debug data! This is hard to avoid though, as I do find this data invaluable for detecting errors. And because an app can crash and lose all its data, I flush each line of my debug logs to disk with a hard drive flush on each line…

…so interestingly all the time seems to be inside OutputDebugString, which is only of any real use when the debugger is running. However! I *do* want to see that data in both release builds, and debug builds. Maybe I need a flag to tell if a debugger is present when the debug engine starts up? As a first pass I should at least build up a char* with the newline in to avoid twice the OutputDebugString calls. Here is the new code and timings.

Ooooh. I’ve halved the time of it. I’ve done the same with my non-directx debug code too. Now I’ll try changing that thread stuff… It turns out that SetThreadDescription is windows 10 only, so I need a different system (and apparently would need to update my platform SDK…urrrgh), so maybe best to just skip calling that code when no debugger is detected?

This works (using isDebuggerPresent) but the profiler actually trips that flag too, so to set it work I needed to compare time stamps on debug files. Without the debugger, it looks like time from app start to menu ready is 0.395 seconds. With the debugger its… 0.532 seconds.

That sounds pretty small, but actually I’m quite happy as I lost ZERO functionality, and the changes to the debug data will affect any time that data is written, not just during startup. 9Its not a lot, but there is *some*, and I’m an efficiency obsessive.

I think I’ll put a clause around the debugengines OutputDebugString and nuke that unless IsDebuggerPresent() too :D

There are no comments yet

Today is the day! At last someone has taken the time to think of the billionaires… We release an expansion for ‘Production Line’ today, which adds the following features…

Support for Gull wing doors like these:

And for Scissor-wing doors like these:

And also Butterfly-wing doors like these…

And not least a brand new body type called ‘supercar’. It looks like this:

This is the first DLC for Production Line, and it will be available direct from us at www.positech.co.uk/productionline/dlc.html as well as the humble store, GoG and Steam. its $4.99. I did get in touch with a bunch of youtubers and streamers and sent out some keys, but if you want to see exactly how it all works with my smiling face involved, you can check out this blog video I made about the expansion here:

I honestly have no idea if this DLC will be of interest to just a few super-car geeks or to everyone playing the game, so its a bit of a gamble, but I do like having a stupidly expensive car to build myself, and I definitely think it adds to the late game, and adds to the experience for people who want to build super-sprawling factories with some incredibly high-end options.

The DLC actually launches at 9.00AM PDT which is apparently 5.00PM where I am in the UK. here is the link to grab it or wishlist it:

There are no comments yet

So unless you have not heard, i’m making an expansion pack for my car factory game ‘Production Line‘ which adds a new car type (supercar) and also some new door options as features for various car models, and those door types are Scissor-wing doors, Gull-wing doors and butterfly doors. I’ve done a video that shows these off (below) and also we have a web page for the dlc that you can find here.

I thought I’d put down in writing how this came about and why this was the DLC I decided to do. Personally I’m a fan of DLC, but only if its actually ‘adding’ to the game, or giving the player options, rather than just tweaking some numbers and selling a few text files for $5.99. That seems a bit exploitative, so I always focus on doing DLC where we have new art content at the very least. There is a LOT of art that goes into a single car body design in Production Line, because every car is completely different in terms of where the wheels are, what the headlights look like, even what the axles and drive shaft looks like and where they go. Nothing currently gets simply re-used between body styles.

Adding a super-high end car to the game makes sense from a design and balance point of view, because it means we can leave the balance of the existing game in place, and effectively just add a new style option for the luxury cars, and add a tiny section of the public who only want to buy a super-car shaped vehicle. In the real world, most sales at that price point already are accounted for by people buying sedan and sports style bodies, but there are some people who would want to spend the same budget on a car with a crazy shape.

Plus crazy designed cars are cool right?

Once I knew we wanted a supercar, having crazy doors is pretty much a no-brainer. people love cars with mad doors, and some research showed that butterfly, gull and scissor were the most common options on modern real world supercars. In theory putting all this in the game was going to be mostly artwork, and then some balancing by me. pretty easy huh?

It turns out..NO! This was extremely tricky because of the way the design of those doors overlaps other parts of the car when they are open (which involved a lot of re-ordering of existing components), and the nightmarish fact that gull wing doors actually require a different ‘roof’, to allow for the cutouts n the side for the door hinges. This means I had to redefine these door ‘styles’ as roof types, which also ensures you cant have gull-wing doors and a panoramic sunroof…

As a result, door styles have to be like drive-trains: set in stone when the car chassis is first created and unchanged for the lifetime of that existing chunk of metal. In game-terms, this means that switching your luxury sedan to use gull wing doors will not have any visible effect for a while, as the cars right at the start of the line are then set to be gull-wing, but they have to get all the way to the door panel slot for you to notice.

This also involved a huge amount of special case code-spaghetti, because if you have gull wing doors, then certain other things change. You cannot stick normal windows or wing mirrors on a gull-wing door, and so-on. There was quite a bit of complexity involved in swapping some components for others to accommodate the new system.

I like to give my artists total freedom to create cool stuff and tell them I will worry about making it work. I get better art that way, but more work for me… It turns out that yes…its very cool that the supercar’s engine is behind the driver, but that caused me some hassle to ensure that when the engine gets fitted, its the back part of the car (trunk) that opens up to have it fitted, and not the front… (99% of players wont notice, but I like to surprise the 1% who do). As we were doing this anyway, it made sense to support the same capability for the sports cars, which also look like they have the engine in the back.

So all of that stuff is now DONE and tested and working in the game. The installer is done, and I’m working on the steam build, and mod-panel integration. All I need to do before release is keep testing the balance of this stuff to ensure that the doors are expensive and awkward to fit (but not unprofitable or too annoying) and that demand for these features and for the supercar makes sense.

Right now, the game is being tested with supercar sales only taking place in the luxury price band (well yeah…) and with a market size of 4.76%. I think thats maybe a bit high, and obviously in the real world its <1% but this is a game and needs to be fun :D

In terms of balancing the doors, they are currently just ordinary door panels..BUT the slot-upgrades are pretty expensive ($195,000 for a slot upgrade for scissor doors compared with just $3,480 for central locking) and they require sensors and servos, which lead to a bit of a bottleneck (intentionally). Plus the actual fitting time to add those doors is considerable, more than doubling the max-optimized time to fit ordinary door panels.

The final piece in the puzzle is the design cost. All door types need to be researched separately and also you need to research the new supercar body style. These are quite lengthy research items, but there is nothing stopping you skipping all the other car bodies and making supercars or gull wing doors your early game priority!

If you like the sound of all this stuff you can wishlist the DLC using the widget below.

If you are a youtuber looking to cover the game and want to make a key request you can get in touch through keymailer (below) or email me your steam curator name.

There are no comments yet

Its always worth occasionally grabbing a full run-time-profile of your game to see whats taking up all that processor power. Today I dug into it a bit with aqtime and wanted to see what was chewing up the PC the most on a (fairly) big map.

If I ignore child threads (that take a lot of the load-off in terms of route-finding and other things) and just concentrate on the main thread, I can see that 73% of a frame is in the drawing code, and 22% is processing. I suspect on crazy big maps that processing chunk goes higher, as the drawing code scales to some extent with the screen size, and a big map on a small laptop is still my target for improvement.

If I break open that 22% I get this:

SlotManager::Process is obviously the biggest problem here. Its also something that scales quite badly because on really jam-packed maps, there is a lot of slots, and for my sins, EVERY item placed down is a slot, even if its just a conveyor with one entrance and one exit… so whats going on in there…? 95% of the time happens in ProductionSlots (the rest is facilities or supply stockpiles…

…so quite a mixture here, which is a good sign really, as it shows no real obvious glaring screw ups! Because I’m really worried about maps with just a lot of conveyors on them I guess the thing I really need to look at is Processvehicle(), as the other top offenders don’t seem to be related to those theoretically simple slots…

Hmmm. so CheckExitSlotFree() is taking up a lot of time, and some digging around shows that in my short sample run I called that on average of 1,248 times per frame. Ouch. Thats a lot, but to be expected with so many slots, and them all wanting to know their immediate status. The sad thing is, this is likely to be 99.5% of the time, a lot of slots going *is the slot in front of me empty* and the answer coming back *no, just like it was the last 100 times you asked*.

There is a fundamental algorithmic screw-up here, in that polling a value, rather than being notified when it changes, is hugely inefficient. Its also incredibly reliable and fault tolerant, which is why its such a popular method… What exactly takes so long anyway in that function?

Aha… so it looks like a big chunk of the time is actually spent just notifying the popup GUI above a vehicle or a slot, what message to display which is almost *all* cases is none. This seems crazy, and clearly a target for optimization. On its own, this function is taking up a tiny part of total run-time, but a larger part of the main (blocking) thread, and certainly seems to be inefficient. What takes up most of the code? actually its the code that works out the width and height of any of the strings based on the UI font, and thus resizes that notice. Yikes, am I *really* calling that every frame?

Out of 1,992,097 calls to SetType() only 55,280 update it, so only 2.7%, but its still a lot. Inside that function we set the size and the text without even checking if its onscreen, or if we are zoomed in far enough to see it anyway. I can simply set that to be something thats done ‘lazily’ once I know we are onscreen and zoomed in…

Ok done that and…. Hahaha. That function now averages 0.0001ms vs 0.0007ms before the change. Thats a decent hours work :D

That reduces the overall time for a slot processing a vehicle from 0.0012ms to 0.0009ms. For big factories, thats a welcome change.

So you may have missed it, but we launched Production Line out of Early Access almost exactly a week ago. It looks like we have had a pretty good launch. We were in various charts in various categories, sold a lot of copies, got some good word of mouth coverage, and a fairly minimal amount of bug reports. I have remained relatively calm, and relatively sane, and am still motivated to improve the game and continue to do (some) work on it for the next few months. Woohoo.

TBH this is the smoothest game launch is Positech’s history. This is the first time I’ve done early access, so its the first time I’ve had literally tens of thousands of people hammering the game code *before* I officially declare the game *done*. Frankly, these days most indie games get a way bigger EA launch than a final product launch, so its not as gentle a ramp up as it sounds, but it still resulted in a pretty bug free departure from Early Access

Something I was very happy with was that I could set the ‘final’ build live for all the EA players, not touch the game *at all* for a few days, and when I was absolutely sure everything was fine, just literally click the ‘release out of early access’ button knowing that things were pretty stable. I highly recommend this!

So far we have had ONE patch since release, which fixed a short list of things, and there is another one on the way in maybe a week or so, which will be the accumulation of a bunch of bug fixes (even some pretty rare crashes) and some UI features people have asked for like a camera speed slider, autosave interval slider, and some extra stats. It feel so good to be in a position where people are saying ‘the game needs a better UI for feature X’ instead of ‘the game needs feature X’.

I definitely have plans to do some paid content for the game (DLC) alongside regular updates. I’m obviously not in a position to even tease anything like that yet, but as I prefer content-heavy DLC to code-heavy DLC (its just optimal given that I am the only coder), such things do not take *much* time to do (although there is of course a big art budget cost).

Right now I’m pretty happy with how Production Line has gone. Even if the game makes NO MORE MONEY AT ALL, I’ll be happy (but amazed!), and I think over the long tail of the next 2+ years there is a good chance of it making at least 50% of its current earnings again, which I’d be very happy with. I’m currently minded to *not* go mad with sales and discounts, as I think this is getting a bit out of control and games are being devalued, but I may think differently about that in a years time.

Anyway, this is nothing but good news, which is a change from the usual indie ‘I sold no games and have eaten my pets’ stories, but I’m not going to pretend things went wrong when they didn’t. I always blog openly about my screw-ups (2 recent games still in the RED for me :(), so I may as well be honest when things go well.

Thanks to everyone who bought the game so far!