Game Design, Programming and running a one-man games business…

Solar farm progress update: Pre-installation

So, it turns out there has not been a proper solar farm update since this one, back when we finally got the planning permission. It probably feels like nothing has happened since, which is more accurate than you might think, but stuff is changing much faster now, so time for an update!

For anyone new to the blog, I own a small company that is building a 1.2MWp solar farm in England as a side-project to making video games. 1.2MWp is tiny by solar farm standards, but it will power 300 homes, so its not *that* tiny. This is the first time I’ve ever got involved in any sort of ‘infrastructure’ project, so I’m learning as I go along…

I knew back in October when we got permission, that nothing was going to happen straight away, because any construction site that is in that part of England, in winter, in the rain and the mud and the rain and also in the rain… is basically untenable. I can not remember the exact number of large truckloads of equipment we need to get to this field, but I know its more like 30 than 20. Put that together with all the manpower that has to get there, and its just going to be an absolute mudslide of horrific proportions. When I first visited the site, the landowner met me on a quadbike, and I can see why…

(Imagine this…but with a thousand billion times more rain and mud)

Anyway, there was a lot of other stuff that needed doing before we actually stick things in the ground, so there has actually been some non-visible progress. Most of the progress was, sadly, in things I didn’t even know we needed.

The contractor who will build (and likely operate/maintain) the solar farm basically builds an array of low voltage (about 400v) solar panels, all of the inverters and the cabling to a substation. Thats basically all they do. They know how to wire up a LOT (3,024) of solar panels at a voltage of about 400v. Thats considered low voltage. The actual UK power grid at this size, tends to be operating at 11,000v (11kv) or 33kv or even higher. That means you need a transformer that steps up the voltage to 11kv, and switchgear that handles all that stuff. You then need to cable it from your substation to the DNO substation, which is where their responsibility begins.

It feels weird to me that this means 2 substations, but apparently it does. It also means that you need a high-voltage expert to design and handle all of the 11kv stuff, which goes beyond ‘ouch that hurts’ and right up to the ‘all that was left was the flaming boots’ level of danger, so I guess its no surprise you need a 3rd party expert for that stuff. Also I discovered that big transformers are full of oil. Who knew?

So…stuff has now got more complicated because the new structure of this project is now:

  • Positech Energy (my company)
  • The site construction contractor
  • The landowner (farmer)
  • The Distribution Network Operator or DNO (Grid people)
  • High Voltage Consultant

No doubt during construction a bunch of other 3rd parties will be subcontracted to the construction contractor, but thats their problem. Also, eventually there will be an insurance company, plus the people who buy our electricity through a PPA (Power purchase agreement). Hilariously, that may happen through a PPA broker, so there may be yet another party involved. Are you starting to see where your electricity bill goes yet? :D

Anyway…so we now have a high voltage consultant who I deal with directly, but who has to liaise with the construction company and also the DNO, and this involves hilariously complicated email chains where everyone is waiting for everyone else until I step in and yell WHY IS NOTHING HAPPENING, but hopefully in a politer tone :D.

So…moving on from the fact that we now have a high voltage consultant, lets talk about all the other sudden expenses and complications. Ladies and gentleman, I introduce you to the excitement of pullout-tests. Apparently, you have to do pullout tests. Who knew? This is basically paying some engineering experts to go bang some metal posts into the field at a variety of points, then do their best to pull them out again, and measure how much force is required to pull them out. They use special machines and gadgets:

You get a rather tedious number of photos like this, and the eventual TL:DR; of it is that ‘its fine’ which is good to know, albeit a very unfulfilling way to spend £6,795. Yes really. You can buy a very nice guitar for that. Why are pullout tests needed? well they determine what kind of ground mount kit you will need to keep the panels in the ground. Why so important? well solar panels are actually pretty light, and catch the wind like crazy, and we have over 3,000 of them. We are basically anchoring a giant expensive sail to the top of a hill in the midlands. If we don’t secure these puppies nicely, we are just one storm away from them all turning up in Liverpool one day.

Anyway, after all the pain and stress of paying for pullout tests, comes the new excitement around earthing studies. Yes earthing studies. Did you not know solar farms need to be earthed! haha, you fool. Yes of course, we all knew that *loud coughing*. Its to stop a big lightning strike pulverizing the whole farm. Did I mention we are sticking 60 tons of solar panels on the top of a remote field? Anyway, in order to work out HOW to earth the farm, you need to know the resistance of the soil, which apparently, is a thing that varies.

A lot.

Anyway, thats of course another expense, and its all very depressing, and it takes time. Not only that, but TWO completely different companies carry out earthing designs for separate parts of the installation. The DNO (grid people) design the earthing for their substation, and our HV contractor designs the earthing for our bit. Its totally separate. This seems nuts to me, and ripe for optimisation, but I’m not an electrical engineer so maybe there are reasons for this.

Anyway, it means you get sent exciting diagrams like this:

I think we all understand this right? *louder coughing*. It also means you get a design for how all the fancy underground earthing stuff will be arranged, which is apparently generally in a grid pattern, presumably because you are earthing a LOT of stuff, and a single earth rod stuck in the ground isn’t going to cut it:

Oh, in other news I actually have the final quote for the install of the farm. I’ll give a proper breakdown once the farm is constructed, but you will enjoy the hilarity of reading about when I got the quote at 6PM on Friday, with a VERY ambiguous sentence in it, that led me to believe the farm was going to cost £450,000 more than I anticipated. I did not sleep at all Friday night, and was *relieved* on Monday when this was clarified.

Oh how we laughed

The final hilarious piece of news is that due to changes we had to make to the plant layout between getting planning permission refused, and then granted, I have 200 more solar panels than I need. I am still being charged monthly to store them, and this is not fun. I THINK I might be able to stick 8 of them in my driveway to replace my old 12 year old ones, but I’m not sure about that yet. The others will likely be sold to the construction company, at cost (still a net loss as I’ve stored them for 6 months+).

Anyway… the short version of this blog post is that there is PROGRESS. We are anticipating making the first 30% payment to the construction company today, which will mean we can plan actual install dates, and actually order the ground mount kit, inverters and other stuff like cabling. We will use 15km of DC solar cable, for example. Thats a lot of cable.

The good news is that stuff looks like its starting to actually happen now. I have been told that the cutout-switch has already arrived to the town next to the farm, and some other equipment used by the grid connection company has been ordered already. This is a very *good sign* as it means the grid connection will happen this decade, at least.

TL:DR; Gamedev is easy. Building stuff is hard.

Economics of solar batteries (big and small)

I’m lucky enough to be the proud owner of a 9.5kwh battery in my cellar. Its a GivEnergy battery (A UK company, although the actual battery is predictably made in China). Its AC-coupled, which means it sits on the ‘house’ side of the fusebox/consumer unit. We got it installed about 5 months ago and its been super awesome. Not flawless by any means, but its just so cool as a gadget for someone like me.

There are many reasons to get a home battery. Its a cool gadget, its also an incredibly strong way to reduce your energy bills (we basically run our whole house 24/7 on off-peak electricity at 75% off), and its also a great thing to partner-up with solar panels to ensure you use all that free power and don’t go exporting it to the evil energy company for a pitiful rate. I have to admit, although I was fully aware that we exported a lot of power on those days we were out, or we were not running much stuff, I had totally underestimated the impact. I’m currently running a big desktop PC/Monitor, router, wifi boosters and a bunch of other stuff, all from solar, on a cloudy day in march in the UK, AND filling the battery slightly…

Obviously the real reason to get a home battery is because you get to obsess over charts and stats:

I’m going to talk a bit more about the economics of the battery though, rather than the coolness of it, or its usefulness in balancing the grid and helping enable more renewable energy.

The economic bonus of my battery is it lets us buy off-peak electricity that costs (currently) £0.12 per kwh as opposed to peak power which costs £0.43 per kwh. In other words, each unit of power we use saves us £0.35. We use roughly 600kwh per month, but I reckon half of that is for the car, and can be done off-peak through charger scheduling regardless of the battery. So that leaves 300kwh a month or a saving of £105 a month, or £1,260 per year at current electricity rates.

There is an additional economic benefit though, which is that we are self-consuming all of our export. To look at how we manage this, I just need to look at the total solar->battery flow for the last month, which is about 40kwh. Thats energy that would previously have been wasted (exported to the grid, but not metered, so the amount doesnt matter). Thats another 40 * 0.43 so £17.20 a month, or a total of £206 a year.

So the income from our battery, at current prices, is 206 + 1260 = £1,466 a year. The installed cost for the battery was £6,720. So the payback period is 4.5 years. Thats really not bad, given the battery will definitely outlast that. This is also with a pretty rubbish old array of terrible 12 year old solar panels. So it seems like the economics of home battery storage are pretty good.

However…

Now lets take a look at grid-level storage, and the size that I was considering. For those new to my blog, I run an energy company that is building a 1.2mwp solar farm here in the UK. Originally we were considering battery storage, but dropped the idea. We were originally looking to add a 500kwh, later increased to 583kwh battery, in a shipping-container enclosure that was AC-coupled to the farm. We are export-limited to 900kw despite peaking at 1.2kw, so this would be a good way to do peak shaving. I blogged about how that works in detail here.

So why are we not doing it? Basically its the economics. They just do not work, for that scale, at the current time, for a variety of reasons. The main one is that the costs have gone up, but also the cost of larger storage systems have gone down. It looks like economies of scale are massively at work here.

From what I understand, a 583kwh battery unit like the one pictured above is basically deployed as a kit. You get a lot of equipment, including pallets full of batteries, and a shipping container enclosure for all the gear, and then its assembled on site to create a finished, grid-connected 583kwh battery. The installation of this is about £28k out of a total installed cost of roughly £370k.

Lets pick apart the finances of this!

Firstly, a 583kwh battery is the same as 9.7 Tesla model 3 standard range batteries. A brand new Tesla model 3 standard range in the UK costs £43,000. 9.7 of those cars costs £417,000. That is MORE than the grid battery…but Jesus Christ its not MUCH more, and the 9.7 cars come with stuff like wheels, doors, touchscreens and…other car stuff. There is NO WAY that the cost to Tesla for making/buying the batteries in a model 3 are anything close to as high as people are being charged for a grid connected battery.

Whats going on?

Actually, the economics seem to be the case that grid-connected utility storage is being manufactured in a grossly inefficient way, and hugely overcharged for, especially by companies not called Tesla. Lets take a look at the real core of the problem by looking at a battery in a Tesla:

The actual batteries are the little cylindrical cells. They are being slowly changed to be a different configuration, but the same design principles apply: pack the batteries in with as efficient a volume-utilization as possible. Note that its actually much more safety critical in a car to prevent battery fires than in a shipping container sat in a soggy field, so if anything, this is the ultra-conservative and inefficient Tesla battery pack. Also…its a car, so weight REALLY matters (whereas with stationary storage we absolutely don’t care about weight), so thats another constraint. What we are looking at here then, SHOULD be the really inefficient way to do things.

But how are things done for grid scale utility batteries? Well… let me show you :D. This is a lithium ion battery for grid storage:

Thats an 8.1kwh battery. Not enough? It doesn’t matter because we can put them in groups in a rack!

Thats still not enough though, so we can then put the racks in a box!

And so it goes on, with what seems to be a Russian-doll approach to stacking boxes inside boxes inside boxes, until the amount of casings, handles, surrounds, spacers, racks, shelves and other non-battery components becomes ridiculous. Worse still, it seems that a lot of these are still being *assembled on site*. This is, as I am sure you must be aware, ludicrously inefficient.

This is the Tesla Megapack:

Its still racks-in-a-box, for now, but its not built on-site, or in a ‘room’ where you can walk down an aisle. its a big solid block of batteries (plus supporting equipment) that is manufactured like this at their factory. The whole thing is shipped as a single already-connected block, that you then hook up to your grid connections. No pallets, no wasted space. No lengthy install.

This is just a lot of tech-geek inside baseball until you realize the economics of it. The best way to find out the *real* price for these things is to look at recent installs. There was a recent one in the UK: https://driveteslacanada.ca/news/europes-largest-battery-energy-storage-project-opens-in-uk-using-tesla-megapacks/ with 196 megapacks for £75million. That comes out at a per-MWh of installed battery cost of £222k. Thats 42% cheaper than the quote I got for a 583kwh pack.

So the answer is simple right? Just ditch the battery provider you got a quote from and use Tesla? No. The megapack STARTS at 3MW. Thats a £666k installed cost, on top of the solar farm. It *might* make sense if the output of the grid connection was higher, and the farm itself was larger, but nobody would couple a 3MWH battery with a 900kw limited 1.2kwp farm. It would just be idle too much of the time to justify the expense.

We are still in the early days of grid-connected storage. Its accelerating like absolute crazy right now, so we are getting there…but there is a lot of change happening. As an investor in green-tech, I cannot see any other grid storage company surviving after the next 2 years other than Tesla. (The megapack is not surprisingly sold out for the next 2 years). The grid-storage business is VERY price sensitive. You cannot just shrug your shoulders with a 42% cost difference…

So how does that leave me? Well it leaves me building a solar farm with, for now, no battery storage (although with planning permission to add it later if the numbers change). I would love to revisit it when BESS prices drop, and definitely will. I think we are going to see a real push to make those systems cheaper, and thats just based on design optimization and economies of scale, not even considering better battery chemistry or any tech breakthroughs.

For home storage though, its a laughably good deal. If you can afford a home battery, get one.

Every band I ever saw live (I think…)

I keep meaning to do this, and this blog is as good a place as any. I used to go see a lot of live music, overwhelmingly its heavy metal, but theres a lot of other stuff in there. I will probably not remember all of em, but here goes:

Metal

Ozzy Osborne, Judas Priest, Iron Maiden, Rush, Dream Theater, Van Halen, David Lee Roth, Bon Jovi, Winger, Kings X, Extreme, Mr Big, Crimson Glory, Queensryche, Motorhead, Wolfsbane, Little Angels, Leatherwolf, Vow Wow, ACDC, Skid Row, Loudness, Saxon, Spider, Manowar, Phantom Blue, Alice Cooper, Zodiac Mindwarp.

Thrash Metal

Metallica, Testament, Anthrax, The Stupids, Cancer, Deicide, Acid Reign, Onslaught, Skyclad, GWAR,

Guitar Bands

Joe Satriani, Steve Vai, Yngwie Malmsteen, George Thorogood.

Punk

The Cult, The Damned, Killing Joke, Siouxsie and the Banshees.

Goth

The Mission, The Cure, Dr & the Medics, Hawkwind, All About Eve.

Pop/Rock

Coldplay, Sting, Jeff Beck, Howard Jones, Simple Minds, Status Quo., Bonnie Tyler, ZZ Top, Pink Floyd.

I probably forgot a whole bunch of gigs here. I saw a lot of goth/punk stuff, but was always drunk and cannot remember most of the names. Also, some of these bands I saw a lot. Ozzy Osborne at least 4 times, Vow Wow at least 10, Dream Theater maybe 10 times too. I’m probably forgetting a bunch of support bands too. I may have seen New Model Army and Fields of the Nephilim and some similar bands, but honestly that whole decade I was drunk and kinda not sure whats a real memory.

Bands I never saw but wish I had: Guns N Roses, Slayer, Queen, The Police, Ritchie Kotzen.

Speeding up my game from 59fps to 228 fps.

I recently saw a comment online that the ‘polls’ screen in Democracy 4 was horribly slow for a particular player. This worried me, because I pride myself in writing fast code, and optimizing to a low min spec. The last thing I want to hear is that my game seems to be performing badly for someone. I thus went to work on improving it. This involved about 15 mins looking at the code, about an hour musing, while trying to sleep, and about 20 mins coding the next day, plus an hour or more of testing, and profiling. Here is what I did, and how I did it.

The game in question is the latest in my series of political strategy games: Democracy 4. its a turn-based 2D icon-heavy game that often looks a lot like a super-interactive infographic. Here is the screen in question, which shows the history of the opinion polling for all of the different voter groups:

There is actually quite a lot of stuff being drawn on that screen, so I’ll explain what is going on, in ‘painters algorithm‘ terms.

Firstly, there is a background screen, containing a ton of data and icons, which is displayed in greyscale (black & white) and slightly contrast-washed out to de-emphasize it. This is non-interactive, and is there just to show that this is, effectively, just a pop-up window on top of the main UI, and you can return there by hitting the dialog close button at the top right. This is drawn really easily, because its actually a saved ‘copy’ of the backbuffer from earlier, so the whole background image is a single sprite, a quad of 2 triangles, drawn in a single draw call, filling the screen.

Secondly I draw the background for the current window, which is very simple as its basically a flat white box, with a black frame around it. This is actually 2 draw calls ( TRIANGLESTRIP for the white, LINESTRIP for the frame).

Then I draw a grid that represents the chart area, as a bunch of single-pixels lines. Thankfully, done efficiently as a single big draw call.

Then for each of the poll lines I do the following:

  • A single draw call for a bunch of thin rectangles representing the faded out lines from pre-game turns
  • A single draw call for a bunch of circular sprites representing the dots for pre-game turns
  • A single draw call for the lines from player-turns (full color)
  • A single draw call for the dots for player turns (full color)

Then a bunch of other stuff, such as the filled-progress-bar style blue bars at the right (all a single draw call) and then text on them, and then the title, the buttons at the top, and the close button.

In total, this amounts to a total number of draw calls for this screen of 102. This is almost best-case though, because its possible for that grey bar at the bottom to fill up with a ton of icons too, which would make things worse.

Now…you may well be sat there puzzled, thinking ‘but cliff, who cares? 102 draw calls is nothing? My FragMeister 9900XT+++ video pulverizer can apparently do 150,000 draw calls a millisecond’. You would be correct. But let me introduce you to the woes of a ten year old laptop, and not a bad 10 year old laptop, a pretty hefty, pretty expensive HP elitebook 8470p, which was inexplicably a gift to me from intel (thanks guys!), and came with their HD 4000 graphics chip.

I appreciate free laptops, and the intel GPA graphics debugging tools are incredible, but the horsepower of the actual GPU: not good at all. It could only manage 58fps on that screen. Assume some extra icons at the bottom, and assume a poorly-looked after laptop with some crapware running the background, and we could easily drop to 40fps. Assume a cheaper-model laptop, or one a few years older, and we suddenly have bad, noticeable ‘lag’. Yikes.

When you delve into the thrills of the intel Graphics Frame Analyzer, the problem is apparent. This chip hates lots of draw calls. It hates fill-rate too, but the draw calls seem to really annoy it, so its clear that I needed to get that number down.

The obvious stupidity is that I am drawing 21 sets of lines individually instead of as a group. There was actually method to the madness though. The lines were composed of lines AND dots, and they were being drawn differently. Each thick ‘line’ between 2 dots is actually just a flat-shaded rectangle. To do this, I create a ‘sprite’ (just a quad of 2 triangles), with the current texture set to NULL, and draw it at the right angle. Draw a whole bunch of them and you get a jagged thick ‘line’. The dots however, use an actual texture, a sprite thats a filled circle. To draw them, I need to set the current texture to be the circle texture, then draw a quad where the dot should be.

Luckily I’m not a total idiot, so I do these in groups, not individually as line/dot/line/dot. So I draw 32 lines, then I draw 32 dots, and thats 2 draw calls, one with a null texture, one with a circle texture. Actually its WORSE, because I make the mistake of treating the pre-game (simulated) turns as a separate line, because its ‘faded out’. This is actually just stupid. When you send a long list of triangles to a GPU, the color values are in the vertexes, you don’t need to swap draw calls to change colors! but hey ho.

Clearly when I wrote this code, I thought ‘well one uses a null texture, and another uses a circle texture, so this needs to be 2 separate calls’. and because I cant even make a single dotted line 1 call, no way can I make the whole thing 1 call right?

This was stupid. I could have changed my ‘circle’ texture to be 2 images, a circle, and a flat white quad, and change the UV values so that the dots used the circle, and the rectangles used the flat white quad, but there was actually an even easier method. I just had both systems use the circle texture, but set up UVs on the rectangles so that they were using only the very very center of the circle sprite, which was effectively all white anyway…

By setting those UVs, I can avoid having to use a NULL texture. An all-white texture is exactly the same thing. This way, all the lines and dots are exactly the same thing, its just a big long list of sprites, which means just a big long triangle list, which means a single draw call. And because its just 1 draw call, that means it can merge with the next line, and the next. So instead of doing 21x2x2 = 84 draw calls, I can just do one. Just one. Awesome.

Amusingly, in my actual code, I was sloppy and didnt even use the matching UVs perfectly, but it doesn’t matter:

/////////////////////////////////////////////////////////////////////////

void GUI_AnimatedGraphLine::RenderThickLines(V2* ppoints, int countpoints,
 float width, RGBACOLOR color)
{
      BaseSprite sp;
      sp.SetUV(0.49f, 0.48f, 0.51f, 0.51f);
      sp.width = width;
      sp.SetVertexColor(color);

Just making this one change made a ridiculous difference to the frame rate for this screen, sending it from 89fps to 228 fps, a completely huge win. It will be in the next update to the game.

Its absolutely not the final word on speeding up that part of the UI though. I have some very un-optimised nonsense in there to be honest. Those last little lines at the right of the chart, that connect the plotted lines to their matching text bars…. those are done as a draw call separate from the main graph. Why? Simplicity of code layout I guess. There is also a LOT of wasted CPU stuff going on. The position of all those lines and dots is calculated every frame, despite the fact that this window is not draggable or resizable, and they never change. I am 99% sure the game is always GPU bound on screens like this, so I haven’t bothered looking into it, but its still a waste.

According to the GPA Frame analyzer, the BIG waste of time in this screen is definitely the fact that I have massive overdraw going on with this whole dialog box. The window is a perfect, simple rectangle, so its easy to work out how much fill-rate was wasted drawing that background image behind it. There is zero opacity, so it would actually be fine to cookie-cutter out the top, bottom, left and right of this screen and then only draw the sections of the background image I needed. That could boost the FPS even more.

In practice though, we always have to strike a balance between code readability and simplicity, and the target of fast code. I could probably code all sorts of complex kludges that speed up the code but cause legibility hell. Arguably, I have made my code more confusing already, because there is no simple bit of code where ‘the dotted line for this group gets drawn’. There is code that copies some data into a vertex buffer…and then you have to trust I remember to actually draw it later in the screen…

Luckily I’m the only person looking at this code, so I don’t need to debate/explain myself to anybody else on this one. I’ll take happy players of old laptops over some minor inconvenience regarding following the code by me later.

I’m always surprised by how seemingly simple code can be slow on some GPUs. Whats your experience of hardware like this? Do you have an HD4000 intel GPU? Whats it like for games in your experience?

New expansion coming for Democracy 4

I’ve been working on this a while, but today I made the DLC page public, so I should now tell everyone about it :D. I’m working on a new expansion pack for Democracy 4, coming real soon now. here is the blurb:

Democracy 4 – Event pack adds 45 new events to the game.

Some of the new events act as warning signals. Maybe your push for more technology is going to lead to sudden unemployment as AI replaces jobs? Is pollution is now so bad that schools are warning children not to play outside? A famous celebrity getting car-jacked in broad daylight is a sign that crime is out of control! Other events may shake things up when you thought they were under control, such a sinking oil tanker, a collapse of the power grid or staggeringly cold winters. If you are fortunate, some random events may change your country for the better, such as a super-productive harvest, medical breakthrough or a breakthrough in trade talks.

Give Democracy 4 a boost with these extra events to liven up the game. Will you still be able to maintain control of the country, and command the respect of the voters? or will your term in office be derailed by the tragic course of events!

…so thats the blurb. Still testing right now, but hopefully releasing at the end of this month. I updated the DLC page here.