Programming RTS Unit selection outlines is a pain. I had an amazing GUI mocked up by an artist, and most of it is now in Gratuitous Tank Battles, but today’s todo list included unit selection outlines. He had mocked up this:
And now I want to batter him with a baseball bat.
(I always tell artists to do the best stuff they can possibly imagine, and let me worry about how I’ll make it work. Reach for the stars etc…)
You might think it’s easy. Just draw the image bigger first, with a flat shaded effect (I use render states, but YMMV), and then draw the unit on top. WRONG!
Firstly, that means your UI doesn’t shine through smoke or other effects layered on top, which isn’t as cool. Secondly. it means the units shadows are cast onto its own selection UI (yuck) thirdly, it just plain doesn’t work. It works with squares or circles or other basic shapes, but take a complex image, scale it up, then draw the smaller image inside it. See what I mean?
What I really need is some way of doing what photoshop calls the ‘stroke’ effect, where the outline of an image is expanded. Not expanded directly from the image center, but expanded in all directions.
One solution mentioned online is to draw the flat-shaded bit (enlarged selection) 4 times, moving it up down left and right by 1 pixel each time. That’s great if you want a 1 pixel outline, but 1 pixel sucks, and beyond that, you will get corner issues, plus… 4x rendering potentially every unit in your army every frame is not nice.
Another solution would be to have extra versions of each sprite, already-scaled up in photoshop, and render them for the enlarged versions instead. This has issues where the image already touches the sprite corners, and in any case, that means that the selection outline is a fixed percentage of the unit size, rather than a uniform 4 or 8 pixels, which would look tons sweeter.
So… how did I fix it?
I haven’t yet… It’s driving me bonkers :( I am considering an offscreen render target that blurs a matted sprite, but that wouldn’t be crisp, which I think would look better. I wonder how they did those outlines in age of empires 2?
Note, almost all discussions online about this refer to 3d meshes. I use 2d sprites with alpha channels. Totally different :(
edit: this is what I have so far, quite like it, may compromise on it…