What do I mean by unstable? Balancing encounters usually feels like trying to balance a ball on top of a hill - it tends to fall one way or the other with ease, while keeping it in balance is a feat. In RPG terms, you either end up throwing something way too easy at the players and they end up roflstomping it, or the encounter is too hard and the player characters end up dying. Ideally, you want the characters to pull through but at a cost.
For example, in our most recent Godbound game, Evicting Epistle, in 24 sessions of playing we had about 2 good encounters - one army v army, and one 3v1 brawl. A lot of other battles were either PCs wiping the floor with the enemies, or running with their tails between their legs not to get murdered.
So what issues contribute to a combat system being unstable?
High numbers with high variance
Humans are not machines, they have problems conceptualising large numbers and doing two digit arithmetic on the fly. How much longer a 65HP character will survive against an opponent that deals 8 damage than a 47HP one? Hard to tell when you have to make up an encounter on the spot.
This issue gets compounded when there is a high variance between the results (the possible range of numbers and how probable they are to appear). If an enemy does 1D10 damage, you can kill your 50HP character in 5 hits, or 10, or 37, etc. One or two bad rolls and you are out of there, or you can still be around 20 crappy rolls later, who knows? Prey to your luck deity of choice.
Having low, somewhat predictable numbers is generally better. For example, in Fellowship, characters usually have 5 levels of health and possibly a few points of armour and healing items. Enemies usually deal 1 damage. You can roughly prepare an encounter based on such small numbers easily, and players can see ahead of time if their actions will expose them to the danger of getting taken out or not.
Similarly, in Chronicles of Darkness, health and damage output of characters is usually in single digits, so you know that "this enemy that can deal 5 damage" will kill you in about two successful hits. The variance is a bit higher, but the outcomes are usually somewhat predictable and still small enough to wrap your head around.
Action economy and focusing
A lot of RPG combat revolves around everyone taking turns to perform actions. Usually the side that can take more actions (by say, having more characters) is at an advantage. If you compound this by focusing many actions against a single character (remember to always "cap the pointy hat" and go for the wizard...), you can start unbalancing the action economy more and more by removing characters from the equation. This feels particularly cheap if used against PCs since it's a cheap yet effective tactic, and it feels as if the GM has a gripe with them in particular.
In general, a lot of Powered by the Apocalypse avoid this issue by making the enemies act when a PC fails. This means enemies don't have an action economy at all, so there isn't much to unbalance, beyond figuring out how much health / damage the players have in total vs how much health / damage the enemies have, in aggregate. Five players have five times the health to soak with after all!
Another contributing factor to unstable combat is the high lethality of said system. If a character or enemy can go down in one or two hits, you are not only throwing the action economy out of balance, but you can potentially upset a lot of players by killing their characters unceremoniously.
Sure, some systems can embrace and run with it. Cyberpunk's firefights usually were one hit one death scenarios where the players knew what they sign up for whenever they drew their guns.
If you move away from "losing combat means losing a character" systems, you can open up new roleplaying opportunities - perhaps the PCs get captured / put in prison and now they have to escape, instead of just rolling up new characters.
AoE and force multipliers
Area of Effect damage dealing abilities and other force multipliers can further throw things out of whack. Instead of fighting 1-on-1 you have to deal with someone potentially hitting a group of enemies and applying the same damage multiple times. Suddenly the you have someone taking on 5 enemies and killing them off while the rest of the group gangs up on the remaining straggler.
I have ranted about this before, how in Godbound you can have a character that wipes out entire armies with a single attack, but most of the characters there can tap into an AoE smite that becomes quite powerful at higher levels. Not only that, but you can easily bring a small army with you to combat and have them attack everyone, etc.
Setup time and iterating
A perhaps less obvious factor contributing to unbalanced combat is how much time it takes to set up and how easily you can iterate on it. This somewhat ties to asymmetric character complexity. If it takes 10-20 minutes to prepare an encounter and it's over in 5 minutes, something's probably wrong. Same if it goes on for two hours, then it becomes a slog.
Ideally, you'd figure out what worked for a given encounter and try iterating on it as time goes on - "3 guards weren't a challenge, maybe 5 will work better next time". However, if your next encounter uses completely different set of enemies that rely on completely different mechanics ("2 beholders!"), you may not get to iterate on the encounters too much.
If this gets compounded, you may not get too many encounters per session due to how long they take, and you may not iterate on the same encounters too much due to wildly disparate enemies, you may never perfect your encounters.
Fellowship generally has an easy time with this - most enemies have the same amount of health points and their powers mostly differ in flavour. So encounter to encounter a similar amount of enemies will usually be a similar challenge, and encounters themselves take a few minutes to set up at most, so you can easily iterate and tweak things to get pretty much what you'd expect out of it.
Combat vs non-combat PCs
Party composition can also affect combat stability. If you have some PCs that are very combat-focused and some that are very much the opposite, it's hard to have combat encounters that challenge one group and don't outright kill the next. This is further exaggerated if characters can grow to multiply their effectiveness, while others fall behind on the treadmill.
Godbound and Stars Without Number have pretty much been like that for us - having one or two characters that are all about combat, and then inevitably you'd have someone that can't hit enemies and does nominal amount of damage, getting frustrated in the process.
On the other hand, Fellowship once again shines by making playbooks that always have some offensive capabilities, as well as having a system that is versatile enough for everyone to be able to contribute ("I may not be able to kill them, but I can run around screaming to distract them while someone else takes them down!").
To make a somewhat stable combat system for an RPG, you generally want to:
- Operate on small numbers for health and damage that are easy to comprehend
- Keep damage somewhat predictable
- Minimise the effect of action economy on balance
- Manage the lethality of combat
- Avoid AoE attacks
- Make encounters quick to set up and somewhat consistent (be it point-wise, challenge-wise or something else)
- Give options for everyone to contribute or do something meaningful, even if they didn't build the character for combat
Now, not all of those are always necessary and sometimes you will want to create some conscious exceptions, but they are good to keep in mind when designing a fun combat system.
Wargames might be all about besting your opponent and winning at all cost, but that may not always make for a good tabletop combat. You want to challenge your players, not outright kill their characters.