Balance mostly deals with different factions in strategy games like Starcraft or Age of Empires, where each faction has its own strengths and weaknesses, but they have to remain balanced on the stereotypical scale. If one faction is too powerful, then that scale tips and the game is unbalanced. Balance issues are not always clear; it takes testing in real game conditions to make sure, but it is worth the work to make sure a game or mod is the best it can be. Blizzard Entertainment tweaked Starcraft 1 for years to make sure it was balanced, and it appears they're doing the same with Starcraft 2.
In first-person shooters, things aren't any easier to see if they're balanced. How do you decide a weapon does too much damage? Or if a monster has too much health? What are you balancing it against? In a multiplayer game like Quake 3, you balance it with the other weapons; because each item is readily available, they should be equally useful in general. In a single-player game like Doom, you balance it according to a progressing staircase. Some monsters are more powerful than others, according to this staircase. Let's look at the basic Former Human, Imp, Hell Knight, Baron of Hell and Cyberdemon. At the bottom of this staircase is the Former Human; it's very weak, almost incapable of killing the player, and usually placed in groups. Imps are a bit more powerful but dodgeable, and still common because it's not very strong. The Hell Knight, however, is much more damaging. Just as dodgeable, but it also has more health to deal with. These are rarer than imps. Barons are the same as Hell Knights, but double the health, so it stands to reason these would be rarer. Cyberdemons have a whopping four times the health of a Baron and is capable of one-shotting the player. It's obviously a boss.
There are two ways to see if something truly fits into this staircase; the first is seeing if it fits the difficulty the monster was intended for. Was the imp meant to be a common weakling? Was the Cyberdemon meant to be a boss? If the answer to these questions is yes, then very good. The second thing to make sure of is to see if this balance is actually FUN. Play Nuts.wad without any cheats, gameplay mods or anything like that. Is that fun?
There are several factors that go into the difficulty of a monster. Health and damage being the most obvious. Then there are numbers of attacks, type of attacks, speed of attacks, movement speed, gimmicks, rarity and situation. I will say first that you DO NOT EVER balance one extreme with another. If a monster has an insta-kill hitscan attack, and one health, that's not fun! It's either a pushover, or too damn frustrating! Glass cannons are stupid to fight. Yes, I'm guilty of making one (The Repeater Zombie), but recently I have rebalanced it quite a bit. If an attack does 500 damage, but moves so slowly that an ant will outrun it, that's not fun, that's not balanced, that's just stupid.
Health and damage are the two most common ways of balancing a monster. The original Doom used these because primarily because Id didn't have a lot of other gimmicks to add. Think of the Imp and Baron of Hell. Functionally, they're exactly the same. The only difference is health and damage. Another common one is the type of attack; projectile versus hitscan. Projectiles are dogable, so they should do more damage, whereas you can't dodge hitscan, you have to rely on luck to avoid damage. Random spreads are sometimes used for balancing hitscans, but remember that with projectiles launched with a random angle, it's possible to dodge into the attack.
Speed is another balancing factor, but not just the movement speed of the monster. There's also attack speed (how quick between attacks, how short the delay before it attacks) and projectile speed. Revenants are quick, and that makes them tricky to hit sometimes, which balances their low health. Arachnotrons also have a short delay between attacks, but there's a delay before they fire that you can take advantage of, either to duck behind cover or to interrupt them. Don't make something too fast that it's silly, though; speed is one of those things that can look downright hilarious if something is too fast or too slow.
Gimmicks are the hardest things to get right, because it's all unique. By gimmick, I mean some unique feature in the monster that doesn't fit into any other category. It could be the Archvile's resurrection ability, or the Revenant's homing rockets. The only way these can really be balanced is with proper testing.
Keep in mind graphics go into balance as well. If a monster doesn't look powerful, it's not going to be well-received when it takes away 75 health with one attack. Often, it's best to decide where it goes in the progression of difficulty simply by how it looks and plan around that. The Banshee in the Beastiary right now (as of 12/07/11, it may get updated) doesn't look or sound like the horrible powerhouse it is. It doesn't make sense for the weakest monster in a mod to have more special effects than the final boss, as well. Also, taking existing, weak monsters and suddenly making them more powerful isn't a great idea, either, since you're messing with the player's expectations, which is really frustrating to the player. An imp with 800 health that throws rockets is rather silly.
The total number of attacks a monster has, while it doesn't necessarily make a monster unbalance, goes into the overall quality of the monster as well. Weaker monsters don't need a lot of attacks, especially since they'll be dead before the player sees all of them. Bosses benefit from having several unique attacks, but it doesn't make a boss. Make sure the attacks the monster has are used to best effect, and that they aren't thrown in just because.
For the purposes of the R667 Beastiary, you don't have to worry about rarity and situation, but you have to keep it in mind for testing your monster. The best way I've thought of to test situation is to replace a monster in an equivalent place in that staircase of difficulty and test out some real maps. When you test a monster by summoning it, you're setting the conditions. You're deciding what weapons you have when you fight it, how much health and armor you have, how large the area is, how much cover you have. In general, these conditions really suck for testing monsters because you aren't seeing how they work in real gameplay. I admit, I like to hit Give All and test monsters out by going to the middle of Doom 2 Map13 (Or Heretic E3M3, depending on what game it's intended for) and summoning the monster in question, but I've been doing this a while and I can keep in mind how different the situation could be. Also, if you gear a monster for a specific situation and are concerned a modder might place it in a map in a situation where it's too tough or too weak, say what it's intended for in the Info lump. That way, it's the modder's fault and not yours!
Also, keep in the mind the skill of people playing. People don't play in the same exact way you do, and if you made the monster, you already have an advantage over those other players because you know what to expect, just like if you made a map you know where all the monsters and secrets are. Will the average player find this fun? It's okay to gear something towards more experienced players, but just bear in mind the community in general. This is good advice for anything you do in modding, whether it's making a monster, a weapon, or a map pack.
Hopefully, this tutorial helps people. At the very least, it'll give me something to point monster makers to when they submit a really fast monster with 40 health that does 140 damage with fast, homing attacks. This is geared towards monsters, but some of the principles (balancing one extreme with another, for example) carry over into weapons.