The 3rd edition of Dungeons & Dragons is notoriously bad at dealing with parties above 10th level or so. But this isn’t really anything new. Back with AD&D and even before that with the first D&D game, lots of people are in agreement that the game really works best in the range from 3rd to 10th level. After that things start getting increasingly wonkey.
This came as a bit of a surprise to me. Even though they are all D&D, there’s a huge number of major differences between the various editions. The most common reason I’ve seen discussed for 3rd edition being bad at high level play is that wizards got more spells per day and lost most of the limitations and weaknesses they had in AD&D. But that alone can’t be reason if the problem goes back all the way to the mid 70s.
Higher level spells are certainly problematic a lot of times. It’s not so much that each spell taken for itself is a major dealbreaker (though there are a few contestants for that position) but that the wizard class in particular and the cleric class to a lesser extend, are too broad in what kind of things their magic can do. D&D magic is not good at one thing. It’s good at all things. A wizard class that would be really strong in one situation would not be too disruptive to the game. But D&D always aimed to be somewhat generic and so any kind of magic effect you could think of is available to wizards and clerics. Simply so that you can make the kind of spellcaster that you want to make, regardless of what work of fiction inspired you. Later 3rd edition classes that have a much more tighter focus on specific fields of magic are widely regarded as much better classes when it comes to being a team player. Magic being too broad is certainly a contributing factor, but I believe that’s still not actually the main reason for the wonkeyness of high level D&D.
I believe the real reason for the issue lies in the most basic action resolution shared by all editions of D&D since day one: The 1d20+modifiers roll.
As characters advance in levels, different classes advance different stats at different rates. In 3rd edition the bonus to attack for a fighter is double the bonus for a wizard of the same level. At low levels this is a small difference. +2 against +1. But as you go to higher levels and get a lot of additional modifiers from other sources, you can end up with +40 against +15. Even though it seems like the modifiers have scaled roughly evenly, the 1d20 roll to which they are added does not. The d20 never scales up. What you get eventually is situations where one character couldn’t even fail with the roll of a 1 and a different character couldn’t succeed with a roll of a 20. As modifiers increase and dice stay the same, uncertainty decreases and there is less and less “chance” to anything. It develops more and more into a simple yes/no.
You also have to consider that while you can increase the power of enemies, a lot of obstacles in the game remain static or have a maximum difficulty. At some point, and often pretty early on, you can’t make a wall more steep or a storm more deadly. Eventually you reach the point where all noncombat obstacles become trivial. This point might actually come much later, but it seems that regardless of edition the effect already becomes noticable and makes the game less satisfying around 10th level.
This problem is an intrinsic one of the basic 1d20 action resolution. No matter how much you tweak classes, monsters, and spells, this is something that can not be easily fixed without a complete replacement of the whole system. This is why high level D&D always has been wonkey and always will be wonkey. The most practical solution to that which I see is the one people have been using all the time: Stop continue playing with characters that have reached high level. Instead I propose to treat the game, regardless of edition you use (though 4th and 5th might be an exception here) as only covering the level range of 1-10. After that, you have reached maximum level.
An interesting option for OD&D, AD&D, and B/X would be to basically run it as “Epic 9”. In the 3rd edition variant “Epic 6”, characters only advance to 6th level and after that get one more feat for every additional 5,000 XP. Since characters in older editions already stop getting more Hit Dice and rolling hit points after 9th level, E9 seems to be a good cutoff point. The game does not have to stop there. You can still get the fixed increases of hit points from leveling up (though not chance to hit or saving throws) and find new magic items and discover new spells (of 1st to 5th level only).