I think the reason this persists is because everyone is doing it. Thus your open-plan, inefficient office is only competing with other equally open-plan, inefficient offices. We are all in a less-productive equilibrium together.
This of course gives those willing to make offices for everyone (like, say, Fog Creek) a competitive advantage. But your average corporate manager doesn't care about that. They still get their office and get paid.
The opportunity costs from an open plan office are invisible and difficult if at all possible to quantify, but the savings from cramming twice as many people into the same space are very real and tangible.
This of course gives those willing to make offices for everyone (like, say, Fog Creek) a competitive advantage. But your average corporate manager doesn't care about that. They still get their office and get paid.