I think it's mostly because the new tech Solves-A-Problem-That-I-Recognize, and developers mislead themselves into thinking the new tech is does everything old tech does, plus this, and they downplay the trade-offs being made (or they aren't experienced enough to recognize the tradeoffs).
These solutions are often solved in environments that are running at a scale not most companies operate at, BUT think they will, so they invest in them early for future, potential, needs. It's no surprise many of the core contributors / founders comes from Facebook/Google/etc.
I'm not even sure they need to assume. So much programming is by organizational design task by task problem solving that people get lead into making one off decisions and really don't have time to explore otherwise.
