The problem may be too much investment in drivers. Modern graphics card drivers are little operating systems unto themselves; they even have built-in compilers. They are incredibly complex, yet written in unsafe languages for maximum performance, with tons of ugly hacks for compatibility.
What we really need is a simplification of graphics APIs so that graphics drivers don't need to be so complex. That may be impossible without convergence in hardware though.
That graphics drivers are a mess internally says more about the insufficient resourcing to do it properly and cleanly and refactor than it does about the really rather trivial and well-understood APIs that the driver implements.
You think OpenGL and DirectX are "trivial"? I'm sorry, but an API that involves passing text strings of complete programs for the driver to compile, link, and asynchronously run on a vector coprocessor is not trivial to implement, not to mention the complex NUMA memory management going on.
What we really need is a simplification of graphics APIs so that graphics drivers don't need to be so complex. That may be impossible without convergence in hardware though.