I'm using Arch, but this applies to any distro if you don't use a DM and instead use xinit.

I much much prefer the simplicity of only using what I at least partially understand - the more magic (ie a DM) the more shit to go wrong that you don't understand.