You might want to give plain TeX a try--it's surprisingly lightweight (once you learn how to change font sizes and/or decide not to care). A lot of the bloat comes from LaTeX and using 'eplain' will give you most of the crossreferencing and bibliography commands (if you need it). Running latexmk -pvc in a separate terminal will give you rebuilt dvis on the fly (or pdfs). AucTeX can generate inline previews for plain tex just as well as LaTeX.

Of course, org mode is the standard recommendation for "emacs based markdown replacement with inline math previews"