There are some already (one of the openllamas I think, and that discord group made one with the linear attention recently although trained on fewer params I think)

Maybe not easy to switch to a non-meta-llama ?