I also trained pkda and gdn2, which are about as fast as attention, but both were worse than attention. Since I made it save the weights, once FineWeb training finishes I'll remove the components and evaluate them.
Translated from Japanese
commit fef0eab