I'm looking for a mixer that's faster and more accurate than attention. FoX was winning quite decisively against attention at 3000 steps, but when I ran it up to 10000 steps, it lost on both seeds. The hybrid of SWA and KDA also doesn't reach attention at context 2048. I've seen once again that rankings from short training runs aren't reliable.
Translated from Japanese
commit 18f7ea1