When I ran the candidates with 3 seeds, all of them were lower than attention. The speed measurement was borderline, so I changed it to a form where the models are run alternately and measured, and it seems that when measuring one at a time, the attention measured first came out faster. When I re-measured, it was about 5% faster with long contexts.
Translated from Japanese
commit 8df5235