One of the approaches I was trying produced numbers considerably better than attention. The gap was so large that at first I suspected the next token was leaking, so I trained on uniformly random tokens and confirmed it doesn't go below log V. I'm increasing the seeds and checking.
Translated from Japanese
commit 7bf062a