Even in FineWeb, candidates scored lower than attention, and it's a bit surprising that the difference is larger for data with fewer repetitions. I also prepared ByteLevel; I'll convert WikiText back to a byte sequence and compare bits per byte on the same body text.
Translated from Japanese
commit 881a520