When I measured the suffix read time for each stage, the byte-side reads accounted for most of the step. Comparing with tiny at the same wall-clock time, the one without reads was the best, and it's losing on speed as usual.
Translated from Japanese
摘果みかん
@mikan-919
Hello
- github.com/mikan-919
- Joined October 2026
Commit activity
30 commits in the last 30 days
longest streak 3 dayscurrent 2
504 / 1,000 XP496 XP to wheel
Today
Changed shiro's color scheme from earthy warm tones to cool-leaning grays. It's easy and nice that swapping the tokens changes every screen.
Translated from Japanese
I split everything that used to be laid out on a single page into three tabs. Every time you add a feature by tacking it on below, it naturally gets cluttered. Just making the empty state a single line without a border makes it feel a lot calmer.
Translated from Japanese
When I removed the attention components one by one, it got worse no matter which one I removed. I thought one of the two consecutive projections was unnecessary, but it's actually needed. Removing it or replacing it doesn't work, so it seems we're going to have to change the…
Translated from Japanese
I moved the backend processing that used to be run from the CLI over to buttons in the WebUI. Writing to the same DB from a separate process while the server is running is obviously dangerous, so it's good that it's now consolidated in one place. Streaming progress over WebSocket…
Translated from Japanese
Yesterday
I stopped looking for replacements and set up an experiment that removes components from attention one at a time. I was able to load the training code from the previous theme as-is, so I only added about 40 lines.
Translated from Japanese
Adding the moving average dropped it by only 0.009 bpb. It does have an effect, but I don't think it's big enough to say it replaced attention.
Translated from Japanese
I reviewed all the WebUI animations. Grouping button press effects into the scale property makes things easier because it doesn't conflict with transform. For images, I made them expand from the thumbnail and then return.
Translated from Japanese
Split attention's work into local syntax, exact recall, and distant gist, then add the third via moving average. I could write it with a single cumulative sum, but with a short half-life even float64 overflows, so I added an assert. I think the odds of it working are no better…
Translated from Japanese
I counted whether widening the index increases the number of long matches. The number of 24-byte matches barely increases even when the range is expanded 64 times. I think it's good to count first what can be understood in a few minutes without training.
Translated from Japanese
The idea of pulling bytes and reading in chunks: the amount read increased, but bpb didn't change. The scarcity of long matches wasn't due to boundaries; the data simply didn't have them to begin with. I also found that the log statistics were being inflated by the padding frame,…
Translated from Japanese
Since chunk-level matching was hardly used, I tried pulling by bytes and reading in chunks. At 30 steps, the proportion not read dropped from 0.80 to 0.47, but whether it stays that way as training progresses remains to be seen.
Translated from Japanese
幅を1.5倍にして指数を学習させてみたけど、固定との差はseedで符号が変わる程度だった。NixOSのWSLでTritonを動かすのにlibcudaのパスを環境変数で渡す必要があって、そこでちょこっと詰まった
I thought the code was gone so I rewrote it, but everything was in the Downloads on my main machine. I deleted the rewrite and imported the original. I want to stop leaving research code sitting in Downloads.
Translated from Japanese
I added a mechanism where shiro proposes types from typegap and evaluates them through replay. Types are held as data rather than code, so once adopted, no restart is needed. Only adoption is still decided by myself.
Translated from Japanese