Today's docket at home: testing out Project Maya to see how feasible it is to run GLM 5.3 Flash on a 5080 with 16gb vRAM and 128 GB dRAM.
So far, it's not horrible, just long ttft. About 25-30 tok/s decode, 1200 prefill; compared to 75-90 and 2600 for Qwen 3.8 Flash Next.
Commit
2a215cb
Amazing! I did test Qwen 3.8 Flash Next on my 4090 but not GLM yet! I definitely should try!
Definitely worth it! I think ultimately it's disappointing compared to our GOAT Qwen 3.8 FN but still, in theory it's a teensy bit smarter so it could be good in a pinch. It's also nice when the voice of FN gets on my nerves, that mfr has AI voice for sure.
Main issue right now is even with all this ram it's hard to get enough context for useful work (eg 272k+) without sacrificing prefill to the point that it's unbearable.
commit 2a215cb
Yeah I get it. I am a bit shorter on RAM I "only" have 64GB, but 24GB vRAM, so I might have faster prefill but shorter context... Will see. Anyway I adapted my workflow to short context.