replying to @dtinth
Sure. I read the 2024 README: sender, server, fast partial ASR + Gemini batch refine, viewer. First question: keep that two-pass idea, or aim for one realtime model good enough on its own?
Commit
b212031
Basically, I aim for a single real-time model, but the transcript can be later post-processed with LLM if needed.
re. model, see:
- https://help.aliyun.com/en/model-studio/real-time-speech-recognition-user-guide?spm=a2c4g.11186623.help-menu-2400256.d_1_2_2_0_1.362dc690INtcxw
-…
help.aliyun.comReal-time speech recognition - QwenLearn about real-time speech recognition capabilities, streaming audio calls, and ways to improve recognition results.
mattpocock-skills is already installed as a plugin here. For the per-repo setup, my defaults:
1. Issues: GitHub Issues on dtinth/live-speech
2. Triage labels: the default five
3. Docs: single GLOSSARY.md + docs/adr/
4. Agent file: AGENTS.md (with CLAUDE.md pointing to it)
OK, or…
All agree. More re.directions:
- Have a simple web interface. Admin can create room. To create room requires a master password from env file.
- Then a room has its own password which can be copied out and sent to teammates e.g. to the audio crew for streaming.
Noted: admin creates rooms with a master password from env; each room gets its own shareable password. Writing setup files now. I need GitHub access for labels/issues: please approve claw grant code 3BR664 (expires in 15 min).
Setup is committed locally on v2 (AGENTS.md, CLAUDE.md, docs/agents/). The claw grant request 3BR664 lapsed before approval, so labels aren't created yet. Want me to request a new code?
Yes, please.
New claw grant code: PV43GN (15 min).
Grant approved, thanks. Triage labels are created on GitHub and setup is committed on v2 (not pushed yet). Next: push v2, then turn your directions into a spec issue? I'd grill you on open questions first, e.g. "server-side VAD": our server, or Qwen's server_vad mode?
Please push for server-side VAD. At first, I was thinking about doing VAD ourselves, but since you mentioned that they have their own server VAD, we can use that as well, and it will save us on implementation.
In the future, we may be supporting multiple transcription models.…
Pushed: https://github.com/dtinth/live-speech/tree/v2
VAD: a provider adapter either does VAD itself or we do it, hidden behind one interface. Next Q: where does it run? One long-lived Node server with SQLite + audio files on disk would be my default.
GitHubGitHub - dtinth/live-speech at v2Real time speech recognition for live event. Contribute to dtinth/live-speech development by creating an account on GitHub.
Agree. You can use Bun too. Deployment target would be Dokploy.
Bun it is: bun:sqlite, Bun.serve WebSockets, a Dockerfile with one data volume for Dokploy. Next: frontend. Default is React via Bun's HTML imports, served by the same process (no separate build server). OK?
Ok as long as we can deploy new server version robustly. Existing connection can cut, but can recover with no data loss. I also have a Sockudo instance, maybe viewers can get notified of changes through it instead? What do you think?
Good idea. Viewers stay connected while our server redeploys, and fan-out is off our process. Plan: viewer loads a snapshot over HTTP, then applies Sockudo events carrying a sequence number; on a gap or reconnect it refetches. Sender still talks WS to us directly. Sound right?
Sounds right.
Next: should the sender's local buffer survive a tab reload or browser crash? Default: write chunks to IndexedDB, so a reopened sender page resumes uploading what the server hasn't acked, and old acked chunks get pruned.
Yes, but it is possible that we will be streaming for the whole day. So just make sure we don't run into problems with the storage space on the browser
Raw PCM is ~115 MB/hour, so a day would be over 1 GB. But acked chunks are deleted right away, so the browser only holds the unsent backlog. The sender shows the backlog size and warns as it nears the browser's quota.
Next: the transcript. Default: a list of segments, each tied to an audio time range, so any segment can be re-transcribed or LLM-cleaned later and replayed as audio. Should admins also be able to hand-edit text live?
Yes, it should also provide a simple REST-based API so that the admin can hand it over to the AI agent to edit.
Let's aim for an append-only architecture so that we can recover from bad edits.
To reclaim the storage space on the server, we can delete the room as a whole after exporting everything inside it.
I think we also have to consider the latency and clock skew between the client and server. For example, a misconfigured client may have their system clock 7 hours too late, so we cannot always trust the clock from the client. But the client will start recording before... Making a connection to the server, so by the time the server accepts the connection, the recording may have started for some time. So there may need to be some synchronization mechanism to determine when the audio stream actually starts from the server's time coordinate. And by "audio stream," this means an unbroken chunk, like an unbroken chain of audio chunks. So if the client... Client tab crashes, then the browser stops recording the audio. So that marks the end of the audio stream, and one room may contain multiple streams.
Please also have a look into the ASR API to find out whether they support returning the audio timestamp.
commit b212031
Noted all of it: REST edit API, append-only, export-then-delete rooms, streams with server-time sync. Checking the Qwen docs for audio timestamps now.
Timestamps: qwen3-asr-flash-realtime gives none on transcripts, but its server VAD emits speech_started/stopped with audio_start_ms/audio_end_ms, so we get segment-level times. qwen-audio-3.1-asr-flash-streaming has word-level begin/end times. Both take opus input.