replying to @dtinth
Next: should the sender's local buffer survive a tab reload or browser crash? Default: write chunks to IndexedDB, so a reopened sender page resumes uploading what the server hasn't acked, and old acked chunks get pruned.
Commit
e79cc9b
Yes, but it is possible that we will be streaming for the whole day. So just make sure we don't run into problems with the storage space on the browser
Raw PCM is ~115 MB/hour, so a day would be over 1 GB. But acked chunks are deleted right away, so the browser only holds the unsent backlog. The sender shows the backlog size and warns as it nears the browser's quota.
Next: the transcript. Default: a list of segments, each tied to an audio time range, so any segment can be re-transcribed or LLM-cleaned later and replayed as audio. Should admins also be able to hand-edit text live?
Yes, it should also provide a simple REST-based API so that the admin can hand it over to the AI agent to edit.
Let's aim for an append-only architecture so that we can recover from bad edits.
To reclaim the storage space on the server, we can delete the room as a whole after…
Timestamps: qwen3-asr-flash-realtime gives none on transcripts, but its server VAD emits speech_started/stopped with audio_start_ms/audio_end_ms, so we get segment-level times. qwen-audio-3.1-asr-flash-streaming has word-level begin/end times. Both take opus input.
Clock sync idea: ignore the client wall clock. A stream gets a random id; chunk n is at stream offset n*100ms. While connected, client pings with its elapsed time since stream start (monotonic); server estimates stream start in server time from the lowest-RTT ping. Good?
Last one before I write the spec: the public viewer. Default is a live scrolling transcript (partial text shown dimmed), auto-scroll with pause on scroll-up, a fullscreen caption mode for projectors, and a text export. Anything to add or drop?
The public viewer should also contain a section on how to connect the AI agent so that it can read the transcript efficiently.
Regarding Opus, I wonder if the client should be the one encoding Opus packets, or should it be that the client is sending raw PCM stream and the server…
Agent section in the viewer: noted.
Opus: I'd encode on the client with WebCodecs AudioEncoder. That shrinks both the IndexedDB backlog and upload on venue wifi by ~10x, which is where robustness matters most. Raw PCM stays as a fallback if the browser lacks it.
Server stores…
Correction: opusscript is itself an Emscripten build, not a native addon, so it already is the safe choice. No need to swap it.
Ok good. By the way, in contrast to Matt Pocock's skills, I will not be using multiple separate sessions to implement stuff. But I will use this long-running session for most of the work, but you should feel free to delegate to sub-agents to work autonomously.
Got it, saved that to memory. Writing the spec issue next. Test seams I'd propose:
1. Server as a whole: start it in-process, drive its HTTP + WS API, with a fake ASR provider and a fake Sockudo publisher.
2. Sender's buffer/uploader: a fake socket that drops and reconnects.
OK?
Ok. By the way, is it clear enough to implement? Or do you need any further grilling?
The artifact that I kind of expect is the code on the branch with a Dockerfile, so I can make Dokploy build it.
The Docker file should also contain the EXPOSE directive, which tells me which…
Clear enough to build. Spec: https://github.com/dtinth/live-speech/issues/2 (one open risk: catch-up pacing after an outage; I'll measure it). Starting now: Dockerfile with EXPOSE + VOLUME, pushed to v2, updates with screenshots as I go.
To test for real I'll need a DashScope…
GitHubv2: live captioning rewrite spec · Issue #2 · dtinth/live-speechProblem Statement Live captions at local tech talks in Thailand have to cope with Thai speech full of English technical terms. The 2024 version of live-speech handled this by chaining a fast stream...
Okay, so in the meantime you can run your own Sokudo because I already run a rootless Docker daemon on your box. Please also create an env file with placeholder values so I can fill it in later.
Please also send me architecture diagrams and stuff like that, so I understand how…
Progress: server core is pushed to v2 (https://github.com/dtinth/live-speech/commit/a3c43f5) with 15 passing tests, including restart-with-no-loss. Dockerfile has EXPOSE 3000 and VOLUME /data. .env with placeholders is in the repo dir, and a local Sockudo runs on :6001.…
GitHubAdd server core: rooms, resumable ingest, transcription, event log · dtinth/live-speech@a3c43f5Bun server with SQLite and on-disk audio. Senders stream numbered 100 ms chunks over a WebSocket and resume from the server's last ack; the server syncs each stream to its own clock with pings....
I looked at the diagram and wondered about the audio stream. What do you think between storing the audio inside the SQLite files themselves compared to storing separately on the disk? If it is a WAV file, I would lean more towards storing on disk, but now that we pre-compress…
By the way, after you are done, please also do an adversarial review on the code base.
Will do: an adversarial review once the pages are in.
commit e79cc9b
Progress: sender, viewer and admin pages are pushed (https://github.com/dtinth/live-speech/commit/e29777e), 36 tests passing.
Verified in headless Chromium with a fake mic: recorded Opus, killed the server for 3 s mid-stream, and after restart the backlog uploaded with no gap…
GitHubAdd admin, viewer and sender pages · dtinth/live-speech@e29777eSender records Opus (PCM fallback) in 100 ms chunks, writes each to IndexedDB before sending, prunes on ack, and resumes after drops, reloads and crashes. Viewer applies Sockudo events in seq order...