replying to @dtinth-bot
Correction: the 2002+ text era is 2,462 notices, not ~3k.
Commit
fb8f913
Can we make like a database of a changelog of each bus route? So given a route, we can see when it was established, when was it revised, etc. with deep links into the PDF.
On it. First pass: parse route numbers out of the titles (no OCR needed) and see how far that gets us.
Thanks. Prototype an Astro website, and send screenshots.
Known wart, see the Bangkok route 1 shot: air-con ปอ. 1 and ordinary 1 share a page, and the 2021 reform's numbers (4-70E, 5-1) are in the same list. Next could be: split those, then use text layers for the body-only notices. Which first?
Wart? Elaborate pls
Wart = known flaw. Bangkok "route 1" page mixes 3 different things: air-con ปอ.1 คลองจั่น-ปากคลองตลาด (1978, 1982), ordinary สาย 1 ถนนตก-ท่าเตียน (2007), and the 2021 reform notice.
Hmm, still don't get the picture. Feel free to make infographics/diagrams in case it will help my understanding haha
Diagram. A: my "route 1" page merges 3 unrelated things just because the titles say "สายที่ 1". B: your reform PDF is one notice for 258 routes (G/B/R/Y codes renumbered to 1-xx…4-xx), but its title names only route 1. Text layers fix both.
Correction: text layers fix B. For A, the 1978/1982 notices are scans, so that one needs OCR. Next I'll check which years have text layers.
Text layer check, 2 sampled PDFs per year: every PDF from 2002 on has one, every one before is a scan. So 2,355 notices (2002+) are readable now; the other 4,979 (1955–2001) need OCR. Want me to start parsing route bodies for 2002+?
That would be a good next step, but how are you representing the result right now?
Before parsing bodies, I'd change it to: one row per route mentioned in a notice (action, number, name, PDF page, quoted text). Then which mentions are "the same route" becomes a separate layer we can fix by hand. Sound right?
Yes, sounds good. When you extract text from PDF, please put text in a file so we don't have to redo the text extraction every time we source text from PDF.
Caveat: citations are notice-to-notice. Omnibus notices ("หมวด 2 จำนวน 33 เส้นทาง") make naive chains fan out, so per-route chains need the route number to match. Next: build the "same route" layer from that, then redo the site. OK?
Yeah. Make me a diagram of the data architecture, like how the site queries this info and how data end up into the site, what processing took place.
Data architecture. All offline scripts writing files into data/; the site is static Astro and reads only events.json at build time (titles, grouped by number). The PDF/text/mentions/citations steps are done but not wired in yet; step 8 connects them.
Hmm, can't we normalize them into a single list of events? Some PDF we can create event from title alone, others we need to extract from text. ykwim?
Yes, makes sense: one route_events list, one row per (notice, route), source = text when we have it, title otherwise, plus page/quote and the notice it amends. The site then reads only that. Doing it.
Done. One list now: data/route_events.json, 7,525 rows, one per (notice, route). 4,272 from PDF text (badge "เนื้อหา", with quote + link to PDF page), the rest from titles. 2,119 know which notice they amend. The site reads only this file.
Now I wonder how many notices were we able to extract the events from. Maybe you can visualize it as a funnel where:
1. The event is evident in the title. That's the first item in the funnel.
2. The ones that have the text layer in the body. That's the second one that we are able to take out of the funnel as a shortcut.
3. The rest is the ones that do not specify the route number in the title, and we don't have the text layer or the OCR result, and we have to transcribe them by ourselves.
commit fb8f913
Funnel: 7,334 notices. Title gives the route for 4,201 (57%), text layer for 410 more (6%). 2,723 left: 2,440 are 1955–2001 scans (the real transcription job), 255 have text my parser missed (fixable), 28 have no PDF link.