Cite the Record
Menu

Architecture

Under the hood

Cite the Record looks like a citation surface, but the depth that makes the citations defensible sits in the layers behind it. This page walks the stack — what runs where, what's automated by LLMs at build time, what is deliberately NOT automated at request time, and why each piece is shaped the way it is. Aimed at civic-tech and govtech peers who want to see whether the architecture is serious enough to be useful.

How AI is used

AI can phrase a search query. Ranking and record retrieval remain deterministic, and results stay tied to official sources. Other LLM-assisted descriptions and labels are prepared offline and reviewed before publication.

For AI use today, start with live search and supported record pages. A read-only direct connection for AI tools is built but coming later; it is not yet deployed.

The unified search box on /members takes a postcode, place name, or member name and routes to the right surface. Three independent backends, one resolver, one component.

  • Postcode → constituency. Direct call to postcodes.io, the public ONSPD-derived service. Production may swap for a Cloudflare Worker proxy or a self-hosted idealpostcodes/postcodes.io.db Docker image (pinned digest) — the lib surface stays the same.
  • Place name → constituency. uFuzzy (~10KB MIT, designed for typeahead) over a gazetteer of UK 2024 Westminster Parliamentary Constituency names + OS Open Names cities/towns/villages, each point spatially joined to the WPC boundary polygon. The gazetteer was lifted from an earlier project by the same author, where the boundary-mapping pipeline is exercised against PostGIS + postcodes.io evidence and a published test harness covering known overlap failure modes.
  • Member name → person profile. Same uFuzzy engine over a separate person index built from civic.entity. Fat-finger typos resolve cleanly: "frmoe" → Frome, "smith" → Smyth.

The resolver merges results people-first, places-fill. Type badges (PERSON / CONSTITUENCY / TOWN / CITY) do the disambiguation work that a search-mode toggle would otherwise force on the user.

The interesting bit for civic-tech peers is the geoparser, which is currently in the planned column for cite-the-record but already proven in an earlier project: token-aware longest-non-overlap span recognition over a 44k-entry global gazetteer, suppressing embedded false-positives like "Vale" inside "Abington Vale". That project’s test harness exercises the pipeline against PostGIS + postcodes.io evidence and a 2,349-case known-overlap suppression suite that runs in CI. The acceptance bar is independent reproduction of the overlap suite, not a single accuracy figure.

Document intake and OCR

Parliament publishes a long tail of records as PDFs, scanned committee evidence, and assorted document attachments. Cite the Record's document toolchain takes each source artefact and produces:

  • An extracted_text rendering (text-extractable PDFs go to Apache Tika; scanned PDFs route via PaddleOCR or Document AI depending on quality)
  • A glyph-suppression layer so OCR artefacts (a smudge interpreted as a redaction bar) don't leak into the surfaced text
  • A retention envelope that holds the original PDF bytes, the converted OCR text artefact, the document-level OCR confidence score, and the toolchain version that produced the result

The toolchain runs in containers: a Tika service, a PaddleOCR worker, a Document AI client, and a shell orchestrator that routes each input to the right path based on document characteristics. Each container is pinned by digest; outputs are deterministic per input + toolchain-version, which is what makes the provenance hashes meaningful across re-runs.

For a journalist citing a passage from a 200-page committee evidence pack, the value chain is: PDF page → CTR-converted OCR text → "Cite the Record's [ CITE ] button". The original source PDF and the converted text are companion artefacts — we deliver both side-by-side so a reader can verify quoted passages directly against the source bytes. CTR does not claim character-level positional fidelity between the converted text and the source PDF (an earlier design attempted per-passage bounding-box overlays; we dropped that approach because accurate alignment was infeasible at production scale). Attached-file text is not searchable at launch. Launch search covers record titles, excerpts, body text and summaries.

LLM-assisted API pulls and parser maintenance (in progress)

Parliament's APIs are stable for the things they cover. The hard bit is the gaps — which endpoints to hit for a given record family, how to discover new endpoints when Parliament publishes them, how to translate inconsistent enum values across years of API drift into a single normalised schema. The shape this is taking, as the intake pipeline matures, is LLM-assisted analysis at build time that:

  • Reads the OpenAPI / Swagger surfaces, proposes a query plan for a given source family, and emits the actual fetch script as deterministic Python (the LLM writes the plan; the runner is plain code).
  • Detects when an upstream API response shape changes versus the cached schema, proposes a delta, and proposes a parser change for operator review. Schema drift gets reviewed; it doesn't silently corrupt downstream data.
  • Suggests entity-resolution rules when a name appears in two source families with slightly different spellings ("Dame Angela Eagle" in one, "Angela Eagle" in another). The rule is proposed; a human merges or rejects.

The pattern across all of these: the LLM acts as a senior developer who reads documentation, writes plans, and proposes patches — not as an oracle whose answers are trusted at face value. Every output is a PR or a checked-in plan; nothing executes on production data without operator merge. Current status is shown through the site's coverage and roadmap pages.

LLM-assisted database write tooling (planned)

The intake pipeline writes to a clean PostgreSQL schema (civic.public_record, civic.entity, civic.record_topic, et al). Hand-writing SQL migrations and per-record upsert logic for every source family would be brittle; the planned tooling layer uses LLM assistance to:

  • Generate idempotent upsert statements from a parsed-row shape and a target table, with the LLM emitting plain SQL that the human reads before it touches the DB.
  • Propose migration scripts when a parser changes — the LLM compares the old and new shapes and writes the ALTER TABLE or backfill script as a versioned migration file.
  • Audit existing rows against the parser version that wrote them and flag drift candidates for re-parse.

The discipline that will make this safe: writes happen only via reviewed migration files in a version-controlled folder. No tool, no agent, and no LLM has direct credentials to the production DB. The clean-side database is rebuildable from the source-cache archive + the migration history — an LLM mistake is rollback-able. Current status: migrations and upserts are hand-authored today; the LLM-assistance layer above is a Phase-2 ergonomics improvement, not a v1 promise.

Provenance and evidence storage

Every fetch from a Parliament API produces a source_fetch_event row holding the request URL, timestamp, response status, parser version, and a SHA-256 of the response payload. The payload itself is retained as:

  • Inline retention in source_fetch_event.retained_payload_bytes for small JSON responses
  • Object-store retention in a SHA-256-keyed evidence store for larger artefacts (PDFs, HTML pages, video transcripts)
  • OCR / extracted-text retention alongside the source artefact, with the toolchain version that produced it

Every parsed row that ends up in a visible civic table carries source_fetch_event_id and source_evidence_id columns. This is what makes the [ CITE ] button defensible: the SHA-256 in the citation matches the saved copy we keep. If a future re-fetch finds the source has changed, the new fetch becomes a new event with a new hash, and the original record retains its original evidence — provenance never gets overwritten.

The page surface leads with the official source and, where available, retained evidence. Technical provenance stays secondary. See the provenance contract on /about.

Planned structured data and schemas

Per-record machine-readable files are not published at launch. The /schemas pages document planned field shapes and clearly mark canonical files and verified examples as pending.

Launch access is through live search and supported record pages. No corpus-wide file surface ships at launch.

MCP server

A read-only Model Context Protocol (MCP) service is built but coming later; it is not yet deployed. When released, it is intended to expose these read-only tools:

  • search_records(query, kind?, dateRange?, limit?) — search source-linked records
  • get_record(stable_id) — one record by stable ID
  • get_topic(slug) — one topic profile
  • get_person(slug) — one person profile
  • cite(stable_id, format?) — format a citation for one record
  • list_recent_records(record_type?, since?, limit?) — list recent records within a selected boundary
  • discover() — describe the planned tool list and coverage

The planned service will be deployed separately from the site. Transport, authentication and connection details will be stated when deployment is ready. Responses are intended to retain official source links and the same neutrality limits as the site.

Constituency-mention geoparser (planned)

The differentiator from Hansard direct and from TheyWorkForYou: when a record mentions a place ("Frome", "Yateley", "Trawden"), Cite the Record will surface which constituency that place sits in, and link the record to the MP for that constituency. The pipeline is proven in an earlier project and pending port to production cite-the-record:

  • A global gazetteer of every OS Open Names / OSNI place point spatially joined to a 2024 WPC boundary polygon (44k entries; gazetteer construction validated against PostGIS + postcodes.io evidence in that project’s test harness)
  • A token-aware span recognizer that finds all gazetteer matches in arbitrary text with longest-non-overlap selection — so "Vale" inside "Abington Vale" suppresses to the longer span
  • A resolver that ties each retained span to the correct constituency + MP with deterministic features (target-office context, speaker-is-constituency-MP context, ambiguity-register policy)
  • An evaluation harness over 2,349 synthetic overlap cases that must all suppress correctly before promotion

Result: every Hansard contribution, written-question title, and committee-evidence passage carries a mentions array in its shard, naming the constituencies it references. Charities tracking issues in their local areas can search every parliamentary record that mentions their patch. That's a real product gap in the current public-record landscape; the deterministic geoparser is what makes the claim trustworthy enough to feed into reporting.

Local-only debate-clip recorder

/video is a peer surface to the citation experience: a browser tool that records short clips from a Parliament TV stream the user is watching, burns in caption + source overlay, and emits a caption-ready MP4 in under a minute. The trust story:

  • No backend video compute. The tool does not proxy Parliament streams, does not fetch HLS manifests, does not call any Cite the Record API. The user's browser is the only thing that touches the source.
  • Local capture only. Uses navigator.mediaDevices.getDisplayMedia to record from a tab/window the user explicitly picks in the browser's own picker dialog.
  • Native MP4. MediaRecorder writes native MP4 on desktop Chrome/Edge (no WebM fallback that journalists' workflows can't ingest), at Parliament's native 25fps cadence.
  • WebCodecs + Mediabunny for the final trim/rebuild path — the user's exported MP4 is a clean re-mux from the in-memory scratch, not a transcode.
  • Optional File System Access streaming so long clips can stream chunks to a local file as they record, avoiding the in-memory cap.
  • Canvas compositor for the burn-in overlay (speaker name + Hansard timestamps + source URL).

The video tool ships as static assets merged into the main site at build time (single deploy unit). The JS runs in the user's browser; nothing the user records ever leaves their machine unless they choose to share the resulting MP4. For a constituency office producing social content from a debate, this collapses a 20-minute workflow (download → cut → caption → re-export) into one screen.

Hosting and edge

Production runs on Cloudflare. Three deploy units, one domain:

  • Main site (Astro) — site pages and the /video static assets. Cloudflare Pages.
  • Search API — server-rendered Astro route, automatically promoted to a Cloudflare Pages Function.
  • Direct AI-tool service — built but not yet deployed; planned as a separate worker for lifecycle and isolation.

The hosted Postgres for search index + topic assignments is Neon (HTTP driver from Cloudflare, no connection-pool issues). Embedding vectors stay on the operator's workstation — they're large, only needed by the offline topic-detection pipeline, and shouldn't bloat the hosted DB. Topic discovery uses a locally run CPU-capable embedding model selected by offline cost-and-quality testing, HDBSCAN clustering, and a Codex-driven labelling step that proposes each candidate topic for operator approval.

AI agents and review discipline

Implementation is divided across roles: the operator (direction and approval), a Codex agent (intake-pipeline implementer), and a Claude agent (UI design lead + standing reviewer). Every Codex-authored pull request goes through a defined audit loop before merge — an internal Codex re-review on a freshly spawned subagent, then a Claude path-B review invoked from a neutral execution directory with no session context — and the review chain is sealed in an append-only audit ledger.

The discipline exists because LLM agents drift toward narrow compliance ("the contract clause was checked") while broader issues persist ("but the broader system is still broken"). The ledger and reviewer-separation rules give the operator an auditable review record before merge.

Why this matters for civic tech

These patterns — evidence retention, deterministic parsing, a local-only recorder and separate review — can generalise to other civic-data projects. Feedback and questions are welcome.

Feedback, criticism, and "have you considered..." are welcome — feedback@bower.im.