FCA Compliance Scanner

The challenge

The app demoed well and was quietly broken. Its build shipped with TypeScript and ESLint errors silenced, live database service keys committed to git, auth that let anyone scan, rate limiting that failed open, and an ingest script that had been producing an empty knowledge base for months while reporting success every time. On top of that it cited the wrong rulebook at whole sectors — COBS rules quoted at mortgage adverts, where they legally don't apply.

What we built· built by our founder, Haseeb Attaullah

• Audited the inherited codebase function by function and published a defect register, then fixed the silent failures first — because every serious defect in this system had been one that reported success
• Two-engine scanner by design: Claude judges non-compliant claims ("guaranteed 15% return"), while plain deterministic code checks for statutory text that is missing — asking the model to judge firm details such as an FRN flagged them as missing on essentially every scan
• RAG grounding over eight FCA sourcebooks from the JSON the Handbook's own web app loads (the site is a client-side app — scraping it silently yields an empty corpus), routed by sector so a mortgage advert is judged under MCOB rather than COBS, with citations to rulebooks that don't govern the product stripped out
• Required-warning checks for capital at risk, past performance and charges: a flag fires only when the warning is required and missing, so the FCA's own prescribed wording is never flagged
• Severity decided in code from a versioned rule pack instead of by the model — every scan records the rule-pack version, the Handbook snapshot date and a hash of the system prompt, so any past result can be explained
• Fail-safe scanning: if retrieval, the corpus or the model fails, the scan says so — a partial scan never shows as a pass, and the user's quota is refunded
• A scanner health endpoint that asserts on what the corpus actually contains — chunk counts, embedding dimensions, live SQL function arity — plus prompt caching with a byte-frozen system prompt, a 34-case eval harness, and coverage thresholds on the compliance engine

How it fits together

Our founder's part

Haseeb Attaullah: Took over a Bolt.new prototype and made it shippable — audited it, rebuilt the scanner as two engines, fixed the corpus pipeline, and added the eval harness, scan provenance and fail-safe scanning.