Social service information is usually public and almost never usable. A shelter publishes its intake rules as a PDF flyer. A county program buries eligibility in three paragraphs of a web page. Someone who needs a bed tonight has to parse that while their phone is at 8%.
Mercy Networks turns those documents into structured, searchable records. You ask for “bed tonight near Plano” and get back services with the requirements that actually decide whether you qualify.
The part that is actually hard
Getting a model to return structured JSON is a solved problem. Knowing whether that JSON is right is not, and this is a domain where being confidently wrong sends someone across town to a shelter that turned them away two years ago.
Ingestion takes PDFs and program URLs and keeps three representations of every service: the original source text, the eligibility passage, and the structured payload. That redundancy is deliberate. When extraction improves — or breaks — the source is still there to re-run against, instead of being lost behind whatever the model decided the first time.
Measuring it
There are ten hand-labeled documents and a scorer that runs against them, with threshold gating and comparison to a stored baseline.
The scorer is deterministic and has its own tests. No model grades another model’s output. An LLM judge would introduce exactly the variance the harness exists to detect — when the score moves you would have no way to tell whether the extractor improved or the judge drifted. Scoring runs offline with no API key, so a passing eval means something specific.
Quoting is scored separately from the structured fields. The model is asked to quote the passage that states eligibility, and that quote has to appear in the source. A run can get every field right and still fail here, which is the case worth catching: correct-looking structure attached to a sentence the document never contained.
Empty sets count as correct. Most documents genuinely state no requirements, and penalizing that drowns the signal. Inventing a requirement drives precision to zero. Over-extraction is the failure that hurts someone, so it costs more.
What it caught immediately
The schema required a quote. rawEligibilityText was declared as a non-empty string. Feed it an annual report or a donor letter — a document with no eligibility rules at all — and there was no valid response. The model had to invent something to satisfy the type. The schema was manufacturing hallucinations, and no amount of prompt tuning would have fixed it. The field now accepts an empty string, and the prompt says outright that empty beats invented.
A dead on-device session. The Chrome built-in AI provider was cached in a module-level singleton that closed over whatever window.ai existed at first call. A provider created before the user granted permission kept serving that stale handle afterward. It is keyed on the AI object now.
The test suite had never run. A missing dependency, an unconfigured path alias, and disabled globals — three independent failures, all before the first assertion. It passes 26 tests now.
None of these were visible until something was measuring. That is the argument for building the harness first.
Next
Field-level provenance: show which sentence produced which value. Confidence scoring, so low-confidence extractions land in a review queue instead of the database. And more fixtures, until the number means something across the range of documents these organizations actually publish.