Mercy Networks started from a practical failure mode: social-service information is often technically public but operationally unusable. A shelter, meal program, or medical service may publish its eligibility requirements in a PDF flyer or a dense program page, but the person who needs the answer has to translate that document while time is short.
The application turns those documents into searchable, structured eligibility records. A caseworker can ask a plain-language question such as “bed tonight near Plano” and get back services with the requirements that actually matter.
The engineering problem
Calling an LLM is not the hard part. The hard part is deciding whether the extracted answer is correct enough to put in front of someone in crisis.
The ingestion pipeline accepts PDFs and program URLs. It stores three representations of every service together: the original source text, the eligibility-relevant passage, and the structured JSON payload. That redundancy is deliberate. If a parse goes wrong, the source is still there to re-extract from instead of being lost behind a bad result.
Measuring extraction instead of trusting it
The project now has a deterministic eval harness: ten hand-labeled documents, a scorer with threshold gating, and baseline comparison so prompt or model changes can be measured rather than eyeballed.
Two design decisions carry most of the weight.
The scorer is deterministic and has its own tests. No model grades another model’s work, because an LLM judge introduces exactly the variance the harness exists to measure. If the score moves, you would never know whether the extractor changed or the judge did. Scoring runs offline with no API key, so a green eval means the extractor improved rather than the scoring drifting.
Grounding is scored separately from field accuracy. The model is asked to quote the passage stating eligibility, so that quote must appear verbatim in the source. A run can get every structured field right and still fail this check — which is precisely the case worth catching, because plausible structure attached to a fabricated citation is worse than an obvious error.
Empty sets score as correct, because most real documents genuinely state no requirements and penalizing that would drown the signal. Inventing entries the document never mentioned drives precision to zero. That asymmetry is intentional: over-extraction is the failure mode that actually hurts users.
What the harness found on its first run
Three real defects, none of which were visible before there was something measuring them.
The schema made hallucination mandatory. rawEligibilityText was declared as a non-empty string. For a document that states no eligibility rules — an annual report, a donor letter — there was no valid answer, and the model was structurally forced to invent a quote to satisfy the schema. The field now permits an empty answer, and the prompt states plainly that empty is preferred over invented.
A stale on-device AI provider. The Chrome built-in AI provider was cached in a module-level singleton that closed over whichever window.ai existed on the first call. A provider created before the user granted permission kept serving a dead session afterwards. It is now keyed on the AI object itself.
The test suite had never run. Three independent reasons — a missing dependency, an unconfigured path alias, and disabled globals — meant it failed before reaching a single assertion. The suite now passes 26 tests, and the build is clean.
Why this is a senior project
- It separates source text, extracted passage, and normalized structured data, so extraction is recoverable.
- It treats evals as part of the system rather than a demo afterthought.
- It keeps scoring deterministic and independently tested, so improvement is provable.
- It surfaces its own limitations instead of hiding them.
The next milestone is not “add more AI.” It is making the AI accountable: show which source span produced each structured field, route low-confidence records to human review, and expand the fixture set until the score means something across a wider range of real service documents.