neural-tools is a published TypeScript CLI for generating the boring but failure-prone parts of AI infrastructure: MCP servers, Claude commands, agents, skills, plugins, and the deployment glue around them.
The point is not to hide generated code behind a framework. Generated projects are ordinary readable files you own, review, and change. That is the opinion behind the tool — scaffolding should remove repetitive setup, not build a box you cannot inspect.
Skills are selected by their description, and nobody tests that
A Claude skill is chosen by its description. The model reads the available descriptions and decides which to load, which makes that one paragraph effectively executable routing logic. If it is too vague the skill never fires and is invisible. If it is too broad it hijacks unrelated work.
neural-tools eval skill <path> checks that contract. It validates frontmatter, name and directory agreement, description specificity, leftover template text, and trigger fixture shape. It runs entirely offline — no model, no network, no API key — which is what lets it gate CI. Generated skills now scaffold evals/triggers.json by default, so the labeled prompts exist before the skill is written rather than never.
Selection itself is model-driven, so the model stays inside the system under test and never inside the scorer. Trigger outcomes are graded with plain precision and recall, weighted toward precision: a skill that fires when it should not does more damage than one that occasionally misses.
Pointing the checker at the generator’s own output was a useful moment. It flagged that the default template has a description which never says when to use the skill, and a body still full of boilerplate — the tool arguing with itself, correctly.
The finding I did not expect
The package ships a semantic cache. Benchmarking it produced the most valuable result of the whole project.
The embedding function was a placeholder — a string hash mapped across 384 dimensions. Measured across ten paraphrase pairs and ten unrelated pairs:
- Paraphrase similarity: mean
-0.0118 - Unrelated similarity: mean
-0.0028 - Separation:
-0.0090
The separation is negative. Paraphrases scored lower than genuinely unrelated prompts — the signal was not weak, it was absent. Hit rate was 0% at every threshold from 0.99 down to 0.50. With the default similarity threshold of 0.95, a package named “semantic cache” was in practice an exact-match cache that could only match byte-identical strings.
The fix was to make the embedder injectable so a real model can be supplied, warn loudly when it is not, and ship the benchmark that proves the difference. The benchmark reports hit rate and false-hit rate separately, because a cache miss costs one model call while a false hit serves a wrong answer, and a single blended number hides the second.
I could have quietly swapped the embedding function. Publishing the measurement is more useful, and it is the honest version.
Tests, where there were none
The CLI’s test script was echo 'Tests coming soon' on a package already published to npm. It now has 22 tests covering frontmatter parsing — including colons inside values, quoted values, and CRLF line endings — every lint rule, trigger scoring, and fixture validation.
Why this is a senior project
The hard part of AI tooling is not generating files. It is leaving behind contracts that can be tested. A generator that also scaffolds the harness for what it generates is making an argument: AI infrastructure without evals is incomplete.
Open work: run the semantic-cache benchmark against a real embedding provider and document measured thresholds, generate MCP eval harnesses the same way skills now get trigger fixtures, and build a public plugin bundle pairing a skill, an MCP, commands, and evals.