neural-tools is a CLI for generating the parts of AI infrastructure that are tedious and easy to get subtly wrong: MCP servers, Claude commands, agents, skills, plugins, and the deploy glue around them. It is published on npm and the generated projects are ordinary readable files you own outright, not a framework you are stuck inside.
A skill is selected by its description
This is the part people miss. When Claude decides which skill to load, it reads the descriptions and picks. That paragraph is not documentation — it is routing logic that happens to be written in English. Too vague and the skill never fires and might as well not exist. Too broad and it hijacks requests meant for something else.
Nobody tests it, so neural-tools eval skill does: frontmatter validity, name matching its directory, whether the description is specific enough to disambiguate, leftover template text, fixture shape. It runs with no model and no network, which is what lets it gate CI.
Generated skills now ship with evals/triggers.json — labeled prompts that should and should not select the skill. Written at creation time they exist; deferred to later they never do.
Pointing the checker at the generator’s own template was a useful moment. It flagged that the default description never says when to use the skill and the body is still boilerplate. The tool disagreeing with its own output is a good sign it is measuring something real.
The semantic cache did not work
The package ships a semantic cache. I benchmarked it and found it could not match anything.
The embedding function was a placeholder — a string hash smeared across 384 dimensions. Across ten paraphrase pairs and ten unrelated pairs:
- Paraphrases: mean similarity
-0.0118 - Unrelated prompts: mean similarity
-0.0028
The separation is negative. Paraphrases scored lower than unrelated text, which means there was no signal at all — not a weak one. Hit rate was 0% at every threshold from 0.99 down to 0.50. At the shipped default of 0.95 it matched byte-identical strings and nothing else. A package with “semantic cache” in the name was an exact-match cache.
The fix was to make the embedder injectable, warn loudly when it is not supplied, and ship the benchmark. The benchmark reports hit rate and false-hit rate separately: a miss costs one model call, a false hit serves a wrong answer, and averaging them hides the one that matters.
I could have quietly swapped the function. Publishing the measurement is more useful to anyone who installed it.
Tests
The CLI’s test script was echo 'Tests coming soon' on a package already on npm. It is 22 tests now — frontmatter parsing including colons inside values, quoted values and CRLF endings, every lint rule, trigger scoring, fixture validation.
Next: generate eval harnesses for MCP servers the same way skills now get trigger fixtures, and run the cache benchmark against a real embedding model to publish measured thresholds instead of a default someone picked by feel.