Evaluating Agent Skills

Why vetting matters

Adopting a skill means letting a file you didn't write steer (and sometimes execute code inside) your agent sessions. The ecosystem's supply is large and its quality variance is extreme: across the Agentiquette index, star counts show near-zero correlation with reliability scores. Popularity tells you a repo marketed well; only the artifact tells you it works.

The five axes

1. Trigger precision

Does the description state when to use it, in task-shaped language, with exclusions? Then test it: write three paraphrases of a task the skill should handle and run them fresh. Three fires out of three is a pass. This is the five-minute test almost nobody runs, and it catches the most common failure in the wild: skills that never fire. A skill that never fires is indistinguishable from no skill.

2. Workflow completeness

Read the steps as a skeptic. Does the happy path actually reach the goal? Do failure paths exist ("if the test still fails...") or does the procedure assume success at every step? Does any step require information the agent won't have? Incomplete workflows don't fail loudly; they get improvised around, which defeats the point of having a skill.

3. Boundaries and safety

Look for an explicit never-do list. Then look at what the skill can touch: skills that run shell commands, install dependencies, or bundle scripts are code-execution surfaces. Read the scripts. This is a security review, not a quality review, and it is non-optional for anything adopted team-wide.

4. Evidence and verification

Does the skill define how to check its own output? Evaluation readiness is a property of the artifact: a skill whose outputs are files, exit codes, or diffs can be checked; a skill whose output is "better behavior" mostly cannot. Prefer checkable. Where the skill claims examples, confirm they are worked examples (input, steps, verified result), not decorations.

5. Documentation truthfulness

Run the install instructions exactly as written. Either they work, or you have learned something important about everything else the document claims. Agentiquette scores install truthfulness inside Documentation for exactly this reason: it is the cheapest proxy for overall care.

Scoring it

The skill evaluation checklist packages these axes as check items with a verdict line. For team decisions, have two people run it independently; disagreement between reviewers is usually a discovery about the skill, not about the reviewers.

Calibrate the outcome

Adopt-with-edits is the normal result. Most good skills need boundaries tightened for your environment, a trigger phrase adjusted to your vocabulary, or a verification step added. That is not a defect; skills are starting points that expect adaptation. Reject outright when: triggers fail the paraphrase test and the body doesn't justify manual invocation, scripts don't survive review, or the workflow assumes an environment you'll never have.

And when a skill passes: re-evaluate on the skill's major updates. A skill is a dependency, and this checklist is its review gate, both at import and on upgrade.