Skip to content

About

AMD's testing harness for AI Agent Skills

Resources

Stars

8 stars

Watchers

0 watching

Forks

Repository files navigation

Skillscope

AMD Agent Skills Python Claude Code GitHub Actions License: MIT

Skillscope

A test harness for AI agent skills. Install the harness, or point a workflow at your repo. Skillscope enables four main types of tests:

  • Structural: Checks whether the skill is well structured (contains the expected files, a proper hierarchy, etc.).
  • Routing: Checks whether the skill triggers when it should and stays quiet when it shouldn't. You may also point to more than one skill here to explore how skills interfere with each other's routing.
  • Behavioral: Runs the agent end to end and uses an LLM judge to confirm the skill behaves correctly.
  • Staleness: Runs the behavioral cases a second time with the skill withheld, and compares. Tells you whether the cases still need the skill, or whether the model has caught up and the suite is passing on its own.

All four read one dataset per skill, <skill>/evals/evals.json. How to write one is in docs/authoring-evals.md.

Important

Early Days: Flags, workflow inputs, and report formats may evolve quickly as this repo takes shape.

Quick start

Run from the root of the repo you want tested.

uv tool install --system-certs git+https://github.com/amd/skillscope

skillscope structural                        # no agent, no tokens
skillscope structural --external             # the same, plus checking external URLs
skillscope behavioral --skill my-skill       # needs an authenticated `claude` CLI
skillscope routing --routing-room my-skill,its-neighbour  # also needs a key in the environment
skillscope staleness --skill my-skill        # the same key: its baseline has to be an empty room

routing needs ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN set, and refuses to start without one — a claude login on its own is not enough. The key is what lets the run point the CLI at its own config directory; without that, the runner's own ~/.claude/skills join the room for every case and the scores describe a room nobody asked for. behavioral installs one skill and grades what the agent produced, so a stray skill is at worst noise there, and a plain claude login is fine.

staleness is in routing's camp, not behavioral's, and asks the same credential question on Claude Code. Its baseline half has to be a room with no skills in it, so the runner's own are not noise there — they are the measurement. Either set one of those variables, or use --engine claude-code, whose guest has no user-level skills at all. It is Claude Code only: unlike routing, which scales its refusal per harness, it refuses any --harness other than claude-code, because there is no staleness leg for another one yet.

Run it from your CI

Simply add an evals.json file to your skill and a workflow that points to our reusable workflow. This will trigger the three graders the workflow runs -- structural, routing and behavioral -- directly from your repo. staleness is not one of them: it runs both halves of every case and is meant to be asked for deliberately rather than on every push.

Skillscope

jobs:
  evals:
    uses: amd/skillscope/.github/workflows/reusable.yml@v0.1.3
    secrets:
      api_key: ${{ secrets.ANTHROPIC_API_KEY }}
    with:
      skills: skills/*

You may also choose to customize or even disable some tests if you prefer it. See docs/usage.md for details.

Commands

Command Does
skillscope structural Skill folders, datasets, and markdown references.
skillscope routing Which skill fires, with several installed together.
skillscope behavioral What a skill does once it has fired.
skillscope staleness Whether those cases still need the skill, and what the agent spends without it.
skillscope select The CI plan for a change, as JSON.
skillscope list-skills The skills that have a dataset, as JSON.
skillscope template The dataset template a new skill starts from.

--help on any of them is the reference. Reports go to stdout as markdown, to $GITHUB_STEP_SUMMARY under Actions, and to JSON under .skillscope/runs/ in the repo under test — worth adding to .gitignore.

Docs

Development

python -m pip install -e .
python -m unittest discover -s tests -t .

Runtime dependencies are PyYAML for frontmatter, inspect-ai for the eval framework both engines are built on, and anthropic for the model client. inspect-ai became required when the last stdlib-only engine was removed; the test suite still runs on the standard library alone, and builds a throwaway repo in a temp directory rather than reading the tree it runs in.

License

MIT. See LICENSE.

About

AMD's testing harness for AI Agent Skills

Resources

Stars

8 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages