prompt-testing

An AI prompt optimizer for writing better prompts and getting better AI results.

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

LLM Reasoning and Generation Benchmark. Evaluate LLMs in complex scenarios systematically.

Ollama Model Test - Figure out the best model for the task

Phoenix-native LLM evaluation, prompt testing, model comparison, red teaming, and observability for Elixir

Blind A/B testing for AI agent skills, .cursorrules, CLAUDE.md, and system prompts. Detects placebo and harmful instructions.

OMK helps people and agents build, maintain, and reuse valuable knowledge from real work and collaboration, with readable content, traceable sources, and revision history for future tasks.

Published: 2026-10-18

Turn failed AI agent runs into replayable regression tests. Catch regressions before you ship.

Snapshot testing for LLM apps and agents, built to run locally and block regressions in CI.

Prompture is an API-first library for requesting structured JSON output from LLMs (or any structure), validating it against a schema, and running comparative tests between models.

Mutation testing for your LLM evals — degrade the system under test and check whether your evals would actually catch a regression.

Semantic diff for LLM prompts — compare prompt versions like git diff

Open-source prompt optimization tool — improve your LLM prompts with AI-powered suggestions and explanations.

Test, compare, and optimize your AI prompts in minutes

Detect prompt regressions before they reach production — per-category accuracy scoring, deterministic validation, and False Improvement detection. Pure Python, zero dependencies.

prompt-evaluator is an open-source toolkit for evaluating, testing, and comparing LLM prompts. It provides a GUI-driven workflow for running prompt tests, tracking token usage, visualizing results, and ensuring reliability across models like OpenAI, Claude, and Gemini.

Open-source simulation harness for testing AI agents. Simulate LLM and tool calls to test edge cases, failure paths, and agent logic without live API calls.

Community Plugin for Genkit to use Promptfoo