Last updated 08-12-2026
Category:
Reviews:
Join thousands of AI enthusiasts in the World of AI!
mshumer/gpt-prompt-engineer - GitHub
mshumer/gpt-prompt-engineer is an open source prompt engineering toolkit that generates, tests, and ranks candidate prompts for a task you define. You describe the use case, supply test cases, and the notebooks create multiple prompt variants, run them against every test case, and sort results with an ELO rating system starting at 1200. It ships as Jupyter notebooks you can run in Google Colab or locally.
Most prompt tools help you write one prompt at a time. gpt-prompt-engineer treats prompt selection like a tournament: dozens of candidates compete on your test cases, and the highest ELO scores surface the winners. Separate notebooks cover classification tasks, Claude 3 Opus with auto-generated test cases, and Opus-to-Haiku conversion for cheaper inference. Optional Weights & Biases and Portkey logging trace each run.
ML engineers, prompt engineers, and AI developers use it when they need reproducible prompt tuning instead of manual trial and error. The repo has 9.7k GitHub stars and runs on your own OpenAI or Anthropic API keys. It is free under the MIT license.
Generates multiple prompt candidates from a task description and user-supplied test cases
Ranks prompts with an ELO rating system starting at 1200 per candidate
Supports GPT-4, GPT-3.5-Turbo, and Claude 3 Opus model backends
Classification notebook scores true/false test cases and prints a results table
Claude 3 notebook auto-generates test cases from input variable definitions
Opus-to-Haiku conversion notebook cuts latency and cost while preserving output quality
Optional Weights & Biases and Portkey logging for experiment tracking
Automates prompt tournament testing instead of manual one-off rewrites.
ELO ranking makes it easy to compare dozens of candidates on the same test cases.
Multiple notebooks cover classification, Claude 3, and Opus-to-Haiku conversion.
Requires your own OpenAI or Anthropic API keys and notebook setup.
Generating many prompt candidates can get expensive at scale.
No hosted UI; you work entirely inside Jupyter notebooks.
Is mshumer/gpt-prompt-engineer free to use?
Yes, mshumer/gpt-prompt-engineer is free and open source under the MIT license on GitHub. You run the Jupyter notebooks in Google Colab or locally and pay only for your own OpenAI or Anthropic API usage.
How does gpt-prompt-engineer rank prompts?
mshumer/gpt-prompt-engineer tests each generated prompt against your test cases and ranks them with an ELO rating system. Every prompt starts at 1200 ELO and gains or loses points based on head-to-head performance on your inputs.
Which models does gpt-prompt-engineer support?
mshumer/gpt-prompt-engineer supports GPT-4, GPT-3.5-Turbo, and Claude 3 Opus depending on which notebook you run. The Opus-to-Haiku conversion notebook pairs Claude 3 Opus planning with Claude 3 Haiku generation for lower cost.
How do you run mshumer/gpt-prompt-engineer?
mshumer/gpt-prompt-engineer runs as Jupyter notebooks hosted on GitHub. Open gpt_prompt_engineer.ipynb in Google Colab, add your OpenAI API key, define a use-case description and test cases, then call generate_optimal_prompt to start the tournament.
Does gpt-prompt-engineer handle classification tasks?
Yes, mshumer/gpt-prompt-engineer includes a Classification Version notebook that evaluates prompts on true/false test cases. Each test case pairs a prompt with an expected output label, and the notebook prints a score table per candidate.
How many GitHub stars does gpt-prompt-engineer have?
mshumer/gpt-prompt-engineer has about 9,700 GitHub stars and 681 forks as listed on the repository page. The project is maintained by Matt Shumer and last received a README update in October 2025.