Skip to content

docs(benchmark): point setup at cecli-cats, note benchmark_classic deprecation #612

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cattishly-rgb wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs(benchmark): point setup at cecli-cats, note benchmark_classic deprecation #612

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 15 additions & 6 deletions benchmark/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,13 @@ without any human review or supervision! The LLM could generate dangerous python
that harms your system, like this: `import os; os.system("sudo rm -rf /")`.
Running inside a docker container helps limit the damage that could be done.

The current harness (`benchmark.py`) organizes the exercises as "Cats": each
exercise carries `cat.yaml` metadata, enabling per-test selection and
deterministic subsets (e.g. `--hash-re '^0'`) for spot-checking models. The
canonical exercises source is
[ErichBSchulz/cecli-cats](https://github.com/ErichBSchulz/cecli-cats), a
Cats-formatted conversion of the polyglot benchmark.

## Usage

There are 3 main tasks involved in benchmarking:
Expand All @@ -41,17 +48,18 @@ There are 3 main tasks involved in benchmarking:
These steps only need to be done once.

```
ORG=Aider-AI
REPO=aider
ORG=cecli-dev
REPO=cecli
# Clone the main repo
git clone https://github.com/$ORG/$REPO.git

# Create the scratch dir to hold benchmarking results inside the main repo:
cd $REPO
mkdir tmp.benchmarks

# Clone the repo with the exercises
git clone https://github.com/$ORG/polyglot-benchmark tmp.benchmarks/polyglot-benchmark
# Clone the canonical exercises source for the Cats harness
git clone https://github.com/ErichBSchulz/cecli-cats tmp.benchmarks/cecli-cats


# Build the docker container
./benchmark/docker_build.sh
Expand All @@ -69,10 +77,10 @@ Launch the docker container and run the benchmark inside it:
./benchmark/docker.sh

# Run the benchmark:
./benchmark/benchmark.py a-helpful-name-for-this-run --model gpt-3.5-turbo --edit-format whole --threads 10 --exercises-dir polyglot-benchmark
./benchmark/benchmark.py a-helpful-name-for-this-run --model gpt-3.5-turbo --edit-format whole --threads 10 --exercises-dir cecli-cats

# Or with OpenRouter models (requires OPENROUTER_API_KEY environment variable):
./benchmark/benchmark.py openrouter-run --model openrouter/deepseek/deepseek-r1:free --edit-format whole --threads 10 --exercises-dir polyglot-benchmark
./benchmark/benchmark.py openrouter-run --model openrouter/deepseek/deepseek-r1:free --edit-format whole --threads 10 --exercises-dir cecli-cats
```

The above will create a folder
Expand Down Expand Up @@ -156,6 +164,7 @@ Note the roadmap priorities:

## Limitations

- `benchmark_classic.py` is deprecated (it predates the async Coder API).
- These scripts are not intended for use by typical `cecli` end users.
- Some of the old (?deprecated) tools are written as `bash` scripts, so it will be hard to use
them on Windows.
Expand Down