Add tuning search based on CompileIQ

[bernhardmgruber]

This is mostly done by claude, trying to migrate the internal cub_tuning_evo scripts. This PR adds a simplified version using a single worker, running benchmarks on a single GPU.

Running:

mkdir build_tune & cd build_tune
cmake .. --preset cub-tuning
CUDA_VISIBLE_DEVICES=0 ../benchmarks/scripts/search_iq.py -R 'cub.bench.transform.babelstream.*' -a 'T{ct}=F32'
 ctk:  13.3.33
cccl:  v3.5.0.dev-121-g575176ff50
🧬 Generation:  0/50|░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░| [elapsed: 00:00 · eta: ?] Evaluating variant {'alg': 3, 'bif': 4, 'pref': 2, 'tpb': 768, 'unrl': 1, 'vsp2': 6}: 0.7940340093966018
Evaluating variant {'alg': 2, 'bif': 0, 'pref': 3, 'tpb': 128, 'unrl': 2, 'vsp2': 1}: 0.7870774540476999
Evaluating variant {'alg': 0, 'bif': -8, 'pref': 1, 'tpb': 640, 'unrl': 3, 'vsp2': 5}: 0.7340502424429781
Evaluating variant {'alg': 1, 'bif': -16, 'pref': 1, 'tpb': 384, 'unrl': 4, 'vsp2': 3}: 0.6930208162203446
Evaluating variant {'alg': 4, 'bif': 12, 'pref': 3, 'tpb': 1024, 'unrl': 2, 'vsp2': 2}: Build failed
Evaluating variant {'alg': 1, 'bif': -12, 'pref': 2, 'tpb': 896, 'unrl': 4, 'vsp2': 5}: 0.19958390512595525
Evaluating variant {'alg': 3, 'bif': 0, 'pref': 2, 'tpb': 128, 'unrl': 1, 'vsp2': 4}: 0.7767127690888875
Evaluating variant {'alg': 2, 'bif': 8, 'pref': 3, 'tpb': 640, 'unrl': 3, 'vsp2': 6}: 0.7872272803730863
Evaluating variant {'alg': 4, 'bif': -4, 'pref': 1, 'tpb': 768, 'unrl': 3, 'vsp2': 2}: Build failed
Evaluating variant {'alg': 0, 'bif': 16, 'pref': 1, 'tpb': 384, 'unrl': 4, 'vsp2': 1}: 0.7388724658257656
...

It's still a bit confusing, because after running, the database shows different results, but it looks like the score reported by analyze.py is just computed differently than the score passed to compile-iq.

$ ../benchmarks/scripts/analyze.py --top=100 cccl_meta_bench.db
cub.bench.transform.babelstream[T{ct}=F32]:
                                          variant     score      mins     means      maxs
9    bif_16.alg_2.tpb_768.unrl_3.pref_1.vsp2_6 ()  1.026887  1.000000  1.025528  1.200000
11    bif_8.alg_2.tpb_640.unrl_3.pref_3.vsp2_6 ()  1.026887  1.000000  1.025528  1.200000
6     bif_0.alg_2.tpb_128.unrl_2.pref_3.vsp2_1 ()  1.026641  1.000000  1.025313  1.200000
10    bif_4.alg_3.tpb_768.unrl_1.pref_2.vsp2_6 ()  1.025089  1.000000  1.023932  1.200000
7     bif_0.alg_3.tpb_128.unrl_1.pref_2.vsp2_4 ()  1.013059  0.999988  1.012499  1.200000
5     bif_0.alg_1.tpb_640.unrl_2.pref_2.vsp2_1 ()  1.012436  1.000000  1.011788  1.166667
0                                         base ()  1.000000  1.000000  1.000000  1.000000
1   bif_-12.alg_0.tpb_384.unrl_1.pref_3.vsp2_2 ()  0.962345  0.600000  0.963987  1.012048
8    bif_16.alg_0.tpb_384.unrl_4.pref_1.vsp2_1 ()  0.962344  0.600000  0.963986  1.012048
4    bif_-8.alg_0.tpb_640.unrl_3.pref_1.vsp2_5 ()  0.953269  0.600000  0.955280  1.012048
3   bif_-16.alg_1.tpb_384.unrl_4.pref_1.vsp2_3 ()  0.885473  0.500000  0.859326  1.001472
2   bif_-12.alg_1.tpb_896.unrl_4.pref_2.vsp2_5 ()  0.256298  0.048387  0.249873  0.409063

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7901deb3-6453-4a1e-a6e7-0fa11c0d14c5

📥 Commits

Reviewing files that changed from the base of the PR and between bd94eca and 6ac776b.

📒 Files selected for processing (2)

• benchmarks/scripts/tests/conftest.py
• benchmarks/scripts/tests/test_score.py

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

📝 Walkthrough

Summary by CodeRabbit

• New Features
  • Added automated benchmark search using brute-force or CompileIQ-based optimization.
  • Automatically selects the search strategy based on estimated workload size.
• Bug Fixes
  • Improved benchmark scoring by weighting available runtime configurations and handling cases with no valid results.
  • Prevented signal-handling errors when benchmarks run outside the main thread.
  • Added SQLite thread-safety validation and enabled safe cross-thread database connections.
• Tests
  • Added coverage for weighted scoring, skipped configurations, failed runs, and performance comparisons.

Walkthrough

Three changes update benchmark execution and search. Signal registration is restricted to the main thread. SQLite connections support validated cross-thread use. Runtime-axis scoring uses accumulated available weights. A new driver selects brute-force or CompileIQ search.

Changes

Benchmark Runtime and Scoring

Layer / File(s)	Summary
Thread-safe benchmark execution benchmarks/scripts/cccl/bench/bench.py, benchmarks/scripts/cccl/bench/storage.py	Signal handlers are registered only from the main thread. SQLite validates serialized support and enables cross-thread connections.
Runtime workload scoring and validation benchmarks/scripts/cccl/bench/bench.py, benchmarks/scripts/tests/conftest.py, benchmarks/scripts/tests/test_score.py	Scoring uses full runtime-axis metadata, accumulates weights for available speedup states, returns -inf when no weight is available, and normalizes by the accumulated weight. Tests cover weighting, skipped states, failed runs, and score ordering.

CompileIQ Benchmark Search

Layer / File(s)	Summary
CompileIQ search driver benchmarks/scripts/search_iq.py	Adds search sizing, parameter-space construction, benchmark objective evaluation, strategy selection, and seeker registration.

Suggested reviewers: gonidelis

Merge Risk: ⚪ Minimal · up to 6ac77

The PR adds a localized tuning-search workflow and related benchmark support; no actionable merge-blocking risk remains based on the available evidence.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

benchmarks/scripts/search_iq.py (1)

120-131: ⚡ Quick win

suggestion: Rename parameter or variable to clarify intent.

Line 125 passes num_rt_workloads to a parameter named num_objectives in get_num_expected_runs(). The function signature and iq_search() (line 83) use num_objectives=1, but the calculation here uses num_rt_workloads. Either rename the parameter in get_num_expected_runs() to reflect its actual usage, or clarify the relationship between RT workloads and the expected-runs heuristic.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 75cb46dd-cbc5-44dd-9b6d-7dd9bc33fb9b

📥 Commits

Reviewing files that changed from the base of the PR and between ee20627 and 09ec91a.

📒 Files selected for processing (3)

• benchmarks/scripts/cccl/bench/bench.py
• benchmarks/scripts/cccl/bench/storage.py
• benchmarks/scripts/search_iq.py

[bernhardmgruber]

ok, here is a confusing bit. analyze.py shows:

                            variant     score      mins     means      maxs
0                           base ()  1.000000  1.000000  1.000000  1.000000
1  bif_-12.tpb_256.pref_2.vsp2_2 ()  0.999754  0.984375  0.999780  1.023256
2   bif_-4.tpb_768.pref_2.vsp2_6 ()  0.997953  0.753906  0.997823  1.166667
3    bif_0.tpb_384.pref_1.vsp2_5 ()  0.996448  0.753906  0.996489  1.166667
4   bif_12.tpb_384.pref_3.vsp2_4 ()  0.996448  0.753906  0.996489  1.166667
5   bif_16.tpb_640.pref_3.vsp2_1 ()  0.996416  0.753906  0.996455  1.166667
6    bif_8.tpb_896.pref_1.vsp2_3 ()  0.977935  0.753906  0.978535  1.023256

Yet, the scores reported to CompileIQ are (ordered by variant as the list above):

Evaluating variant {'bif': -12, 'pref': 2, 'tpb': 256, 'vsp2': 2}: 0.7747535944721688
Evaluating variant {'bif': -4, 'pref': 2, 'tpb': 768, 'vsp2': 6}: 0.7833624486057084
Evaluating variant {'bif': 0, 'pref': 1, 'tpb': 384, 'vsp2': 5}: 0.7720734255442793
Evaluating variant {'bif': 12, 'pref': 3, 'tpb': 384, 'vsp2': 4}: 0.7722660680432303
Evaluating variant {'bif': 16, 'pref': 3, 'tpb': 640, 'vsp2': 1}: 0.7722450212177555
Evaluating variant {'bif': 8, 'pref': 1, 'tpb': 896, 'vsp2': 3}: 0.7580094070089615

analyze.py reports the score monotonically decreasing (highest score first). But the same order of benchmarked variants does neither monotonically increase or decrease, suggesting that the analysis score is not isomorphic to the CompileIQ score. This is either a bug in the score computation or beyond my understanding of the tuning framework.

@gevtushenko as the author of the tuning framework, I kindly ask for an explanation for this observation.

[gonidelis]

analyze.py reports the score monotonically decreasing (highest score first). But the same order of benchmarked variants does neither monotonically increase or decrease, suggesting that the analysis score is not isomorphic to the CompileIQ score. This is either a bug in the score computation or beyond my understanding of the tuning framework.

i think it's something we carry over from evo days. Bench.score normalizes weights over the full declared runtime-axis space, all sizes including 2^32, regardless of whether they were ever measured. It then sums weight * speedup only over states that actually ran. States that were skipped (OOM, compile failure, timeout) are simply absent from that sum but their weight mass stays in the denominator.

analyze.py doesn't have this problem because it builds its axis universe from DB contents. unmeasured states don't exist in there as it uses unique():

cccl/benchmarks/scripts/analyze.py

Lines 106 to 107 in 6992cf8

	for rt_axis in rt_axes:
	rt_axes_values[rt_axis] = list(df[rt_axis].unique())

[coderabbitai]

Actionable comments posted: 1

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0a09e9ad-602a-4978-b4bc-08d7d4f3c4cd

📥 Commits

Reviewing files that changed from the base of the PR and between 3d5d9f6 and 0d9d8d5.

📒 Files selected for processing (3)

• benchmarks/scripts/cccl/bench/bench.py
• benchmarks/scripts/cccl/bench/storage.py
• benchmarks/scripts/search_iq.py

🚧 Files skipped from review as they are similar to previous changes (2)

• benchmarks/scripts/cccl/bench/storage.py
• benchmarks/scripts/search_iq.py

[bernhardmgruber]

States that were skipped (OOM, compile failure, timeout) are simply absent from that sum but their weight mass stays in the denominator.

analyze.py doesn't have this problem because it builds its axis universe from DB contents. unmeasured states don't exist in there as it uses unique():

I believe this is also what claude found out and generated a solution for this PR. I asked @gevtushenko to verify this, since he authored the benchmarking infra structure. If @gonidelis can verify this as well, I am also happy! ;)

If the new logic is fine, then we can merge the PR and we get at least a single GPU CompileIQ-based tuning script. It may even support running on multiple GPUs in the same node, since I think CompileIQ supports this natively, but I think if we want to have a baseline per GPU, we may need to implement that ourselves. I have not looked into this yet.

[gevtushenko]

States that were skipped (OOM, compile failure, timeout) are simply absent from that sum but their weight mass stays in the denominator.

analyze.py doesn't have this problem because it builds its axis universe from DB contents. unmeasured states don't exist in there as it uses unique():

The importance-ordered score computation assign weights non-linearly based on position of the axis value. The idea was to make 2^24 more or less same importance as 2^28, but both to be more important than, say, 2^16. If we apply suggested change, 2^24 can become the first value, which will make it the least important.

We should avoid changing semantics based on the input. Below is an example of questionable variant being favored after the change:

I'll see if I can fix this differently and push to this PR.

[gevtushenko]

Will support multi-GPU in a follow up PR.