Skip to content

Make CONDOR tests robust to igraph stochasticity (#320) - #394

Open
marouenbg wants to merge 3 commits into
netZoo:develfrom
marouenbg:fix/condor-stochastic-tests-320
Open

marouenbg wants to merge 3 commits into
netZoo:develfrom
marouenbg:fix/condor-stochastic-tests-320

Conversation

@marouenbg

@marouenbg marouenbg commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #320 (follow-up to #312).

Problem

CONDOR initialises its community structure with igraph's randomised Leiden algorithm, so the integer community labels it returns are arbitrary and get permuted between igraph releases. tests/test_condor.py pinned the exact labels with assert_frame_equal against per-igraph-version ground-truth files — which is why both gh_*_memb.txt and gh_*_memb_v9igraph.txt existed. Comparing them shows they encode the identical partition (adjusted Rand index = 1.000) with different numbering. The test was fragile to any igraph update, and the point made in #312 stands: a single fixed labelling is not the right thing to assert on.

Changes

tests/test_condor.py — rewritten so it never compares raw labels:

  • partition vs. a single reference up to relabelling (adjusted Rand index ≥ 0.9);
  • final bipartite modularity matches the reference (0.526667, tol 1e-4) — ARI pins the partition, and this also catches a wrong modularity computation (an inflated bipartite_modularity now fails the suite);
  • structural checks (node sets, one community per node);
  • reproducible for a fixed seed and stable across seeds — the "run it multiple times" check recommended in Please refer to igraph instead of python-igraph #312;
  • the netzoopy condor CLI path, run with initial_method="LDN": the CLI default is Louvain, and depending on igraph's RNG stream 10–29% of seeds reach a different BRIM optimum (Q 0.5246, ARI 0.78), whereas Leiden reaches the reference partition for every seed tested (500/500 across igraph 0.9.11–1.0.0);
  • outputs go to pytest's tmp_path instead of the repo root (the old test left tar2.txt/reg2.txt etc. behind as untracked files).

netZooPy/condor/condor.py (small):

  • removed two debug prints that fired even with silent=True;
  • Leiden goes through a _leiden helper that uses igraph's current resolution= keyword and falls back to the deprecated resolution_parameter= only when that keyword is rejected (igraph < 0.10.3). Verified empirically: fallback taken on igraph 0.10.0–0.10.2, primary branch with no deprecation warning on 0.10.4 / 0.10.8 / 0.11.2 / 1.0.0, identical modularity throughout.

Removed the now-redundant tests/condor/gh_*_memb_v9igraph.txt (nothing references them).

Verification

  • 4/4 tests pass locally and in CI; CI's blocking flake8 rule is clean.
  • Independently re-run across six environments (Python 3.9 / 3.12 × igraph 0.9.11, 0.10.2, 0.11.9, 1.0.0): all assertions pass, while the raw labels differ between igraph eras exactly as the old test used to break on.
  • Mutation checks: a label permutation of the reference passes; merging communities (ARI 0.72), moving nodes (0.46), a shuffled partition (−0.08), a wrong node name, and an inflated modularity all fail.

CI note (unrelated to this PR)

The workflow currently fails on tests/test_dragon.py::test_dragon and tests/test_panda.py::test_panda, which are untouched here: both curl -O their data from netzoo.s3.us-east-2.amazonaws.com at test time, and that bucket now returns 403 AccessDenied, so the XML error body is written over the data file. tests/test_condor.py passes in the same runs. This affects every branch (including devel on its next run) until the bucket permissions are restored or the tests stop depending on S3.

Out of scope (pre-existing, surfaced during review)

  • run_condor(..., initial_project=True) raises IndexError in initial_community (projection path).
  • The click command never forwards --resolution to initial_community, and declares it type=int.

🤖 Generated with Claude Code

marouenbg and others added 2 commits September 3, 2026 16:29
CONDOR seeds its community structure with igraph's randomised Leiden
algorithm, so the integer community labels it returns are arbitrary and
get permuted between igraph releases. The old test pinned the exact
labels with assert_frame_equal against per-igraph-version ground-truth
files, which is why two copies existed: they encode the identical
partition (adjusted Rand index 1.0) with different numbering.

The rewritten tests never compare raw labels. They check the partition
against a single reference up to relabelling (ARI >= 0.9), that the
final bipartite modularity is at least the reference value (the
objective CONDOR maximises), the structure of the output, that a fixed
seed is reproducible and that results are stable across seeds, plus the
command-line entry point. Outputs go to pytest's tmp_path instead of
the repository root. The now-redundant *_v9igraph ground-truth files
are removed.

In condor.py, drop two debug prints that fired even in silent mode and
route Leiden through a helper that tolerates igraph's rename of
resolution_parameter to resolution, which silences the deprecation
warning on current igraph while keeping older versions working.

Fixes netZoo#320

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Assert the final modularity matches the reference instead of being at
  least as large: ARI already pins the partition, so ">=" only lost the
  ability to catch a wrong modularity value (a bipartite_modularity that
  returned 2*Q passed the whole suite).
- Run the CLI test with initial_method="LDN". The click default is
  Louvain, whose result varies with the seed and with igraph's RNG stream
  (10-29% of seeds reach a different BRIM optimum across igraph
  versions); Leiden reaches the reference partition for every seed.
- Only fall back to resolution_parameter= when the TypeError is about
  that keyword, so unrelated TypeErrors are not retried under the
  deprecated argument.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant