Skip to content

Fix the unit tests so they can actually run - #315

Open
Hotragn wants to merge 1 commit into
lyogavin:mainfrom
Hotragn:fix/runnable-unit-tests
Open

Hotragn wants to merge 1 commit into
lyogavin:mainfrom
Hotragn:fix/runnable-unit-tests

Conversation

@Hotragn

@Hotragn Hotragn commented Jul 28, 2026

Copy link
Copy Markdown

What

Make the two existing test modules in air_llm/tests/ actually runnable. Right now neither one does.

Why

test_compression.py fails at import. It does:

from airllm import compress_layer_state_dict, uncompress_layer_state_dict

but those functions live in airllm.utils and are not re-exported from the package root (__init__.py exports split_and_save_layers and NotEnoughSpaceException only). So collecting the module raises:

ImportError: cannot import name 'compress_layer_state_dict' from 'airllm'

Because it's a collection error, it takes down the whole test run, not just that module. Separately, the test quantizes with .cuda() tensors, so even once importable it errors rather than skips on a CPU-only machine.

test_automodel.py went stale in the v3.0 rewrite. It still asserts:

'garage-bAInd/Platypus2-7B': 'AirLLMLlama2',
'mistralai/Mistral-7B-Instruct-v0.1': 'AirLLMMistral',
'mistralai/Mixtral-8x7B-v0.1': 'AirLLMMixtral',

but the rewrite routes all standard architectures through the generic AirLLMBaseModel (only ARCH_OVERRIDES archs get dedicated classes), so those three assertions now fail. It also used a relative import (from ..airllm.auto_model import ...) that disagrees with the absolute imports in the other module — the two couldn't be collected by the same run — and it needed network access to resolve every repo id.

Changes

  • test_compression.py: import from airllm.utils; skip instead of error when CUDA is unavailable.
  • test_automodel.py: test the routing offline by faking the config lookup, asserting current behavior — custom archs (ChatGLM/QWen/Baichuan/InternLM) → dedicated classes; standard archs (Llama, Mistral, Mixtral, Qwen2/3, Phi3, Gemma2, DeepseekV3) and unknown/missing archs → AirLLMBaseModel. The original against-the-Hub check is kept with corrected expectations, gated behind AIRLLM_TEST_NETWORK=1.
  • Use absolute imports in both so one invocation collects the whole directory.

No production code is touched, and no new dependency is added (the network/GPU gates use unittest.skipUnless, not pytest markers, so the suite still runs under plain unittest).

Testing

From air_llm/, both runners agree — 3 passed, 2 skipped (the GPU test and the network test):

python -m pytest -v
python -m unittest discover -s tests -t .    # Ran 5 tests ... OK (skipped=2)

Also confirmed the new offline routing assertions genuinely exercise ARCH_OVERRIDES (they fail if the mapping is changed).

Related: I opened #312 proposing a small CI workflow. This PR is useful on its own regardless of that — but it's also the prerequisite that would let CI run the suite green, so I split it out rather than bundling it there.

@Hotragn

Hotragn commented Jul 29, 2026

Copy link
Copy Markdown
Author

This became more relevant with v3.1.0: the new tests/test_kimi_k3_split.py adds 9 tests, but the suite can't run as a directory because test_compression.py still fails at import, and a collection error aborts the whole run:

$ cd air_llm && python -m pytest tests/
9 tests collected, 1 error
ERROR tests/test_compression.py
!!!!!!!! Interrupted: 1 error during collection !!!!!!!!

(Checked out v3.1.0 / 64a4e4f clean to confirm.) The 9 Kimi tests are collected fine — they're just never executed, because from airllm import compress_layer_state_dict raises ImportError (those live in airllm.utils and aren't re-exported from the package root). So today they only run if you name the file explicitly.

This PR fixes exactly that, and nothing else needs to change: the new Kimi tests already use the absolute-import style (from airllm.utils import split_and_save_layers) this PR standardizes on, so they pass unmodified. After it, python -m pytest from air_llm/ runs the whole directory green — the Kimi tests included.

Neither test module in air_llm/tests/ currently runs:

test_compression.py fails at import. It does
`from airllm import compress_layer_state_dict, uncompress_layer_state_dict`,
but those live in airllm.utils and are not re-exported from the package root,
so collecting the module raises ImportError. It also quantizes with .cuda()
tensors, so it errors on a CPU-only machine even once importable.

test_automodel.py went stale in the v3.0 rewrite. It asserts that
garage-bAInd/Platypus2-7B, Mistral and Mixtral resolve to AirLLMLlama2 /
AirLLMMistral / AirLLMMixtral, but the rewrite routes all standard
architectures through the generic AirLLMBaseModel, so those assertions now
fail. It also used a relative import (`from ..airllm.auto_model import ...`)
that disagreed with the absolute imports in the other test module, and it
needed network access to resolve every repo id.

- test_compression.py: import from airllm.utils, and skip (rather than error)
  when CUDA is unavailable.
- test_automodel.py: test the routing offline by faking the config lookup, and
  cover the current behavior (custom archs -> dedicated classes, standard and
  unknown archs -> AirLLMBaseModel). Keep the against-the-Hub check as an
  env-gated test (AIRLLM_TEST_NETWORK=1).
- Use absolute imports in both, so one test run collects the whole directory.

Verified with both `python -m pytest` and `python -m unittest discover -s tests
-t .` from air_llm/: 3 passed, 2 skipped (GPU + network), no new dependency.
@Hotragn
Hotragn force-pushed the fix/runnable-unit-tests branch from c914bfb to f293fb8 Compare July 29, 2026 18:18
@Hotragn

Hotragn commented Aug 21, 2026

Copy link
Copy Markdown
Author

Refreshed against v3.2.0 — the blast radius grew. With test_qwen3_8_split.py added alongside test_kimi_k3_split.py, the collection error now takes down 15 tests instead of 9:

$ cd air_llm && python -m pytest tests/
15 tests collected, 1 error
ERROR tests/test_compression.py
!!!!!!!! Interrupted: 1 error during collection !!!!!!!!

So every test you've written for Kimi K3 and Qwen3.8 is collected and then discarded, because from airllm import compress_layer_state_dict in test_compression.py raises ImportError (those live in airllm.utils, never re-exported from the package root). They only run today if you name each file explicitly.

This PR is still MERGEABLE/clean against v3.2.0 and needs no rebase. It's test-only, touches no production code, adds no dependency, and both of your new test files pass unmodified afterwards — they already use the absolute-import style it standardises on.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant