Move test datasets to the netzoo-data S3 bucket - #396
Merged
Merged
Conversation
The netzoo bucket that test_dragon and test_panda downloaded their data
from now returns 403 AccessDenied. Because the tests fetched it with
os.system('curl -O ...'), which ignores HTTP errors, the XML error body
silently replaced the data file, surfacing as an allow_pickle error in
test_dragon and a 2-row ground truth in test_panda on every branch.
The 14 files (dragonx1/2.npy and gt_panda1-12.txt) now live in the
netzoo-data bucket under the same key layout. Both tests fetch them
through a small _download helper that raises on any HTTP error, so a
future outage fails loudly at the download instead of corrupting the
comparison. Drop the now-unused os imports.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
katehoffshutta
added
ai-contrib
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the CI failures in
tests/test_dragon.py::test_dragonandtests/test_panda.py::test_pandathat currently hit every branch (e.g. #394, #395).Problem
Both tests download their data at test time from
netzoo.s3.us-east-2.amazonaws.com, which now returns403 AccessDeniedto anonymous requests. Because they fetched withos.system('curl -O …'), which ignores HTTP errors, the XML error body was silently written over the data file. That surfaced asValueError: Cannot load file containing pickled data when allow_pickle=Falseintest_dragonand a 2-row "ground truth" (DataFrame shape mismatch) intest_panda— misleading symptoms for what is a download failure. The old bucket cannot be read back from AWS either (authenticated access is denied by its policy).New bucket
The 14 files the tests need —
netZooPy/unittest_datasets/dragonx{1,2}.npyandnetZooPy/tutorial_datasets/gt_panda{1..12}.txt(≈115 MB) — were restored from verified local copies (correct shapes, MD5-checked) into a new bucket,netzoo-data(us-east-2), using the same key layout as before:https://netzoo-data.s3.us-east-2.amazonaws.com/netZooPy/…The bucket policy grants anonymous
s3:GetObjectonly (no listing). All 14 objects were verified anonymously fetchable and byte-identical to the originals. Only these test files were migrated — nothing else from the old bucket was available to copy.Code changes
tests/test_dragon.py,tests/test_panda.py: the base URL is a single module constantS3_BASE_URL, and downloads go through a small_download()helper (urllib.request.urlretrieve) that raises on any HTTP error, so a future outage fails at the download with the URL in the message instead of corrupting the comparison. The now-unusedimport osis dropped.Verification
pytest tests/test_dragon.py tests/test_panda.py→ 8 passed locally, downloading from the new bucket.E9,F63,F7,F82) is clean; no unused imports.Notes
tutorials/gpupanda,tutorials/panda/Controlling_The_Variance…) fetch fromgranddb.s3.amazonaws.com, which is also returning 403. Not exercised by CI; out of scope here.devel, re-running the checks on Make CONDOR tests robust to igraph stochasticity (#320) #394 and Add recipe/recipe.yaml to bumpversion (#377) #395 should turn them green — their failures are exactly these two tests.🤖 Generated with Claude Code