fix: don't disable Actor exit_process just because scrapy is importable#1049
Draft
vdusek wants to merge 3 commits into
Draft
fix: don't disable Actor exit_process just because scrapy is importable#1049vdusek wants to merge 3 commits into
vdusek wants to merge 3 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1049 +/- ##
==========================================
+ Coverage 91.75% 91.78% +0.02%
==========================================
Files 50 51 +1
Lines 3215 3224 +9
==========================================
+ Hits 2950 2959 +9
Misses 265 265
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_get_default_exit_processdecided theexit_processdefault from whetherimport scrapysucceeded, not from whether the Actor is actually running under Scrapy. Any image where Scrapy arrives as a transitive dependency therefore silently disabled thesys.exit()on Actor exit. On the platform, an Actor that signals failure viaActor.fail(exit_code=1)without raising would keep running, end with exit code 0, and get marked SUCCEEDED.Detection now lives in
apify.scrapy._detectionand treats the Actor as running under Scrapy when either of these holds:run_scrapy_actoris driving the process. It sets an explicit flag before the Actor's main coroutine runs, hence before the Actor initializes.SCRAPY_SETTINGS_MODULEenvironment variable is set, by the Scrapy CLI or the Actor's entry point. This is the fallback and is already documented as a detection mechanism._actor.pyconsults this detector only throughsys.modules, so a non-Scrapy Actor never importsapify.scrapy. Importing it pulls in Scrapy itself (~860 ms), and the old importability check would disableexit_processfor exactly such images.Test environments no longer get
exit_process=Falseimplicitly through the scrapy import. The unit_isolate_test_environmentfixture now forces the default toFalseexplicitly, since production code must not detect the test environment (see #641). As a side effect, the unit suite no longer needs thescrapyextra installed to avoidSystemExit.Behavioral change: an Actor that runs Scrapy without using
run_scrapy_actorand without settingSCRAPY_SETTINGS_MODULEnow defaultsexit_processtoTrue. The documented Scrapy setup and the project template always do at least one of those, so legitimate Scrapy Actors are unaffected. I kept this as a plainfix:rather than a breaking change since the old behavior was itself the bug; promote the squash title tofix!:at merge if a major bump is preferred.