fix!: use Scrapy's fingerprint for Scrapy request unique keys

[vdusek]

When converting a Scrapy request to an Apify request, to_apify_request computed the request queue unique key via Crawlee's Request.from_url (use_extended_unique_key=True). Crawlee's normalize_url lowercases the whole URL and strips utm_* params, and the extended key hashes a whitelist of headers. Scrapy's own deduplication (RFPDupeFilter) instead uses a fingerprint that canonicalizes the URL case-sensitively (w3lib.url.canonicalize_url), keeps utm_*, and ignores headers.

The two disagreed in both directions: /Products/Item-A and /products/item-a (and ?utm_source=x&id=1 vs ?id=1) collapsed onto the same unique key, so sites with case-sensitive paths (S3-backed, Django) silently lost pages vanilla Scrapy crawls; meanwhile requests differing only in headers Scrapy ignores were over-split into separate entries.

to_apify_request now derives the unique key from Scrapy's request fingerprint (spider.crawler.request_fingerprinter.fingerprint(request).hex(), honoring a custom REQUEST_FINGERPRINTER_CLASS, with a fallback to the default fingerprinter when the spider is not bound to a crawler), matching RFPDupeFilter semantics. The dont_filter (always-enqueue) and explicit apify_request_unique_key (roundtrip continuity) paths are unchanged.

Breaking change

Unique keys for converted Scrapy requests change value. A run resuming against a request queue that was populated by an older SDK version will not deduplicate new requests against the pre-existing entries, so some already-seen URLs may be re-enqueued once. New queues and fresh runs are unaffected.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.76%. Comparing base (ed1d8eb) to head (eb5e263).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1047   +/-   ##
=======================================
  Coverage   91.75%   91.76%           
=======================================
  Files          50       50           
  Lines        3215     3218    +3     
=======================================
+ Hits         2950     2953    +3     
  Misses        265      265           

Flag	Coverage Δ
e2e	?
integration	57.33% <0.00%> (-0.06%)	⬇️
unit	83.09% <100.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[vdusek]

Scrapy's deduplication doesn't handle two things that matter for scraping:

• Accept-Language headers - requests to the same URL in different languages get deduplicated, so the alternate language content is never fetched.
• utm_* URL params - these tracking params don't affect the content, yet Scrapy keeps them in the fingerprint, so the same page gets crawled more than once.

Based on these findings, I've decided to stay with the Apify (Crawlee) deduplication system. Closing this.