Scrapy 2.14.2: Security Patch and HTTP Compliance Changes

Here at industry sources, we tracked a serious vulnerability in Scrapy 2.14.2 that affects how the framework processes the Referrer-Policy header. Previously, values from this header were being executed as Python callables - meaning arbitrary code could run during header processing. This is a code execution risk that bypasses normal validation layers. If your crawler was handling untrusted websites, malicious actors could inject executable content through response headers.

The fix prevents header values from being interpreted as callable objects. This is a straightforward security patch, but the implications are significant. Any production crawler using an older version should upgrade immediately. The vulnerability doesn't require active exploitation - it happens passively when processing responses from compromised or malicious sources.

For builders, this means auditing any custom middleware or header-processing logic you've written. If you're relying on Scrapy's default behavior, the patch handles the issue. But if you're doing custom header manipulation, ensure you're not accidentally passing unsanitized header values to eval(), exec(), or similar functions.

• Referrer-Policy header values were executed as Python callables
• Affects all versions before 2.14.2 - upgrade is non-negotiable
• No workaround exists - patching is the only solution
• Custom middleware may need review for similar patterns

The second change in 2.14.2 addresses how 301 (Moved Permanently) redirects are handled. Previously, if a server responded with 301 to a POST request, Scrapy would maintain the POST method. The new behavior converts 301 redirects of POST requests to GET requests, aligning with the WHATWG fetch standard and browser behavior.

This might seem like a minor detail, but it affects how your crawler behaves when scraping sites with aggressive redirects. Many older APIs and websites use 301 redirects incorrectly - they treat them as generic redirects instead of permanent ones. When your crawler switches from POST to GET, it changes the request payload structure, which can cause the target server to return different responses or reject the request entirely.

Builders need to test crawlers against their target sites immediately after upgrading. If your scraper relies on POST data surviving a 301 redirect, this update will break it. The fix is correct from a standards perspective, but your application logic needs to account for it. Consider whether you should catch redirects explicitly and handle them differently, or if your target sites need to be fixed to use 307 or 308 redirects instead.

• 301 redirects of POST requests now become GET requests
• Aligns with WHATWG fetch standard and browser behavior
• Request payload will be lost in the redirect
• Sites using 301 for non-permanent redirects will break
• Use 307 or 308 redirects if you need method preservation

Upgrading to 2.14.2 requires a structured approach. Start by running your test suite against the new version in a staging environment. Focus on crawlers that hit your most critical targets - especially those that submit POST requests or handle complex redirect chains. The security fix is backwards-compatible, but the redirect behavior change can alter crawler output.

Set up monitoring for redirect behavior before and after the upgrade. Log cases where 301 redirects occur, and compare response codes and content between versions. If you see failures after upgrading, check whether the target site was responding to POST requests or using redirects unexpectedly. You may need to adjust your site-specific rules or contact site owners about redirect strategy.

For teams with multiple crawlers, stagger the rollout. Upgrade low-risk crawlers first - those scraping stable, well-behaved websites. Monitor for two weeks before upgrading production crawlers that handle sensitive data or high-traffic targets. This conservative approach prevents widespread failures if your site mapping has issues you didn't discover in testing. The momentum in this space continues to accelerate.

• Test in staging with full crawl simulations before production
• Focus testing on POST-based crawlers and redirect-heavy sites
• Monitor redirect behavior changes for 48-72 hours post-upgrade
• Stagger rollout across low-risk to high-risk crawlers
• Have a rollback plan ready in case of unexpected failures