Scrapy 2.14.2: Security patch and HTTP standard alignment

Here at industry sources, we tracked the Scrapy 2.14.2 release and identified a critical security flaw that should concern any builder running older versions. The issue: values from the Referrer-Policy header were being executed as Python callables. This means untrusted header values could trigger arbitrary code execution in your scraping pipeline - a significant attack surface if you're crawling adversarial domains or processing user-controlled responses.

The fix patches this by treating Referrer-Policy values as data, not code. If you're using Scrapy in production, this is the kind of vulnerability that compounds risk over time. A single malicious header in thousands of requests could slip through and compromise your application.

The secondary improvement addresses HTTP specification compliance: 301 redirects on POST requests now correctly convert to GET requests, aligning with RFC 7231. This matters because non-compliant behavior can break integrations with strict API servers or cause data loss when your scraper expects different response shapes than what it actually receives.

• Referrer-Policy header values no longer execute as Python code
• POST to 301 redirect now follows HTTP standard (becomes GET)
• Affects all Scrapy versions prior to 2.14.2

HTTP redirect handling seems mechanical until it breaks your scraper. The old behavior (maintaining POST on 301) was non-standard but predictable. The new behavior follows RFC 7231 - the authoritative HTTP standard. If you've built workarounds to handle this quirk, you need to audit them.

The practical impact depends on your targets. If you're scraping web services that return 301 redirects on POST requests, your pipeline will now send GET requests to the redirected URL. This changes your request shape and can trigger different server behavior. Some services expect the POST data to follow through; others expect the redirect to become a GET. Now Scrapy matches what browsers do.

Builders using Scrapy for API scraping should review redirect handling in their middleware and callbacks. If you're relying on POST data persistence across 301s, you'll need to adjust your approach or pin to an older version (not recommended for security reasons).

• Old behavior: 301 redirects preserved the original HTTP method
• New behavior: 301 redirects convert POST to GET per RFC 7231
• Requires testing against your actual targets before upgrading

The security fix is urgent - the vulnerability is real and easy to exploit if an attacker controls responses. However, the redirect handling change is a breaking change that needs careful testing. Here's the operator-first approach: stage the upgrade in a non-production environment first, then run your scraper against a sample of real targets to verify redirect behavior hasn't broken your extraction logic.

If you have scrapers hitting services that return 301 redirects, capture the before/after behavior. Does the GET request to the redirected URL return the same content? Do redirects to different domains now behave differently? These are the questions to answer before rolling out.

For most builders, upgrading is the right move - security vulnerabilities compound over time and the redirect alignment is actually correct behavior. Plan for it as a maintenance task rather than a hotfix. The momentum in this space continues to accelerate.

• Prioritize upgrading in development first, then stage to production
• Test 301 redirect handling against your specific target domains
• Monitor error rates and response patterns after deployment