3dE6 dZddlZddlmZddlmZddlmZddlm Z ddl m Z ddl m Z mZdd lmZdd lmZdd lmZd Zd ZdZdZdZdZdZdZdZdZGddZGddeZ GddeZ!GddeZ"GddeZ#Gd d!eZ$Gd"d#eZ%Gd$d%eZ&Gd&d'eZ'Gd(d)e!Z(d*e e!e"e#e$e%e&e'e(f DZ)e!e)d+<d0d-Z*Gd.d/Z+dS)1z` RefererMiddleware: populates Request referer field, based on the Response which originated it. N)Tuple)urlparse)safe_url_string)signals) NotConfigured)RequestResponse) load_object) to_unicode strip_url)aboutblobdata filesystemz no-referrerzno-referrer-when-downgradez same-originoriginz strict-originzorigin-when-cross-originzstrict-origin-when-cross-originz unsafe-urlzscrapy-defaultcdeZdZUeZeedfed<eed<dZdZ dZ d dZ d Z d Z d Zd S)ReferrerPolicy.NOREFERRER_SCHEMESnamectN)NotImplementedErrorself response_url request_urls @lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.pyreferrerzReferrerPolicy.referrer)s!###cft|j|jvr||SdSr)rschemerr rurls rstripped_referrerz ReferrerPolicy.stripped_referrer,s7 C== t'> > '>>#&& & ' 'r cft|j|jvr||SdSr)rr"rrr#s rorigin_referrerzReferrerPolicy.origin_referrer0s7 C== t'> > $;;s## # $ $r Fc2|sdSt|ddd|S)a https://www.w3.org/TR/referrer-policy/#strip-url If url is null, return no referrer. If url's scheme is a local scheme, then return no referrer. Set url's username to the empty string. Set url's password to null. Set url's fragment to null. If the origin-only flag is true, then: Set url's path to null. Set url's query to null. Return url. NT)strip_credentialsstrip_fragmentstrip_default_port origin_onlyr )rr$r,s rr zReferrerPolicy.strip_url4s7 4 "##     r c0||dS)zLReturn serialized origin (scheme, host, path) for a request or response URL.T)r,r r#s rrzReferrerPolicy.originLs~~ct~444r c`t|}|jdvrdS||S)N)rF)rr" tls_protected)rr$ parsed_urls rpotentially_trustworthyz&ReferrerPolicy.potentially_trustworthyPs6c]]   ) 5!!#&&&r c.t|jdvS)N)httpsftps)rr"r#s rr/zReferrerPolicy.tls_protectedWs}}#'888r NF)__name__ __module__ __qualname__ LOCAL_SCHEMESrrstr__annotations__rr%r'r rr1r/r rrr$s*7c3h777 III$$$'''$$$    0555'''99999r rc(eZdZUdZeZeed<dZdS)NoReferrerPolicya( https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer The simplest policy is "no-referrer", which specifies that no referrer information is to be sent along with requests made from a particular request client to any origin. The header will be omitted entirely. rcdSrr<rs rrzNoReferrerPolicy.referrerfstr N) r6r7r8__doc__POLICY_NO_REFERRERrr:r;rr<r rr>r>[sA#D#"""r r>c(eZdZUdZeZeed<dZdS)NoReferrerWhenDowngradePolicyaW https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer-when-downgrade The "no-referrer-when-downgrade" policy sends a full URL along with requests from a TLS-protected environment settings object to a potentially trustworthy URL, and requests from clients which are not TLS-protected to any origin. Requests from TLS-protected clients to non-potentially trustworthy URLs, on the other hand, will contain no referrer information. A Referer HTTP header will not be sent. This is a user agent's default behavior, if no policy is otherwise specified. rc||r||r||SdSr)r/r%rs rrz&NoReferrerWhenDowngradePolicy.referrer{sK!!,// 843E3Ek3R3R 8)),77 7 8 8r N) r6r7r8r@!POLICY_NO_REFERRER_WHEN_DOWNGRADErr:r;rr<r rrCrCjsA  2D#11188888r rCc(eZdZUdZeZeed<dZdS)SameOriginPolicya https://www.w3.org/TR/referrer-policy/#referrer-policy-same-origin The "same-origin" policy specifies that a full URL, stripped for use as a referrer, is sent as referrer information when making same-origin requests from a particular request client. Cross-origin requests, on the other hand, will contain no referrer information. A Referer HTTP header will not be sent. rc||||kr||SdSrrr%rs rrzSameOriginPolicy.referrersE ;;| $ $ K(@(@ @ 8)),77 7 8 8r N) r6r7r8r@POLICY_SAME_ORIGINrr:r;rr<r rrGrGsA#D#"""88888r rGc(eZdZUdZeZeed<dZdS) OriginPolicya> https://www.w3.org/TR/referrer-policy/#referrer-policy-origin The "origin" policy specifies that only the ASCII serialization of the origin of the request client is sent as referrer information when making both same-origin requests and cross-origin requests from a particular request client. rc,||Sr)r'rs rrzOriginPolicy.referrers##L111r N) r6r7r8r@ POLICY_ORIGINrr:r;rr<r rrLrLsAD#22222r rLc(eZdZUdZeZeed<dZdS)StrictOriginPolicya0 https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin The "strict-origin" policy sends the ASCII serialization of the origin of the request client when making requests: - from a TLS-protected environment settings object to a potentially trustworthy URL, and - from non-TLS-protected environment settings objects to any origin. Requests from TLS-protected request clients to non- potentially trustworthy URLs, on the other hand, will contain no referrer information. A Referer HTTP header will not be sent. rc||r||s||s||SdSr)r/r1r'rs rrzStrictOriginPolicy.referrersf   | , , 6,,[99 6%%l33 6 '' 55 5  6 6r N) r6r7r8r@POLICY_STRICT_ORIGINrr:r;rr<r rrPrPsA  %D#$$$66666r rPc(eZdZUdZeZeed<dZdS)OriginWhenCrossOriginPolicya https://www.w3.org/TR/referrer-policy/#referrer-policy-origin-when-cross-origin The "origin-when-cross-origin" policy specifies that a full URL, stripped for use as a referrer, is sent as referrer information when making same-origin requests from a particular request client, and only the ASCII serialization of the origin of the request client is sent as referrer information when making cross-origin requests from a particular request client. rc||}|||kr||S|SrrIrrrrs rrz$OriginWhenCrossOriginPolicy.referrersE\** T[[-- - 8)),77 7 r N) r6r7r8r@POLICY_ORIGIN_WHEN_CROSS_ORIGINrr:r;rr<r rrTrTsA  0D#///r rTc(eZdZUdZeZeed<dZdS)!StrictOriginWhenCrossOriginPolicya https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin-when-cross-origin The "strict-origin-when-cross-origin" policy specifies that a full URL, stripped for use as a referrer, is sent as referrer information when making same-origin requests from a particular request client, and only the ASCII serialization of the origin of the request client when making cross-origin requests: - from a TLS-protected environment settings object to a potentially trustworthy URL, and - from non-TLS-protected environment settings objects to any origin. Requests from TLS-protected clients to non- potentially trustworthy URLs, on the other hand, will contain no referrer information. A Referer HTTP header will not be sent. rc4||}|||kr||S||r||s||s||SdSr)rr%r/r1r'rVs rrz*StrictOriginWhenCrossOriginPolicy.referrers\** T[[-- - 8)),77 7   | , , 6,,[99 6%%l33 6 '' 55 5  6 6r N) r6r7r8r@&POLICY_STRICT_ORIGIN_WHEN_CROSS_ORIGINrr:r;rr<r rrYrYsA"7D#666 6 6 6 6 6r rYc(eZdZUdZeZeed<dZdS)UnsafeUrlPolicya https://www.w3.org/TR/referrer-policy/#referrer-policy-unsafe-url The "unsafe-url" policy specifies that a full URL, stripped for use as a referrer, is sent along with both cross-origin requests and same-origin requests made from a particular request client. Note: The policy's name doesn't lie; it is unsafe. This policy will leak origins and paths from TLS-protected resources to insecure origins. Carefully consider the impact of setting such a policy for potentially sensitive documents. rc,||Sr)r%rs rrzUnsafeUrlPolicy.referrers%%l333r N) r6r7r8r@POLICY_UNSAFE_URLrr:r;rr<r rr]r]sA  "D#!!!44444r r]cFeZdZUdZedzZeedfed<e Z eed<dS)DefaultReferrerPolicyz A variant of "no-referrer-when-downgrade", with the addition that "Referer" is not sent if the parent request was using ``file://`` or ``s3://`` scheme. )files3.rrN) r6r7r8r@r9rrr:r;POLICY_SCRAPY_DEFAULTrr<r rrarasL +8.*Hc3hHHH%D#%%%%%r raci|] }|j| Sr<)r).0ps r rhs,    FA   r Fc t|S#t$re t|cYS#t$r5d|}|st |t j|tYYdSwxYwwxYw)z Expect a string for the path to the policy class, otherwise try to interpret the string as a standard value from https://www.w3.org/TR/referrer-policy/#referrer-policies zCould not load referrer policy N) r ValueError_policy_classeslowerKeyError RuntimeErrorwarningswarnRuntimeWarning)policy warning_onlymsgs r_load_policy_classrv"s 6"""     "6<<>>2 2 2 2   >F>>C "3''' c>222ttt   s) B=B:A<7B;A<<BcJeZdZd dZedZdZdZdZdZ dZ dS) RefererMiddlewareNctt|_|)t|d|_dSdS)NREFERRER_POLICY)radefault_policyrvget)rsettingss r__init__zRefererMiddleware.__init__7sB3  V"4X\\BS5T5T"U"UD    V Vr c|jdst||j}|j|jtj|S)NREFERER_ENABLED)signal)r}getboolrrconnectrequest_scheduled)clscrawlermws r from_crawlerzRefererMiddleware.from_crawler<s\''(9::   S! " "  4W=VWWW r cn|jd}|St|tr>|jd}|"t |d}||St|d}|r |n|S)a Determine Referrer-Policy to use from a parent Response (or URL), and a Request to be sent. - if a valid policy is set in Request meta, it is used. - if the policy is set in meta but is wrong (e.g. a typo error), the policy from settings is used - if the policy is not set in Request meta, but there is a Referrer-policy header in the parent response, it is used if valid - otherwise, the policy from settings is used. referrer_policyNzReferrer-Policylatin1T)rt) metar| isinstancer headersr decoder{rv)r resp_or_urlrequest policy_name policy_headerrs rrszRefererMiddleware.policyGsl&&'899  M+x00 M + 3 7 78I J J  M",]-A-A(-K-K"L"LK  )&&(( ( 4@@@6ssuuu!4!4!6!66r c(fd|pdDS)Nc3DK|]}|VdSr _set_referer)rfrresponsers r z:RefererMiddleware.process_spider_output..as3EE1!!!X..EEEEEEr r<r<)rrresultspiders`` rprocess_spider_outputz'RefererMiddleware.process_spider_output`s%EEEEE "EEEEr cVK|pd23d{V}|||WV!6dS)Nr<r)rrrrrs rprocess_spider_output_asyncz-RefererMiddleware.process_spider_output_asynccs]| 1 1 1 1 1 1 1!##Ax00 0 0 0 0 0$||s(ct|trQ||||j|j}||jd||S)NReferer)rrrsrr$r setdefault)rrrrs rrzRefererMiddleware._set_referergs` a ! ! :{{8Q//88quMMH : $$Y999r c\|jdg}|r|jd}|pt|}|||||j}||kr.||jddS||jd<dSdSdSdS)N redirect_urlsr)rr|rrrsrr$pop)rrrredirected_urlsrequest_referrer parent_urlpolicy_referrers rrz#RefererMiddleware.request_scheduledns!,**?B??  E&229==  E --=>> "&++j'"B"B"K"K ###&66E&E++I666665D 222# E E E EEEr r) r6r7r8r~ classmethodrrsrrrrr<r rrxrx6sVVVV [7772FFF111EEEEEr rxr5),r@rptypingr urllib.parser w3lib.urlrscrapyrscrapy.exceptionsr scrapy.httprr scrapy.utils.miscr scrapy.utils.pythonr scrapy.utils.urlr r9rArErJrNrRrWr[r_rdrr>rCrGrLrPrTrYr]rarlrvrxr<r rrsH!!!!!!%%%%%%++++++))))))))))))))******&&&&&& #$@!" &"<)J& (4949494949494949n     ~   88888N888,88888~888$ 2 2 2 2 2> 2 2 2 666666662.*66666666@44444n444(&&&&&9&&&   %#)     4(LELELELELELELELELELEr