3d5 UdZddlZddlZddlZddlmZmZmZmZm Z m Z ddl m Z ddl mZddlmZddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZmZd ed<eZ dZ! d'dedeee e"e#fde$de#fdZ%dZ&ded<eZ'ddddedeee e"e#fde$de"fdZ(GddZ)dede#de#ddfdZ*dede"fd Z+dedee#fd!Z,dd"d#e-d$eedefd%Z.d&Z/dS)(zY This module provides some useful functions for working with scrapy.http.Request objects N)DictIterableListOptionalTupleUnion) urlunparse)WeakKeyDictionary)basic_auth_header)canonicalize_url)RequestSpider)ScrapyDeprecationWarning)urlparse_cached) load_object)to_bytes to_unicodezOWeakKeyDictionary[Request, Dict[Tuple[Optional[Tuple[bytes, ...]], bool], str]]_deprecated_fingerprint_cachec#pK|D]0}||jvr%|V|j|D]}|V1dSN)headersgetlist)rrequestheadervalues 4lib/python3.11/site-packages/scrapy/utils/request.py_serialize_headersrsb W_ $ LLL 0088   Frinclude_headerskeep_fragmentsreturnc|s|rd}nd}tj|tdd}|r&tdt |D}t |i}||f}||vrtj}| t|j | tt|j || |jpd|r(t||D]}| ||||<||S) a/ Return the request fingerprint as an hexadecimal string. The request fingerprint is a hash that uniquely identifies the resource the request points to. For example, take the following two urls: http://www.example.com/query?id=111&cat=222 http://www.example.com/query?cat=222&id=111 Even though those are two different URLs both point to the same resource and are equivalent (i.e. they should return the same response). Another example are cookies used to store session ids. Suppose the following page is only accessible to authenticated users: http://www.example.com/members/offers.html Lots of sites use a cookie to store the session id, which adds a random component to the HTTP Request and thus should be ignored when calculating the fingerprint. For this reason, request headers are ignored by default when calculating the fingerprint. If you want to include specific headers use the include_headers argument, which is a list of Request headers to include. Also, servers usually ignore fragments in urls when handling requests, so they are also ignored by default when calculating the fingerprint. If you want to include them, set the keep_fragments argument to True (for instance when handling requests with a headless browser). aCall to deprecated function scrapy.utils.request.request_fingerprint(). If you are using this function in a Scrapy component because you need a non-default fingerprinting algorithm, and you are OK with that non-default fingerprinting algorithm being used by all Scrapy components and not just the one calling this function, use crawler.request_fingerprinter.fingerprint() instead in your Scrapy component (you can get the crawler object from the 'from_crawler' class method), and use the 'REQUEST_FINGERPRINTER_CLASS' setting to configure your non-default fingerprinting algorithm. Otherwise, consider using the scrapy.utils.request.fingerprint() function instead. If you switch to 'fingerprint()', or assign the 'REQUEST_FINGERPRINTER_CLASS' setting a class that uses 'fingerprint()', the generated fingerprints will not only be bytes instead of a string, but they will also be different from those generated by 'request_fingerprint()'. Before you switch, make sure that you understand the consequences of this (e.g. cache invalidation) and are OK with them; otherwise, consider implementing your own function which returns the same fingerprints as the deprecated 'request_fingerprint()' function.axCall to deprecated function scrapy.utils.request.request_fingerprint(). If you are using this function in a Scrapy component, and you are OK with users of your component changing the fingerprinting algorithm through settings, use crawler.request_fingerprinter.fingerprint() instead in your Scrapy component (you can get the crawler object from the 'from_crawler' class method). Otherwise, consider using the scrapy.utils.request.fingerprint() function instead. Either way, the resulting fingerprints will be returned as bytes, not as a string, and they will also be different from those generated by 'request_fingerprint()'. Before you switch, make sure that you understand the consequences of this (e.g. cache invalidation) and are OK with them; otherwise, consider implementing your own function which returns the same fingerprints as the deprecated 'request_fingerprint()' function.category stacklevelNc3XK|]%}t|V&dSrrlower.0hs r z&request_fingerprint..{C* * $%HQWWYY  * * * * * * rr r)warningswarnrtuplesortedr setdefaulthashlibsha1updatermethodr urlbodyr hexdigest) rrr messageprocessed_include_headerscache cache_keyfpparts rrequest_fingerprintrB"syF2 .2  O : O , M'$<KKKK=A $)* * )/)@)@* * * % % ! * 4 4Wb A AE*N;I * \^^ (7>**+++ %gk.QQQ R R    ',%#&&& $ *+DgNN   $<<>>i  rctj5tjdtt |i|cdddS#1swxYwYdS)Nignore)r0catch_warnings simplefilterbytesfromhexrB)argskwargss r_request_fingerprint_as_bytesrKs  " "CCh'''}}0$A&AABBCCCCCCCCCCCCCCCCCCs8AA AzQWeakKeyDictionary[Request, Dict[Tuple[Optional[Tuple[bytes, ...]], bool], bytes]]_fingerprint_cache)rr cpd}|r&tdt|D}t|i}||f}||vri}|rG|D]D}||jvr9d|j|D||<Et|jt|j ||j pd|d}tj |d} tj| ||<||S) a Return the request fingerprint. The request fingerprint is a hash that uniquely identifies the resource the request points to. For example, take the following two urls: http://www.example.com/query?id=111&cat=222 http://www.example.com/query?cat=222&id=111 Even though those are two different URLs both point to the same resource and are equivalent (i.e. they should return the same response). Another example are cookies used to store session ids. Suppose the following page is only accessible to authenticated users: http://www.example.com/members/offers.html Lots of sites use a cookie to store the session id, which adds a random component to the HTTP Request and thus should be ignored when calculating the fingerprint. For this reason, request headers are ignored by default when calculating the fingerprint. If you want to include specific headers use the include_headers argument, which is a list of Request headers to include. Also, servers usually ignore fragments in urls when handling requests, so they are also ignored by default when calculating the fingerprint. If you want to include them, set the keep_fragments argument to True (for instance when handling requests with a headless browser). Nc3XK|]%}t|V&dSrr(r*s rr-zfingerprint..r.rc6g|]}|S)hex)r+ header_values r zfingerprint..s4---(%((**---rr/r)r8r9r:rT) sort_keys)r2r3rLr4rrrQrr8r r9r:jsondumpsr5r6encodedigest) rrr r=r>r?rrfingerprint_datafingerprint_jsons r fingerprintr[swH>B $)* * )/)@)@* * * % % !  ) )'2 6 6E*N;IL)+ $ 3  W_,--,3O,C,CF,K,K---GFJJLL) !00#GKOOO\(S--//     :&6$GGG"<(8(?(?(A(ABBIIKKi  rc<eZdZdZedZddZdefdZdS)RequestFingerprinteraDefault fingerprinter. It takes into account a canonical version (:func:`w3lib.url.canonicalize_url`) of :attr:`request.url ` and the values of :attr:`request.method ` and :attr:`request.body `. It then generates an `SHA1 `_ hash. .. seealso:: :setting:`REQUEST_FINGERPRINTER_IMPLEMENTATION`. c||SrrP)clscrawlers r from_crawlerz!RequestFingerprinter.from_crawlerss7||rNc|r|jd}nd}|dkr,d}tj|tdt |_dS|dkrt|_dStd|d) N$REQUEST_FINGERPRINTER_IMPLEMENTATIONz2.6a'2.6' is a deprecated value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting. It is also the default value. In other words, it is normal to get this warning if you have not defined a value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting. This is so for backward compatibility reasons, but it will change in a future version of Scrapy. See the documentation of the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting for information on how to handle this deprecation.r#r$z2.7zHGot an invalid value on setting 'REQUEST_FINGERPRINTER_IMPLEMENTATION': z0. Valid values are '2.6' (deprecated) and '2.7'.) settingsgetr0r1rrK _fingerprintr[ ValueError)selfr`implementationr<s r__init__zRequestFingerprinter.__init__s  #$-116NN#N U "  A  M',DQR S S S S =D    u $  +D   ! rrc,||Sr)rf)rhrs rr[z RequestFingerprinter.fingerprint s  )))rr) __name__ __module__ __qualname____doc__ classmethodrarjr r[rPrrr]r]si  [    D*7******rr]usernamepasswordc6t|||jd<dS)zAuthenticate the given request (in place) using the HTTP basic access authentication mechanism (RFC 2617) and the given username and password AuthorizationN)r r)rrqrrs rrequest_authenticaterus (98'L'LGOO$$$rclt|}tdd|jpd|j|jdf}t |jdzt |zdz}|dt |jpdzdzz }|jr||j dzz }|dz }||j z }|S)zReturn the raw HTTP representation (as bytes) of the given request. This is provided only for reference since it's not the actual stream of bytes that will be send when performing the request (that's controlled by Twisted). / s HTTP/1.1 sHost: rs ) rr pathparamsqueryrr8hostnamer to_stringr:)rparsedrzss rrequest_httpreprrs W % %F r2v{1c6=&,PRS T TD  4'(4..8;KKAXfo455 5 ??A3 W_ & & ( (7 22LAA Hrc`|jd}||St|dS)z0Return Referer HTTP header suitable for logging.RefererNreplace)errors)rrer)rreferrers r referer_strr,s7""9--H hy 1 1 11r)spiderdrcTd|vrt|dntfd|D}|dr|rt ||d|d<|dr|rt ||d|d<di|S)zCreate a :class:`~scrapy.Request` object from a dict. If a spider is given, it will try to resolve the callbacks looking at the spider for methods with the same name. _classc.i|]\}}|jv||SrP) attributes)r+keyr request_clss r z%request_from_dict..;s, V V VZS%{?U8U Vc5 V V VrcallbackerrbackrP)rr itemsre _get_method)rrrJrs @rrequest_from_dictr4s /7!mH+ak***K V V V V17799 V V VFuuZ@V@(:??zuuY>F>') ==y ;   rct|} t||S#t$rtd|d|wxYw)z%Helper function for request_from_dictzMethod z not found in: )strgetattrAttributeErrorrg)objnames rrrCsa t99DAsD!!! AAA?4??#??@@@As ! A)NF)0ror5rUr0typingrrrrrr urllib.parser weakrefr w3lib.httpr w3lib.urlr scrapyr rscrapy.exceptionsrscrapy.utils.httpobjrscrapy.utils.miscrscrapy.utils.pythonrr__annotations__rrrGrboolrBrKrLr[r]rurrdictrrrPrrrs:  ????????????????######%%%%%%((((((&&&&&&""""""""666666000000))))))44444444pppp 1 1 3 3>B ii ihuUCZ'89:ii iiiiXCCC hggg&&(( >B >>> >huUCZ'89:> >  >>>>B4*4*4*4*4*4*4*4*nM MMM MMMM g %    "22Xc]2222>B ! ! ! !(6*: !g ! ! ! !AAAAAr