3d=$.dZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZeeeefZd Z dZ!dZ"ddZ#dZ$dZ%dZ&e dZ'dZ(edZ)dZ*dZ+dS)z.Helper functions which don't fit anywhere elseN)deque)contextmanager)partial) import_module) iter_modulesreplace_entities)Item)LocalWeakReferencedCache)ScrapyDeprecationWarning)flatten to_unicodec^|gSt|tst|dr|S|gS)zConvert an argument to an iterable. The argument can be a None, single value, or an iterable. Exception: if arg is a dict, [arg] will be returned N__iter__) isinstance_ITERABLE_SINGLE_VALUEShasattr)args 1lib/python3.11/site-packages/scrapy/utils/misc.py arg_to_iterrsA   c2 3 3Z8P8P 5Lct|ts0t|r|Stdt | |d}n!#t $rt d|dwxYw|d|||dzd}}t|} t||}n$#t$rtd|d|d wxYw|S) aRLoad an object given its absolute object path, and return it. The object can be the import path of a class, function, variable or an instance, e.g. 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware'. If ``path`` is not a string, but is a callable object, such as a class or a function, then return it as is. z:Unexpected argument type, expected string or object, got: .zError loading object 'z': not a full pathNzModule 'z#' doesn't define any object named '') rstrcallable TypeErrortyperindex ValueErrorrgetattrAttributeError NameError)pathdotmodulenamemodobjs r load_objectr+%s* dC  D>> K XDQUJJ X X   Lkk# LLLJ$JJJKKKL:tC!GIIDF   CWc4   WWWU6UUdUUUVVVW JsAA;%B66!Cc*g}t|}||t|dr\t|jD]G\}}}|dz|z}|r|t |z }#t|}||H|S)zLoads a module and all its submodules from the given module path and returns them. If *any* module throws an exception while importing, that exception is thrown back. For example: walk_modules('scrapy.utils') __path__r)rappendrrr- walk_modules)r%modsr)_subpathispkgfullpathsubmods rr/r/Fs D   CKKsJ$!-cl!;!; $ $ AwczG+H $ X...&x00 F#### Krutf-8ctjdtdt|trt j|t j} || dg}n%#t$r| |}YnwxYwt|}t|tr d|DSfd|DS)acExtract a list of unicode strings from the given text/encoding using the following policies: * if the regex contains a named group called "extract" that will be returned * if the regex contains multiple numbered groups, all those will be returned (flattened) * if the regex doesn't contain any group the entire regex matching is returned zHscrapy.utils.misc.extract_regex has moved to parsel.utils.extract_regex. stacklevelextractc4g|]}t|ddgSltamp)keepr).0ss r z!extract_regex..ss*IIIA $777IIIrcRg|]#}tt|ddg$Sr=)r r)rArBencodings rrCz!extract_regex..tsB   JKAx00e}EEE   r) warningswarnr rrrecompileUNICODEsearchgroup Exceptionfindallr )regextextrEstringss ` r extract_regexrR\s MR  %. 5"*--&<<%%++I667 &&&--%%&gG$JIIIIII    OV   s)A==BBctj} |d}|sn||.|S)zCalculate the md5 checksum of a file-like object without reading its whole content in memory. >>> from io import BytesIO >>> md5sum(BytesIO(b'file content to hash')) '784406af91dd5a54fbb9c84c2236595a' Ti)hashlibmd5readupdate hexdigest)filemds rmd5sumr\ysR  A IIdOO     ;;==rc^|duo)d|ddvS)z3Return True if link rel attribute has nofollow typeNnofollow, )replacesplit)rels rrel_has_nofollowrds0 d? JzS[[c-B-B-H-H-J-JJJrc||td|j}|r$t|dr|j|g|Ri|}d}n.t|dr|j|g|Ri|}d}n ||i|}d}|t |jd|d|S)aConstruct a class instance using its ``from_crawler`` or ``from_settings`` constructors, if available. At least one of ``settings`` and ``crawler`` needs to be different from ``None``. If ``settings `` is ``None``, ``crawler.settings`` will be used. If ``crawler`` is ``None``, only the ``from_settings`` constructor will be tried. ``*args`` and ``**kwargs`` are forwarded to the constructors. Raises ``ValueError`` if both ``settings`` and ``crawler`` are ``None``. .. versionchanged:: 2.2 Raises ``TypeError`` if the resulting instance is ``None`` (e.g. if an extension has not been implemented correctly). Nz-Specify at least one of settings and crawler. from_crawler from_settings__new__rz returned None)r!settingsrrfrgr __qualname__)objclsricrawlerargskwargsinstance method_names rcreate_instancerqs"$  NLMM M# 76>22 &6&w@@@@@@$  ) ) '6'B4BBB6BB% 64*6** N6.LLLLLMMM Orc+XKd|D}tj| dV|D]$\}}|tj|=|tj|<%dS#|D]$\}}|tj|=|tj|<%wxYw)z{Temporarily set environment variables inside the context manager and fully restore previous environment afterwards cNi|]"}|tj|#S)osenvironget)rAks r zset_environ..s(999QArz~~a((999rN)rurvrWitems)rn original_envrxvs r set_environr}s :9&999LJf"  &&(( " "DAq "JqMM ! 1  " "L&&(( " "DAq "JqMM ! 1  "s A..;B)c#Kt|g}d}|rb|}t|tjr|r3d}|tj||V|`dSdS)zrSimilar to ``ast.walk``, but walks only function body and skips nested functions defined within the node. FTN)rpopleftrast FunctionDefextenditer_child_nodes)nodetodowalked_func_defs r walk_callablers $==DO ||~~ dCO , , # "O C(..///  r)limitc|tvr t|Sd}tj|r|}t|tr|j}t|ttj|}tjd}| d|}| |}|r,tj d| dd|}tj |}t|D]@}t|tjr$||sdt|<t|cSAdt|<t|S)z Returns True if a callable is a generator function which includes a 'return' statement with a value different than None, False otherwise c^|j}|dup"t|tjo|jduS)N)valuerr NameConstant) return_noders r returns_nonez4is_generator_with_return_value..returns_nones5! TM XZs/?@@XU[TXEX rz (^[\t ]+) rTF)_generator_callbacks_cacheinspectisgeneratorfunctionrrfunc getsourcerHrIsubmatchrLrparserReturn) rrrsrcpatterncodertreers ris_generator_with_return_valuersb --4)(33   "8,,<w'' 9Dw'' %%*\**{{2s## c""  =6/u{{1~~//t<rs44  %%%%%%###### '''''';;;;;;;;;;;;33333333c50   B,:"KKK   F""""    65C@@@"0"0"0J     r