3d'6BdZddlZddlZddlmZddlmZddlmZmZm Z m Z m Z m Z m Z mZmZmZmZddlmZddlmZmZmZddlmZdd lmZdd lmZdd lmZdd l m!Z!m"Z"d edefdZ#defdZ$defdZ%de defdZ&de de'de defdZ(GddeZ)de de'de defdZ*de e defdZ+de e de e defdZ,de e defd Z-de d!e de fd"Z.d#e d!e defd$Z/defd%Z0d&e d'e fde fd(Z1de defd)Z2d*edefd+Z3d*edeeeffd,Z4dS)-z5 Helper functions for dealing with Twisted deferreds N)Futurewraps) AnyAsyncGenerator AsyncIterableCallable Coroutine GeneratorIterableIteratorListOptionalUnion)defer)Deferred DeferredListensureDeferred) Cooperator)failure)Failure) IgnoreRequest)_get_asyncio_event_loopis_asyncio_reactor_installed_failurereturncfddlm}t}|d|j||S)zSame as twisted.internet.defer.fail but delay calling errback until next reactor loop It delays by 100ms so reactor has a chance to go through readers and writers before attending pending delayed calls, so do not set delay to zero. rreactor皙?)twisted.internetrr callLatererrback)rrds 2lib/python3.11/site-packages/scrapy/utils/defer.py defer_failr& s>)((((( A c19h/// Hcfddlm}t}|d|j||S)zSame as twisted.internet.defer.succeed but delay calling callback until next reactor loop It delays by 100ms so reactor has a chance to go through readers and writers before attending pending delayed calls, so do not set delay to zero. rrr )r!rrr"callback)resultrr$s r% defer_succeedr+.s>)((((( A c1:v... Hr'ct|tr|St|tjrt |St |SN) isinstancerrrr&r+)r*s r% defer_resultr/<sI&(## &'/**"&!!!   r'fc ||i|}t|S#t$r+}ttj|cYd}~Sd}~wt $r#ttjcYSwxYw)znSame as twisted.internet.defer.maybeDeferred, but delay calling callback/errback to next reactor loop N)r/rr&rr Exception)r0argskwr*es r%mustbe_deferredr6Ds $DBF### ...'/!,,-------- ---'/++,,,,,-s  A9 A A9 -A98A9iterablecountcallablectfd|Dtfdt|DS)zExecute a callable over the objects in the given iterable, in parallel, using no more than ``count`` concurrent calls. Taken from: https://jcalderone.livejournal.com/24285.html c30K|]}|gRiVdSr-).0elemr3r9nameds r% zparallel..^s= @ @tHHT *D * * *E * * @ @ @ @ @ @r'c:g|]}Sr< coiterater=_coopworks r% zparallel.._s%DDD!--DDDr')rrrange)r7r8r9r3r?rFrGs ```@@r%parallelrJUs] <RSS  " & &q ) ) fh ' '    # # # # JJt     !           r'rcd|_|jtd|_|jD]}|ddS)NT)rUtrapStopAsyncIterationrSrTr))rVrr$s r%_errbackz _AsyncCooperatorAdapter._errbacksW# '((( '  A JJt      r'ct|j|_|j|j|jdSr-)deferred_from_cororP __anext__rU addCallbacksr\r`)rVs r%r[z#_AsyncCooperatorAdapter._call_anextsE11I1I1K1KLL ((GGGGGr'c|jrtt}|j||js||Sr-)rS StopIterationrrTappendrUr[)rVr$s r%__next__z _AsyncCooperatorAdapter.__next__sV =   JJ %%a((("       r')rN)__name__ __module__ __qualname____doc__rr rWrr\rr`r[rrhr<r'r%rLrLbs++Z 7  7 7 7 7 7      DHHHH (      r'rLasync_iterablectt||g|Ri|tfdt|D}|S)z%Like parallel but for async iteratorsc:g|]}Sr<rBrDs r%rHz"parallel_async..s%BBBt~~d++BBBr')rrLrrI)rmr8r9r3r?dlrFrGs @@r%parallel_asyncrqs^ <8 Ld L L Le L LD BBBBBU5\\BBB C CB Ir' callbackscxt}|D]}|j|g|Ri||||S)z7Return a Deferred built by chaining the given callbacks)r addCallbackr))rrinputar4r$xs r% process_chainrxsV A ## a"!"""r""""JJu Hr'errbacksc t}t||D] \}}|||||||!t|tjr||n|||S)zDReturn a Deferred built by chaining the given callbacks and errbacks)r)r# callbackArgscallbackKeywords errbackArgserrbackKeywords)rziprdr.rrr#r))rrryrurvr4r$cbebs r%process_chain_bothrs  Ai**  B      %)) % 5 Hr'cfd|D}t|dd}|dd|S)zYReturn a Deferred with the output of all successful calls to the given callbacks cVg|]%}tjj|gRi&Sr<)rsucceedrt)r=rwrvrur4s r%rHz$process_parallel..sA M M Ma ,EM% ,Q 9 9 9 9b 9 9 M M Mr'T)fireOnOneErrback consumeErrorscd|DS)Ncg|] }|d S)r<)r=rws r%rHz6process_parallel....s...qad...r'r<)rs r%z"process_parallel..s..A...r'c|jjSr-)value subFailure)r0s r%rz"process_parallel..s !':Lr')rrd)rrrurvr4dfdsr$s ``` r%process_parallelrs\ N M M M M M9 M M MDTDEEEANN..0L0LMMM Hr'r#c/Kt|} t|Vn9#t$rYdSt$r |t jg|Ri|YnwxYwM)zWWraps an iterable calling an errback if an error is caught while iterating it. TN)iternextrfr2rr)r7r#rvr4its r% iter_errbackrs hB1 1r((NNNN    EE 1 1 1 GGO%% 0 0 0 0R 0 0 0 0 0 1 1s& A&AArMcK|} |d{VWVn9#t$rYdSt$r |t jg|Ri|YnwxYwY)zWraps an async iterable calling an errback if an error is caught while iterating it. Similar to scrapy.utils.defer.iter_errback() TN)rOrcr_r2rr)rMr#rvr4rs r% aiter_errbackrs     B1 1 &&&&&& & & & & &!    EE 1 1 1 GGO%% 0 0 0 0R 0 0 0 0 0 1 1s7 A-&A-,A-c*t|tr|Stj|st j|rSt st|St}tj tj ||S|S)zYConverts a coroutine into a Deferred, or returns the object as is if it isn't a coroutine)loop) r.rasyncioisfutureinspect isawaitablerrr fromFuture ensure_future)o event_loops r%rbrbs!XNg1!44N+-- %"!$$ $,.. "7#8#L#L#LMMM Hr'coro_f.c<tfd}|S)a-Converts a coroutine function into a function that returns a Deferred. The coroutine function will be called at the time when the wrapper is called. Wrapper args will be passed to it. This is useful for callback chains, as callback functions are called with the previous callback result. c.t|i|Sr-)rb) coro_args coro_kwargsrs r%r0z!deferred_f_from_coro_f..f.s !&&)"C{"C"CDDDr'r)rr0s` r%deferred_f_from_coro_fr's: 6]]EEEE]E Hr'c ||i|}n8#tjtjtjcYSxYwt |tr|Stj|stj |rt|St |tjrtj|Stj |S)zGCopy of defer.maybeDeferred that also converts coroutines to Deferreds.) captureVars) rfailrrrdebugr.rrrrrbr)r0r3r4r*s r%maybeDeferred_coror5sGDBGz'/hnEEEFFFFF&(## *7#6v#>#>*!&)))&'/**"z&!!! =  s  3Ar$cD|tS)a .. versionadded:: 2.6.0 Return an :class:`asyncio.Future` object that wraps *d*. When :ref:`using the asyncio reactor `, you cannot await on :class:`~twisted.internet.defer.Deferred` objects from :ref:`Scrapy callables defined as coroutines `, you can only await on ``Future`` objects. Wrapping ``Deferred`` objects into ``Future`` objects allows you to wait on them:: class MySpider(Spider): ... async def parse(self, response): d = treq.get('https://example.com/additional') additional_response = await deferred_to_future(d) )asFuturerr$s r%deferred_to_futurerEs$ ::-// 0 00r'c@ts|St|S)a .. versionadded:: 2.6.0 Return *d* as an object that can be awaited from a :ref:`Scrapy callable defined as a coroutine `. What you can await in Scrapy callables defined as coroutines depends on the value of :setting:`TWISTED_REACTOR`: - When not using the asyncio reactor, you can only await on :class:`~twisted.internet.defer.Deferred` objects. - When :ref:`using the asyncio reactor `, you can only await on :class:`asyncio.Future` objects. If you want to write code that uses ``Deferred`` objects but works with any reactor, use this function on all ``Deferred`` objects:: class MySpider(Spider): ... async def parse(self, response): d = treq.get('https://example.com/additional') extra_response = await maybe_deferred_to_future(d) )rrrs r%maybe_deferred_to_futurerZs%2 ( ) ) a  r')5rlrrr functoolsrtypingrrrr r r r r rrrr!rtwisted.internet.deferrrrtwisted.internet.taskrtwisted.pythonrtwisted.python.failurerscrapy.exceptionsrscrapy.utils.reactorrrr&r+r/r6intrJrLrqrxrrrrrbrrrrr<r'r%rs                          #"""""IIIIIIIIII,,,,,,""""""******++++++VVVVVVVV    X       X     !H!!!!$x$$$$$" F F" F.6 F F F F FccccchcccL!*-9A Xh/ X     ! -5h-?      *  2      18 1h 1Y 1 1 1 1 1 1'/ 1 1 1 1 1  S       8CN#;        !( !H ! ! ! ! 1(1v1111*!!U8V3C-D!!!!!!r'