3d6ddlZddlZddlmZddlmZddlmZmZm Z ddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZmZejeZGd d eZGd deZe ddZGddeZdS)N)abstractmethod)Path)OptionalTypeTypeVar)Deferred)Crawler)Request)Spider)job_dir)create_instance load_objectceZdZdZdZdZdS)BaseSchedulerMetazN Metaclass to check scheduler classes against the necessary interface cF|t|SN)__subclasscheck__type)clsinstances 5lib/python3.11/site-packages/scrapy/core/scheduler.py__instancecheck__z#BaseSchedulerMeta.__instancecheck__s$$T(^^444ct|do[t|joGt|do7t|jo#t|dot|jS)Nhas_pending_requestsenqueue_request next_request)hasattrcallablerrr)rsubclasss rrz#BaseSchedulerMeta.__subclasscheck__sy H4 5 5 0677 0"344 0122 0.11  0 .//  rN)__name__ __module__ __qualname____doc__rrrrrrs<555     rrceZdZdZedefdZdedee fdZ de dee fdZ e defd Ze d edefd Ze deefd Zd S) BaseSchedulera7 The scheduler component is responsible for storing requests received from the engine, and feeding them back upon request (also to the engine). The original sources of said requests are: * Spider: ``start_requests`` method, requests created for URLs in the ``start_urls`` attribute, request callbacks * Spider middleware: ``process_spider_output`` and ``process_spider_exception`` methods * Downloader middleware: ``process_request``, ``process_response`` and ``process_exception`` methods The order in which the scheduler returns its stored requests (via the ``next_request`` method) plays a great part in determining the order in which those requests are downloaded. The methods defined in this class constitute the minimal interface that the Scrapy engine will interact with. crawlerc|S)zp Factory method which receives the current :class:`~scrapy.crawler.Crawler` object as argument. r%)rr(s r from_crawlerzBaseScheduler.from_crawler6s suu rspiderreturncdS)a Called when the spider is opened by the engine. It receives the spider instance as argument and it's useful to execute initialization code. :param spider: the spider object for the current crawl :type spider: :class:`~scrapy.spiders.Spider` Nr%selfr+s ropenzBaseScheduler.open=  rreasoncdS)a& Called when the spider is closed by the engine. It receives the reason why the crawl finished as argument and it's useful to execute cleaning code. :param reason: a string which describes the reason why the spider was closed :type reason: :class:`str` Nr%)r/r2s rclosezBaseScheduler.closeGr1rct)zV ``True`` if the scheduler has enqueued requests, ``False`` otherwise NotImplementedErrorr/s rrz"BaseScheduler.has_pending_requestsQs "###rrequestct)a Process a request received by the engine. Return ``True`` if the request is stored correctly, ``False`` otherwise. If ``False``, the engine will fire a ``request_dropped`` signal, and will not make further attempts to schedule the request at a later time. For reference, the default Scrapy scheduler returns ``False`` when the request is rejected by the dupefilter. r6r/r9s rrzBaseScheduler.enqueue_requestXs"###rct)a Return the next :class:`~scrapy.http.Request` to be processed, or ``None`` to indicate that there are no requests to be considered ready at the moment. Returning ``None`` implies that no request from the scheduler will be sent to the downloader in the current reactor cycle. The engine will continue calling ``next_request`` until ``has_pending_requests`` is ``False``. r6r8s rrzBaseScheduler.next_requestfs"###rN)r!r"r#r$ classmethodr r*r rrr0strr4rboolrr rrr%rrr'r'%s 7[  6 hx&8     C HX$6    $d$$$^$  $w $4 $ $ $^ $ $hw/ $ $ $^ $ $ $rr') metaclass SchedulerTV Scheduler)boundceZdZdZ ddeededeefdZe de e d e fd Z d efd Z d ed eefd Zded eefdZded efdZd eefdZd efdZded efdZded dfdZd eefdZdZdZdeed eefdZded efdZdeded dfdZdS)rBa Default Scrapy scheduler. This implementation also handles duplication filtering via the :setting:`dupefilter `. This scheduler stores requests into several priority queues (defined by the :setting:`SCHEDULER_PRIORITY_QUEUE` setting). In turn, said priority queues are backed by either memory or disk based queues (respectively defined by the :setting:`SCHEDULER_MEMORY_QUEUE` and :setting:`SCHEDULER_DISK_QUEUE` settings). Request prioritization is almost entirely delegated to the priority queue. The only prioritization performed by this scheduler is using the disk-based queue if present (i.e. if the :setting:`JOBDIR` setting is defined) and falling back to the memory-based queue if a serialization error occurs. If the disk queue is not present, the memory one is used directly. :param dupefilter: An object responsible for checking and filtering duplicate requests. The value for the :setting:`DUPEFILTER_CLASS` setting is used by default. :type dupefilter: :class:`scrapy.dupefilters.BaseDupeFilter` instance or similar: any class that implements the `BaseDupeFilter` interface :param jobdir: The path of a directory to be used for persisting the crawl's state. The value for the :setting:`JOBDIR` setting is used by default. See :ref:`topics-jobs`. :type jobdir: :class:`str` or ``None`` :param dqclass: A class to be used as persistent request queue. The value for the :setting:`SCHEDULER_DISK_QUEUE` setting is used by default. :type dqclass: class :param mqclass: A class to be used as non-persistent request queue. The value for the :setting:`SCHEDULER_MEMORY_QUEUE` setting is used by default. :type mqclass: class :param logunser: A boolean that indicates whether or not unserializable requests should be logged. The value for the :setting:`SCHEDULER_DEBUG` setting is used by default. :type logunser: bool :param stats: A stats collector object to record stats about the request scheduling process. The value for the :setting:`STATS_CLASS` setting is used by default. :type stats: :class:`scrapy.statscollectors.StatsCollector` instance or similar: any class that implements the `StatsCollector` interface :param pqclass: A class to be used as priority queue for requests. The value for the :setting:`SCHEDULER_PRIORITY_QUEUE` setting is used by default. :type pqclass: class :param crawler: The crawler object corresponding to the current crawl. :type crawler: :class:`scrapy.crawler.Crawler` NFjobdirlogunserr(c ||_|||_||_||_||_||_||_||_dSr) df_dqdirdqdirpqclassdqclassmqclassrFstatsr() r/ dupefilterrErLrMrFrNrKr(s r__init__zScheduler.__init__sL[[((        rrr,c rt|jd}|t||j|t|jt|jdt|jd|jd|jt|jd|S)zh Factory method, initializes the scheduler with arguments taken from the crawl settings DUPEFILTER_CLASSSCHEDULER_DISK_QUEUESCHEDULER_MEMORY_QUEUESCHEDULER_DEBUGSCHEDULER_PRIORITY_QUEUE)rOrErLrMrFrNrKr()rsettingsr r getboolrN)rr(dupefilter_clss rr*zScheduler.from_crawlers %W%56H%IJJs&~w7GQQ7+,, 01G HII 01I JKK%--.?@@- 01K LMM    rc(t|dkS)Nr)lenr8s rrzScheduler.has_pending_requestss4yy1}rr+c||_||_|jr|nd|_|jS)z (1) initialize the memory queue (2) initialize the disk queue if the ``jobdir`` attribute is a valid directory (3) return the result of the dupefilter's ``open`` method N)r+_mqmqsrJ_dqdqsrHr0r.s rr0zScheduler.opensD  88::!%5488:::w||~~rr2c|jP|j}t|jtsJ||j||j|S)z (1) dump pending requests to disk if there is a disk queue (2) return the result of the dupefilter's ``close`` method )r`r4 isinstancerJr>_write_dqs_staterH)r/r2states rr4zScheduler.closesc 8 5HNN$$Edj#.. . . .  ! !$*e 4 4 4w}}V$$$rr9c|js<|j|r"|j||jdS||}|r"|jd|jn6|||jd|j|jd|jdS)a Unless the received request is filtered out by the Dupefilter, attempt to push it into the disk queue, falling back to pushing it into the memory queue. Increment the appropriate stats, such as: ``scheduler/enqueued``, ``scheduler/enqueued/disk``, ``scheduler/enqueued/memory``. Return ``True`` if the request was stored successfully, ``False`` otherwise. Fzscheduler/enqueued/diskr+zscheduler/enqueued/memoryzscheduler/enqueuedT) dont_filterrH request_seenlogr+_dqpushrN inc_value_mqpush)r/r9dqoks rrzScheduler.enqueue_requests" tw';';G'D'D  GKK - - -5||G$$  R J !:4; O O O O LL ! ! ! J !t{$SSS  K J !5dk J J Jrc|j)t|jt|jznt|jS)z> Return the total amount of enqueued requests )r`r[r^r8s r__len__zScheduler.__len__s315Ws48}}s48}},,#dh--Wrc0|jdS |j|dS#t$rd}|jr1d}t|||ddd|jid|_|jd|jYd}~dSd}~wwxYw) NFTzUnable to serialize request: %(request)s - reason: %(reason)s - no more unserializable requests will be logged (stats being collected))r9r2r+)exc_infoextrazscheduler/unserializablerf) r`push ValueErrorrFloggerwarningr+rNrk)r/r9emsgs rrjzScheduler._dqpushs 8 5  HMM' " " "$4#   } &6  '155!#T[1  !& J !;DK P P P55555 s' BABBc:|j|dSr)r^rvr;s rrlzScheduler._mqpush/s  grcF|j|jSdSr)r`ror8s rrpzScheduler._dqpop2s! 8 "8<<>> !trcHt|jd|j|jdS)z)r/rErJs rrIzScheduler._dqdirTsO  !122E<<>> * D )))u:: trrJct|d}|sgS|d5}tj|cdddS#1swxYwYdS)N active.jsonutf-8encoding)rrr0jsonload)r/rJpathfs rrzScheduler._read_dqs_state]sE=)){{}} I YYY ( ( A9Q<<                  sAA"%A"rdct|ddd5}tj||ddddS#1swxYwYdS)Nrwrr)rr0rdump)r/rJrdrs rrczScheduler._write_dqs_stateds % ' ' , ,S7 , C C q IeQ                     sA  A A )NNNFNNN) r!r"r#r$rr>r?r rPr=rrAr*rr rr0r4r rrintrrrjrlrpr]r_rIlistrrcr%rrrBrBvsk00j!%%)   '"( ${+    [ d 6 hx&8     %C %HX$6 % % % %w4,hw/(XXXXX w40w4)    &Xc]x} S T     c $ 4      r)rloggingabcrpathlibrtypingrrrtwisted.internet.deferrscrapy.crawlerr scrapy.http.requestr scrapy.spidersr scrapy.utils.jobr scrapy.utils.miscr r getLoggerr!rxrrr'rArBr%rrrs **********++++++""""""''''''!!!!!!$$$$$$::::::::  8 $ $        &K$K$K$K$K$/K$K$K$K$\gm;777 p p p p p p p p p p r