3df$dZddlZddlmZmZmZmZmZmZm Z ddl m Z ddl Z ddl mZddlmZddlmZddlmZdd lmZdd lmZed d ZdZGdd eZdZdS)z This module implements the Request class which is used to represent HTTP requests in Scrapy. See documentation in docs/topics/request-response.rst N)CallableListOptionalTupleTypeTypeVarUnion)safe_url_string)obsolete_setter)Headers)curl_to_request_kwargs)to_bytes) object_ref) escape_ajaxRequestTypeVarRequest)boundc td)anWhen assigned to the ``callback`` parameter of :class:`~scrapy.http.Request`, it indicates that the request is not meant to have a spider callback at all. For example: .. code-block:: python Request("https://example.com", callback=NO_CALLBACK) This value should be used by :ref:`components ` that create and handle their own requests, e.g. through :meth:`scrapy.core.engine.ExecutionEngine.download`, so that downloader middlewares handling such requests can treat them differently from requests intended for the :meth:`~scrapy.Spider.parse` callback. zThe NO_CALLBACK callback has been called. This is a special callback value intended for requests whose callback is never meant to be called.) RuntimeError)argskwargss )rrrAs r__repr__zRequest.__repr__s,4;,,,,,,rc*|Sr@)replacerAs rcopyz Request.copys||~~rc|jD]&}||t||'|d|j}||i|S)zOCreate a new Request with the same attributes except for those given new valuescls)r) setdefaultgetattrpop __class__)r=rrxrXs rrUzRequest.replaces` 3 3A   aq!1!1 2 2 2 2jj//sD#F###rTrX curl_commandignore_unknown_optionsc \t||}|||di|S)a Create a Request object from a string containing a `cURL `_ command. It populates the HTTP method, the URL, the headers, the cookies and the body. It accepts the same arguments as the :class:`Request` class, taking preference and overriding the values of the same arguments contained in the cURL command. Unrecognized options are ignored by default. To raise an error when finding unknown options call this method by passing ``ignore_unknown_options=False``. .. caution:: Using :meth:`from_curl` from :class:`~scrapy.http.Request` subclasses, such as :class:`~scrapy.http.JSONRequest`, or :class:`~scrapy.http.XmlRpcRequest`, as well as having :ref:`downloader middlewares ` and :ref:`spider middlewares ` enabled, such as :class:`~scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware`, :class:`~scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`, or :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`, may modify the :class:`~scrapy.http.Request` object. To translate a cURL command into a Scrapy request, you may use `curl2scrapy `_. )r update)rXr^r_rrequest_kwargss r from_curlzRequest.from_curls?D0 >TUUf%%%s$$^$$$r)spiderrez scrapy.Spiderc|jt|jrt||jn|jt|jrt||jn|jt |jd}|jD]&}||t||'t|tur|j dz|j jz|d<|S)aReturn a dictionary containing the Request's data. Use :func:`~scrapy.utils.request.request_from_dict` to convert back into a :class:`~scrapy.Request` object. If a spider is given, this method will try to find out the name of the spider methods used as callback and errback and include them in the output dict, raising an exception if they cannot be found. )rrr&r._class)rr6r _find_methodr&r9rr)rYrZr7r __module__r\r8)r=redattrs rto_dictzRequest.to_dicts8 && VT];;; %%|FDL999DL)) O 4 4D LLwtT22 3 3 3 3 ::W $ J/C/$.2IIAhKr) Nr*NNNNr+rFNNN)r,r)T)$r8rj __qualname____doc__r)rr/__annotations__rrr9r bytesrr4boolr>propertyr(r"rEr1r rrLr2r r#rSrVrU classmethodrrrdrmrarrrr/sb#Jc3h(,"&,059#!&*%)$(':': ':8$': ': $ ': uUCZ() ':%d4j 012':tn':':':':(#':S "':D>': ':':':':R4X dX # LC LD L L L L (8__Xu== > >C5LhuS%Z'89LdLLLL 8Iy&AA B BD #X-#----$$$$(,#%#% . !#%#%!%#%  #%#%#%[#%J>B/!:drc|rJt|dr:tj|tj}|D]\}}|j|jur|cSt d|d|)z#Helper function for Request.to_dict__func__) predicatez Function z is not an instance method in: )hasattrinspect getmembersismethodrvrH)objfuncmembersnameobj_funcs rriris wtZ(( $SG4DEEE%  ND(  DM1    KKKcKK L LLr)rorytypingrrrrrrr w3lib.urlr scrapyscrapy.http.commonr scrapy.http.headersr scrapy.utils.curlr scrapy.utils.pythonrscrapy.utils.trackrefrscrapy.utils.urlrrrrrirarrrsL HHHHHHHHHHHHHHHHHH%%%%%% ......''''''444444((((((,,,,,,(((((();;;0yyyyyjyyyx M M M M Mr