1egJvdZddlmZddlmZddlmZmZddlm Z ddl m Z ddl m Z dZGd d Zd S) z; Item Loader See documentation in docs/topics/loaders.rst )suppress) ItemAdapter) extract_regexflatten)wrap_loader_context)Identity) arg_to_iterctt5d|jvr|jcdddS dddn #1swxYwY|S)z Allow to use single-argument functions as input or output processors (no need to define an unused first 'self' argument) .N)rAttributeError __qualname____func__)methods 4lib/python3.11/site-packages/itemloaders/__init__.pyunbound_methodrs . ! !## f) ) )?######## )############### Ms?AAc^eZdZdZeZeZeZd"dZ e dZ e dZ dZ dZddd Zddd Zd Zd Zddd ZdZdZdZdZdZd#dZdZdZdddZdddZdddZdZdddZ dddZ!dddZ"dZ#dddZ$dddZ%ddd Z&d!Z'dS)$ ItemLoadera Return a new Item Loader for populating the given item. If no item is given, one is instantiated automatically using the class in :attr:`default_item_class`. When instantiated with a :param ``selector`` parameter the :class:`ItemLoader` class provides convenient mechanisms for extracting data from web pages using parsel_ selectors. :param item: The item instance to populate using subsequent calls to :meth:`~ItemLoader.add_xpath`, :meth:`~ItemLoader.add_css`, :meth:`~ItemLoader.add_jmes` or :meth:`~ItemLoader.add_value`. :type item: :class:`dict` object :param selector: The selector to extract data from, when using the :meth:`add_xpath` (resp. :meth:`add_css`, :meth:`add_jmes`) or :meth:`replace_xpath` (resp. :meth:`replace_css`, :meth:`replace_jmes`) method. :type selector: :class:`~parsel.selector.Selector` object The item, selector and the remaining keyword arguments are assigned to the Loader context (accessible through the :attr:`context` attribute). .. attribute:: item The item object being parsed by this Item Loader. This is mostly used as a property so when attempting to override this value, you may want to check out :attr:`default_item_class` first. .. attribute:: context The currently active :ref:`Context ` of this Item Loader. Refer to for more information about the Loader Context. .. attribute:: default_item_class An Item class (or factory), used to instantiate items when not given in the ``__init__`` method. .. warning:: Currently, this factory/class needs to be callable/instantiated without any arguments. If you are using ``dataclasses``, please consider the following alternative:: from dataclasses import dataclass, field from typing import Optional @dataclass class Product: name: Optional[str] = field(default=None) price: Optional[float] = field(default=None) .. attribute:: default_input_processor The default input processor to use for those fields which don't specify one. .. attribute:: default_output_processor The default output processor to use for those fields which don't specify one. .. attribute:: selector The :class:`~parsel.selector.Selector` object to extract data from. It's the selector given in the ``__init__`` method. This attribute is meant to be read-only. .. _parsel: https://parsel.readthedocs.io/en/latest/ Nc v||_||||}||_||d<||_||_i|_t|D]B\}}|j |g|j |xxt|z cc<CdS)Nselectoritem) rupdatedefault_item_class _local_itemcontextparent _local_valuesritems_values setdefaultr )selfrrrr field_namevalues r__init__zItemLoader.__init__fs  ))) <**,,D  !,T!2!2!8!8!:!: ; ; J L # #J 3 3 3 L $ $ $ E(:(: : $ $ $ $ ; ;c6|j |jjS|jSN)rrrr!s rrzItemLoader._valuesus ; ";& &% %r%c6|j |jjS|jSr')rrrr(s rrzItemLoader.item|s ; ";# ## #r%c |j|}|||jd|j|d|}|S)am Create a nested loader with an xpath selector. The supplied selector is applied relative to selector associated with this :class:`ItemLoader`. The nested loader shares the item with the parent :class:`ItemLoader` so calls to :meth:`add_xpath`, :meth:`add_value`, :meth:`replace_value`, etc. will behave as expected. rrr)rxpathr __class__r)r!r-rr subloaders r nested_xpathzItemLoader.nested_xpathsS=&&u--)))"DNJ $JJ'JJ r%c |j|}|||jd|j|d|}|S)aj Create a nested loader with a css selector. The supplied selector is applied relative to selector associated with this :class:`ItemLoader`. The nested loader shares the item with the parent :class:`ItemLoader` so calls to :meth:`add_xpath`, :meth:`add_value`, :meth:`replace_value`, etc. will behave as expected. rr+r,)rcssrr.r)r!r2rrr/s r nested_csszItemLoader.nested_csssS=$$S)))))"DNJ $JJ'JJ r%)rec|j|g|Rd|i|}|dS|s2|D]\}}|||dS|||dS)a Process and then add the given ``value`` for the given field. The value is first passed through :meth:`get_value` by giving the ``processors`` and ``kwargs``, and then passed through the :ref:`field input processor ` and its result appended to the data collected for that field. If the field already contains collected data, the new data is added. The given ``field_name`` can be ``None``, in which case values for multiple fields may be added. And the processed value should be a dict with field_name mapped to values. Examples:: loader.add_value('name', 'Color TV') loader.add_value('colours', ['white', 'blue']) loader.add_value('length', '100') loader.add_value('name', 'name: foo', TakeFirst(), re='name: (.+)') loader.add_value(None, {'name': 'foo', 'sex': 'male'}) r4N) get_valuer _add_valuer!r"r#r4 processorskwkvs r add_valuezItemLoader.add_values,u?z???b?B?? = F /  & &11%%%% & & OOJ . . . . .r%c|j|g|Rd|i|}|dS|s2|D]\}}|||dS|||dS)z Similar to :meth:`add_value` but replaces the collected data with the new value instead of adding it. r4N)r6r_replace_valuer8s r replace_valuezItemLoader.replace_values u?z???b?B?? = F 3  * *1##Aq)))) * *    E 2 2 2 2 2r%ct|}|||}|r?|j|g|j|xxt|z cc<dSdSr')r _process_input_valuerr )r!r"r#processed_values rr7zItemLoader._add_values}E""33JFF  E L # #J 3 3 3 L $ $ $ O(D(D D $ $ $ $ $ E Er%ch|j|d|||dSr')rpopr7)r!r"r#s rr?zItemLoader._replace_values4 T***  E*****r%chr*t|}tfd|D}|D]}|n||}t||j} ||}*#t$rK}t d|jjd|dt|jdt|d |d}~wwxYw|S)ar Process the given ``value`` by the given ``processors`` and keyword arguments. Available keyword arguments: :param re: a regular expression to use for extracting data from the given value using :func:`~parsel.utils.extract_regex` method, applied before processors :type re: str or typing.Pattern Examples: >>> from itemloaders import ItemLoader >>> from itemloaders.processors import TakeFirst >>> loader = ItemLoader() >>> loader.get_value('name: foo', TakeFirst(), str.upper, re='name: (.+)') 'FOO' c38K|]}t|VdSr')r).0xr4s r z'ItemLoader.get_value..s-@@QM"a00@@@@@@r%NzError with processor  value= error=': ') r rrr Exception ValueErrorr.__name__typestr)r!r#r4r9r:proc_proces ` rr6zItemLoader.get_values(  A&&E@@@@%@@@@@E  D}E&tT\::D U     j///Q8H8H8H#a&&&&R   s A B/$AB**B/ct|j}t|jD]}||}||||<|jS)z Populate the item with the data collected so far, and return it. The data collected is first passed through the :ref:`output processors ` to get the final value to assign to each item field. )rrtuplerget_output_value)r!adapterr"r#s r load_itemzItemLoader.load_itemsX di(( -- , ,J))*55E &+ #|r%c @||}t||j}|j|g} ||S#t $rA}t d|d|dt|jdt|d |d}~wwxYw)z Return the collected values parsed using the output processor, for the given field. This method doesn't populate or modify the item at all. z#Error with output processor: field=rKrLrMrNN) get_output_processorrrrgetrOrPrRrQrS)r!r"rTr#rVs rrYzItemLoader.get_output_value s ((44"466   R00 4;;    *::uuud1gg&6&6&6A@  s A BColor TV

loader.add_xpath('name', '//p[@class="product-name"]') # HTML snippet:

the price is $1200

loader.add_xpath('price', '//p[@id="price"]', re='the price is (.*)') r4N)_get_xpathvaluesr=r!r"r-r4r9r:valuess r add_xpathzItemLoader.add_xpathKsM&'&u3333z6DJDDD2DDDDDDr%cJ|j|fi|}|j||g|Rd|i|dS)z` Similar to :meth:`add_xpath` but replaces collected data instead of adding it. r4N)rur@rvs r replace_xpathzItemLoader.replace_xpathasN'&u3333:vH HHHrHRHHHHHr%cD|j|fi|}|j|g|Rd|i|S)a Similar to :meth:`ItemLoader.get_value` but receives an XPath instead of a value, which is used to extract a list of unicode strings from the selector associated with this :class:`ItemLoader`. :param xpath: the XPath to extract data from :type xpath: str :param re: a regular expression to use for extracting data from the selected XPath region :type re: str or typing.Pattern Examples:: # HTML snippet:

Color TV

loader.get_xpath('//p[@class="product-name"]') # HTML snippet:

the price is $1200

loader.get_xpath('//p[@id="price"]', TakeFirst(), re='the price is (.*)') r4)rur6)r!r-r4r9r:rws r get_xpathzItemLoader.get_xpathhsE*'&u3333t~f?z???b?B???r%c t|}tfd|DS)Nc3bK|])}jj|fiV*dSr')rr-getall)rHr-r:r!s rrJz.ItemLoader._get_xpathvalues..sGUUU*t}*577B77>>@@UUUUUUr%rsr r)r!xpathsr:s` `rruzItemLoader._get_xpathvaluessI ##%%%V$$UUUUUfUUUUUUr%cX||}|j||g|Rd|i|dS)az Similar to :meth:`ItemLoader.add_value` but receives a CSS selector instead of a value, which is used to extract a list of unicode strings from the selector associated with this :class:`ItemLoader`. See :meth:`get_css` for ``kwargs``. :param css: the CSS selector to extract data from :type css: str Examples:: # HTML snippet:

Color TV

loader.add_css('name', 'p.product-name') # HTML snippet:

the price is $1200

loader.add_css('price', 'p#price', re='the price is (.*)') r4N)_get_cssvaluesr=r!r"r2r4r9r:rws radd_csszItemLoader.add_csssF$$$S))z6DJDDD2DDDDDDr%cX||}|j||g|Rd|i|dS)z^ Similar to :meth:`add_css` but replaces collected data instead of adding it. r4N)rr@rs r replace_csszItemLoader.replace_csssG$$S)):vH HHHrHRHHHHHr%cR||}|j|g|Rd|i|S)a Similar to :meth:`ItemLoader.get_value` but receives a CSS selector instead of a value, which is used to extract a list of unicode strings from the selector associated with this :class:`ItemLoader`. :param css: the CSS selector to extract data from :type css: str :param re: a regular expression to use for extracting data from the selected CSS region :type re: str or typing.Pattern Examples:: # HTML snippet:

Color TV

loader.get_css('p.product-name') # HTML snippet:

the price is $1200

loader.get_css('p#price', TakeFirst(), re='the price is (.*)') r4)rr6)r!r2r4r9r:rws rget_csszItemLoader.get_csss>($$S))t~f?z???b?B???r%ct|}tfd|DS)Nc3pK|]0}j|V1dSr')rr2r)rHr2r!s rrJz,ItemLoader._get_cssvalues..s?GG3t}((--4466GGGGGGr%r)r!cssss` rrzItemLoader._get_cssvaluessE ##%%%4  GGGG$GGGGGGr%cX||}|j||g|Rd|i|dS)ac Similar to :meth:`ItemLoader.add_value` but receives a JMESPath selector instead of a value, which is used to extract a list of unicode strings from the selector associated with this :class:`ItemLoader`. See :meth:`get_jmes` for ``kwargs``. :param jmes: the JMESPath selector to extract data from :type jmes: str Examples:: # HTML snippet: {"name": "Color TV"} loader.add_jmes('name') # HTML snippet: {"price": the price is $1200"} loader.add_jmes('price', TakeFirst(), re='the price is (.*)') r4N)_get_jmesvaluesr=r!r"jmesr4r9r:rws radd_jmeszItemLoader.add_jmessF$%%d++z6DJDDD2DDDDDDr%cX||}|j||g|Rd|i|dS)z_ Similar to :meth:`add_jmes` but replaces collected data instead of adding it. r4N)rr@rs r replace_jmeszItemLoader.replace_jmessG%%d++:vH HHHrHRHHHHHr%cR||}|j|g|Rd|i|S)a Similar to :meth:`ItemLoader.get_value` but receives a JMESPath selector instead of a value, which is used to extract a list of unicode strings from the selector associated with this :class:`ItemLoader`. :param jmes: the JMESPath selector to extract data from :type jmes: str :param re: a regular expression to use for extracting data from the selected JMESPath :type re: str or typing.Pattern Examples:: # HTML snippet: {"name": "Color TV"} loader.get_jmes('name') # HTML snippet: {"price": the price is $1200"} loader.get_jmes('price', TakeFirst(), re='the price is (.*)') r4)rr6)r!rr4r9r:rws rget_jmeszItemLoader.get_jmess>(%%d++t~f?z???b?B???r%ct|}tjdst dt fd|DS)Njmespathz6Please install parsel >= 1.8.1 to get jmespath supportc3pK|]0}j|V1dSr')rrr)rHrr!s rrJz-ItemLoader._get_jmesvalues..s?OOt}--d33::<<OOOOOOr%)rsr hasattrrr r)r!jmesss` rrzItemLoader._get_jmesvaluessp ##%%%E""t}j11  H OOOOOOOOOOr%)NNNr')(rQ __module__r __doc__dictrrrerjr$propertyrrr0r3r=r@r7r?r6r[rYr`rgr]rdrBrsrxrzr|rurrrrrrrrr,r%rrrsDDL&hjj'xzz ; ; ; ;&&X& $$X$      <@/////>@D 3 3 3 3 3EEE+++04$$$$$L    000$$$$$$,,,,&<@EEEEE,@DIIIII04@@@@@0VVV 8<EEEEE*<@IIIII,0@@@@@.HHH :>EEEEE*>BIIIII.2@@@@@.PPPPPr%rN)r contextlibr itemadapterr parsel.utilsrritemloaders.commonritemloaders.processorsritemloaders.utilsr rrr,r%rrs  ######////////222222++++++))))))\P\P\P\P\P\P\P\P\P\Pr%