o V`J@sdZddlmZddlmZddlmZddlZddlZddlZddlm Z ddlm Z ddl m Z ddl Z ddlZdd lmZdd lmZejd krOejZnejZ dddZ dddZ  dddZGdddeZddZdS)z(Utilities for text input preprocessing. )absolute_import)division)print_functionN) OrderedDict) defaultdict)md5)range)zip!!"#$%&()*+,-./:;<=>?@[\]^_`{|}~ T cs|r|}tjdkr@t|trfdd|D}||}n3tdkr4t|t|}||}n|D]}||}q6nfdd|D}t|}||}| }dd|DS)aConverts a text to a sequence of words (or tokens). # Arguments text: Input text (string). filters: list (or concatenation) of characters to filter out, such as punctuation. Default: ``!"#$%&()*+,-./:;<=>?@[\]^_`{|}~\t\n``, includes basic punctuation, tabs, and newlines. lower: boolean. Whether to convert the input to lowercase. split: str. Separator for word splitting. # Returns A list of words (or tokens). r csi|] }t|tqS)ordunicode.0csplitr8lib/python3.10/site-packages/keras_preprocessing/text.py /sz)text_to_word_sequence..csi|]}|qSrrrrrrr:scSsg|]}|r|qSrr)rirrr ?z)text_to_word_sequence..) lowersys version_info isinstancer translatelen maketransreplacer)textfiltersrrZ translate_maprZtranslate_dictseqrrrtext_to_word_sequences&        r'cCst||t|||dS)aOne-hot encodes a text into a list of word indexes of size n. This is a wrapper to the `hashing_trick` function using `hash` as the hashing function; unicity of word to index mapping non-guaranteed. # Arguments text: Input text (string). n: int. Size of vocabulary. filters: list (or concatenation) of characters to filter out, such as punctuation. Default: ``!"#$%&()*+,-./:;<=>?@[\]^_`{|}~\t\n``, includes basic punctuation, tabs, and newlines. lower: boolean. Whether to set the text to lowercase. split: str. Separator for word splitting. # Returns List of integers in [1, n]. Each integer encodes a word (unicity non-guaranteed). ) hash_functionr%rr) hashing_trickhash)r$nr%rrrrrone_hotBs r,csBdurtndkrddt||||d}fdd|DS)aConverts a text to a sequence of indexes in a fixed-size hashing space. # Arguments text: Input text (string). n: Dimension of the hashing space. hash_function: defaults to python `hash` function, can be 'md5' or any function that takes in input a string and returns a int. Note that 'hash' is not a stable hashing function, so it is not consistent across different runs, while 'md5' is a stable hashing function. filters: list (or concatenation) of characters to filter out, such as punctuation. Default: ``!"#$%&()*+,-./:;<=>?@[\]^_`{|}~\t\n``, includes basic punctuation, tabs, and newlines. lower: boolean. Whether to set the text to lowercase. split: str. Separator for word splitting. # Returns A list of integer word indices (unicity non-guaranteed). `0` is a reserved index that won't be assigned to any word. Two or more words may be assigned to the same index, due to possible collisions by the hashing function. The [probability]( https://en.wikipedia.org/wiki/Birthday_problem#Probability_table) of a collision is in relation to the dimension of the hashing space and the number of distinct objects. NrcSstt|dS)N)intrencodeZ hexdigest)wrrrr(sz$hashing_trick..hash_function)r%rrcs g|] }|ddqS)rr)rr0r(r+rrrs z!hashing_trick..)r*r')r$r+r(r%rrr&rr1rr)_s!r)c@s|eZdZdZ       ddd Zd d Zd d ZddZddZddZ ddZ d ddZ d ddZ ddZ ddZdS)! TokenizeraOText tokenization utility class. This class allows to vectorize a text corpus, by turning each text into either a sequence of integers (each integer being the index of a token in a dictionary) or into a vector where the coefficient for each token could be binary, based on word count, based on tf-idf... # Arguments num_words: the maximum number of words to keep, based on word frequency. Only the most common `num_words-1` words will be kept. filters: a string where each element is a character that will be filtered from the texts. The default is all punctuation, plus tabs and line breaks, minus the `'` character. lower: boolean. Whether to convert the texts to lowercase. split: str. Separator for word splitting. char_level: if True, every character will be treated as a token. oov_token: if given, it will be added to word_index and used to replace out-of-vocabulary words during text_to_sequence calls By default, all punctuation is removed, turning the texts into space-separated sequences of words (words maybe include the `'` character). These sequences are then split into lists of tokens. They will then be indexed or vectorized. `0` is a reserved index that won't be assigned to any word. Nr Tr Frc Ksd|vrtd|d}|rtdt|t|_tt|_ ||_ ||_ ||_ ||_ ||_||_||_tt|_i|_i|_dS)NZnb_wordszDThe `nb_words` argument in `Tokenizer` has been renamed `num_words`.z Unrecognized keyword arguments: )warningswarnpop TypeErrorstrr word_countsrr. word_docsr%rr num_wordsdocument_count char_level oov_token index_docs word_index index_word) selfr:r%rrr<r=r;kwargsrrr__init__s"     zTokenizer.__init__cCsl|D][}|jd7_|jst|tr*|jr't|tr#dd|D}n|}|}n t||j|j|j}|D]}||jvrG|j|d7<q6d|j|<q6t |D] }|j |d7<qQqt|j }|j dddd|j durvg}n|j g}|d d |Dtt|ttdt|d|_d d |j D|_t|j D] \}}||j|j|<qdS) aUpdates internal vocabulary based on a list of texts. In the case where texts contains lists, we assume each entry of the lists to be a token. Required before using `texts_to_sequences` or `texts_to_matrix`. # Arguments texts: can be a list of strings, a generator of strings (for memory-efficiency), or a list of list of strings. rcSg|]}|qSrrrZ text_elemrrrrrz*Tokenizer.fit_on_texts..cSs|dS)Nrr)xrrrsz(Tokenizer.fit_on_texts..T)keyreverseNcss|]}|dVqdS)rNr)rZwcrrr sz)Tokenizer.fit_on_texts..cSsi|]\}}||qSrr)rr0rrrrrsz*Tokenizer.fit_on_texts..)r;r<rlistrr'r%rr8setr9itemssortr=extenddictr rr!r?r@r>)rAtextsr$r&r0ZwcountsZ sorted_vocrrrr fit_on_textssB      zTokenizer.fit_on_textscCsD|jt|7_|D]}t|}|D] }|j|d7<qq dS)a%Updates internal vocabulary based on a list of sequences. Required before using `sequences_to_matrix` (if `fit_on_texts` was never called). # Arguments sequences: A list of sequence. A "sequence" is a list of integer word indices. rN)r;r!rMr>)rA sequencesr&rrrrfit_on_sequencess zTokenizer.fit_on_sequencescCt||S)aNTransforms each text in texts to a sequence of integers. Only top `num_words-1` most frequent words will be taken into account. Only words known by the tokenizer will be taken into account. # Arguments texts: A list of texts (strings). # Returns A list of sequences. )rLtexts_to_sequences_generator)rArRrrrtexts_to_sequences  zTokenizer.texts_to_sequencesc cs|j}|j|j}|D]_}|jst|tr.|jr+t|tr'dd|D}n|}|}n t||j |j|j }g}|D],}|j|}|dur^|rX||krX|durW| |q<| |q<|jdurh| |q<|Vq dS)aTransforms each text in `texts` to a sequence of integers. Each item in texts can also be a list, in which case we assume each item of that list to be a token. Only top `num_words-1` most frequent words will be taken into account. Only words known by the tokenizer will be taken into account. # Arguments texts: A list of texts (strings). # Yields Yields individual sequences. cSrDrrErFrrrr0rz:Tokenizer.texts_to_sequences_generator..N) r:r?getr=r<rrLrr'r%rappend) rArRr:oov_token_indexr$r&vectr0rrrrrWs:       z&Tokenizer.texts_to_sequences_generatorcCrV)aYTransforms each sequence into a list of text. Only top `num_words-1` most frequent words will be taken into account. Only words known by the tokenizer will be taken into account. # Arguments sequences: A list of sequences (list of integers). # Returns A list of texts (strings) )rLsequences_to_texts_generator)rArTrrrsequences_to_textsFrYzTokenizer.sequences_to_textsccs|j}|j|j}|D]A}g}|D]2}|j|}|dur8|r2||kr2|dur1||j|q||q|jdurE||j|qd|}|Vq dS)aTransforms each sequence in `sequences` to a list of texts(strings). Each sequence has to a list of integers. In other words, sequences should be a list of sequences Only top `num_words-1` most frequent words will be taken into account. Only words known by the tokenizer will be taken into account. # Arguments sequences: A list of sequences. # Yields Yields individual texts. Nr )r:r?rZr=r@r[join)rArTr:r\r&r]ZnumZwordrrrr^Ts&     z&Tokenizer.sequences_to_texts_generatorbinarycCs||}|j||dS)zConvert a list of texts to a Numpy matrix. # Arguments texts: list of strings. mode: one of "binary", "count", "tfidf", "freq". # Returns A Numpy matrix. )mode)rXsequences_to_matrix)rArRrbrTrrrtexts_to_matrixts zTokenizer.texts_to_matrixc CsL|js|jrt|jd}ntd|j}|dkr |js tdtt||f}t|D]v\}}|s4q-tt }|D]}||krAq:||d7<q:t | D]R\}} |dkr_| |||<qP|dkrn| t||||<qP|dkryd|||<qP|dkrdt | } t d|jd|j |d} | | |||<qPtd |q-|S) aConverts a list of sequences into a Numpy matrix. # Arguments sequences: list of sequences (a sequence is a list of integer word indices). mode: one of "binary", "count", "tfidf", "freq" # Returns A Numpy matrix. # Raises ValueError: In case of invalid `mode` argument, or if the Tokenizer requires to be fit to sample data. rzKSpecify a dimension (`num_words` argument), or fit on some text data first.Ztfidfz7Fit the Tokenizer on some data before using tfidf mode.countZfreqrarzUnknown vectorization mode:)r:r?r! ValueErrorr;npZzeros enumeraterr.rLrNlogr>rZ) rArTrbr:rGrr&ZcountsjrZtfZidfrrrrcs@  zTokenizer.sequences_to_matrixc Csht|j}t|j}t|j}t|j}t|j}|j|j|j |j |j |j |j |||||d S)a:Returns the tokenizer configuration as Python dictionary. The word count dictionaries used by the tokenizer get serialized into plain JSON, so that the configuration can be read by other projects. # Returns A Python dictionary with the tokenizer configuration. ) r:r%rrr<r=r;r8r9r>r@r?)jsondumpsr8r9r>r?r@r:r%rrr<r=r;)rAZjson_word_countsZjson_word_docsZjson_index_docsZjson_word_indexZjson_index_wordrrr get_configs$    zTokenizer.get_configcKs(|}|jj|d}tj|fi|S)aReturns a JSON string containing the tokenizer configuration. To load a tokenizer from a JSON string, use `keras.preprocessing.text.tokenizer_from_json(json_string)`. # Arguments **kwargs: Additional keyword arguments to be passed to `json.dumps()`. # Returns A JSON string containing the tokenizer configuration. ) class_nameconfig)rm __class____name__rkrl)rArBrotokenizer_configrrrto_jsons  zTokenizer.to_json)Nr Tr FNr)ra)rq __module__ __qualname____doc__rCrSrUrXrWr_r^rdrcrmrsrrrrr2s( 6+ 7 r2c Cst|}|d}t|d}t|d}t|d}dd|D}t|d}dd|D}t|d }td i|}||_||_||_||_ ||_ |S) zParses a JSON tokenizer configuration file and returns a tokenizer instance. # Arguments json_string: JSON string encoding a tokenizer configuration. # Returns A Keras Tokenizer instance ror8r9r>cSi|] \}}t||qSrr.rkvrrrrz'tokenizer_from_json..r@cSrwrrxryrrrrr|r?Nr) rkloadsrZr5rNr2r8r9r>r?r@) Z json_stringrrror8r9r>r@r?Z tokenizerrrrtokenizer_from_jsons r~)r Tr )Nr Tr )rvZ __future__rrrstringrr3 collectionsrrZhashlibrrkZnumpyrgZ six.movesrr rr"r7r'r,r)objectr2r~rrrrsB          )  . _