o ={c:6@sdZddlZddlZddlZddlmZddlmZddZ edGdd d ej Z ed dd d Z ed      dddZ dS)aiUtilities for preprocessing sequence data. Deprecated: `tf.keras.preprocessing.sequence` APIs are not recommended for new code. Prefer `tf.keras.utils.timeseries_dataset_from_array` and the `tf.data` APIs which provide a much more flexible mechanisms for dealing with sequences. See the [tf.data guide](https://www.tensorflow.org/guide/data) for more details. N) data_utils) keras_exportcCsFgg}}t||D]\}}t||kr||||q ||fS)aCRemoves sequences that exceed the maximum length. Args: maxlen: Int, maximum length of the output sequences. seq: List of lists, where each sublist is a sequence. label: List where each element is an integer. Returns: new_seq, new_label: shortened lists for `seq` and `label`. )ziplenappend)maxlenseqZlabelZnew_seqZ new_labelxyr end_index=%i` is disallowed, as no part of the sequence would be left to be used as current step.) r ValueErrordatatargetslength sampling_ratestride start_index end_indexshufflereverse batch_size) selfrrrrrrrrrrr r r __init__{s6      zTimeseriesGenerator.__init__cCs$|j|j|j|j|j|jS)N)rrrrrr r r __len__s zTimeseriesGenerator.__len__csjrtjjjjdjd}njjj|}t|t |jjjdj}t fdd|D}t fdd|D}j rZ|ddddddf|fS||fS)Nr)sizecs$g|]}j|j|jqSr )rrr.0rowrr r sz3TimeseriesGenerator.__getitem__..csg|]}j|qSr )rr!rr r r$s.) rnprandomrandintrrrrarangeminZarrayr)rindexZrowsiZsamplesrr rr __getitem__s& zTimeseriesGenerator.__getitem__c Cs|j}t|jjtjkr|j}zt|}Wnty*}ztd||d}~ww|j }t|j jtjkr<|j }zt|}WntyU}ztd||d}~ww|||j |j |j |j |j|j|j|jd S)zReturns the TimeseriesGenerator configuration as Python dictionary. Returns: A Python dictionary with the TimeseriesGenerator configuration. zData not JSON Serializable:NzTargets not JSON Serializable:) rrrrrrrrrr)rtype __module__r&__name__tolistjsondumps TypeErrorrrrrrrrrr)rrZ json_dataerZ json_targetsr r r get_configs:    zTimeseriesGenerator.get_configcKs(|}|jj|d}tj|fi|S)aReturns a JSON string containing the timeseries generator configuration. Args: **kwargs: Additional keyword arguments to be passed to `json.dumps()`. Returns: A JSON string containing the tokenizer configuration. ) class_nameconfig)r6 __class__r0r2r3)rkwargsr8Ztimeseries_generator_configr r r to_jsons  zTimeseriesGenerator.to_json)rrrNFFr) r0r/ __qualname____doc__rrr-r6r;r r r r r7sG * #rz0keras.preprocessing.sequence.make_sampling_tableh㈵>cCsVd}t|}d|d<|t||ddd|}||}td|t|S)a2Generates a word rank-based probabilistic sampling table. Used for generating the `sampling_table` argument for `skipgrams`. `sampling_table[i]` is the probability of sampling the word i-th most common word in a dataset (more common words should be sampled less frequently, for balance). The sampling probabilities are generated according to the sampling distribution used in word2vec: ``` p(word) = (min(1, sqrt(word_frequency / sampling_factor) / (word_frequency / sampling_factor))) ``` We assume that the word frequencies follow Zipf's law (s=1) to derive a numerical approximation of frequency(rank): `frequency(rank) ~ 1/(rank * (log(rank) + gamma) + 1/2 - 1/(12*rank))` where `gamma` is the Euler-Mascheroni constant. Args: size: Int, number of possible words to sample. sampling_factor: The sampling factor in the word2vec formula. Returns: A 1D Numpy array of length `size` where the ith entry is the probability that a word of rank i should be sampled. gX9v?rrg??g(@)r&r)logZminimumZsqrt)r Zsampling_factorZgammaZrankZinv_fqfr r r make_sampling_tables  "rBz&keras.preprocessing.sequence.skipgramsr?TFcsjg}g} t|D]O\} } | sq|dur|| tkrqtd| |} tt|| |d} t| | D]#}|| krV||}|s@q3|| |g|rQ| ddgq3| dq3q|dkrtt| |}dd|Dt|fddt|D7}|r| ddgg|7} n| dg|7} |r|durt dd}t |t|t |t| || fS)asGenerates skipgram word pairs. This function transforms a sequence of word indexes (list of integers) into tuples of words of the form: - (word, word in the same window), with label 1 (positive samples). - (word, random word from the vocabulary), with label 0 (negative samples). Read more about Skipgram in this gnomic paper by Mikolov et al.: [Efficient Estimation of Word Representations in Vector Space](http://arxiv.org/pdf/1301.3781v3.pdf) Args: sequence: A word sequence (sentence), encoded as a list of word indices (integers). If using a `sampling_table`, word indices are expected to match the rank of the words in a reference dataset (e.g. 10 would encode the 10-th most frequently occurring token). Note that index 0 is expected to be a non-word and will be skipped. vocabulary_size: Int, maximum possible word index + 1 window_size: Int, size of sampling windows (technically half-window). The window of a word `w_i` will be `[i - window_size, i + window_size+1]`. negative_samples: Float >= 0. 0 for no negative (i.e. random) samples. 1 for same number as positive samples. shuffle: Whether to shuffle the word couples before returning them. categorical: bool. if False, labels will be integers (eg. `[0, 1, 1 .. ]`), if `True`, labels will be categorical, e.g. `[[1,0],[0,1],[0,1] .. ]`. sampling_table: 1D array of size `vocabulary_size` where the entry i encodes the probability to sample a word of rank i. seed: Random seed. Returns: couples, labels: where `couples` are int pairs and `labels` are either 0 or 1. Note: By convention, index 0 in the vocabulary is a non-word and will be skipped. NrrcSsg|]}|dqS)rr )r"cr r r r$lszskipgrams..cs,g|]}|ttddgqS)r)rr'r()r"r,vocabulary_sizewordsr r r$osgcA) enumerater'maxr*rrangerintrr(seed)ZsequencerFZ window_sizeZnegative_samplesrZ categoricalZsampling_tablerLZcoupleslabelsr,ZwiZ window_startZ window_endjZwjZnum_negative_samplesr rEr skipgramssN5        rO)r>)rCr?TFNN)r=r2r'Znumpyr&Z keras.utilsrZ tensorflow.python.util.tf_exportrr SequencerrBrOr r r r s(   @ '