c1dZddlZddlZddlZddlmZddlm Z m Z ej e Z ddZ ddZGd d Zd Zd Zd ZdZdZdS)uQThis module contains functions to compute confirmation on a pair of words or word subsets. Notes ----- The advantage of indirect confirmation measure is that it computes similarity of words in :math:`W'` and :math:`W^{*}` with respect to direct confirmations to all words. Eg. Suppose `x` and `z` are both competing brands of cars, which semantically support each other. However, both brands are seldom mentioned together in documents in the reference corpus. But their confirmations to other words like “road” or “speed” do strongly correlate. This would be reflected by an indirect confirmation measure. Thus, indirect confirmation measures may capture semantic support that direct measures would miss. The formula used to compute indirect confirmation measure is .. math:: \widetilde{m}_{sim(m, \gamma)}(W', W^{*}) = s_{sim}(\vec{v}^{\,}_{m,\gamma}(W'), \vec{v}^{\,}_{m,\gamma}(W^{*})) where :math:`s_{sim}` can be cosine, dice or jaccard similarity and .. math:: \vec{v}^{\,}_{m,\gamma}(W') = \Bigg \{{\sum_{w_{i} \in W'}^{ } m(w_{i}, w_{j})^{\gamma}}\Bigg \}_{j = 1,...,|W|} N)aggregate_segment_simslog_ratio_measureFcg}d}t|D]\}}g}d} |D]j\} } t| ds| g} t| ds| g} ||| | V#t$r| dz } YgwxYw| dkr!|dz }t d| ||t||||dkrt d||S)uFor each topic segmentation, compute average cosine similarity using a :class:`~gensim.topic_coherence.text_analysis.WordVectorsAccumulator`. Parameters ---------- segmented_topics : list of lists of (int, `numpy.ndarray`) Output from the :func:`~gensim.topic_coherence.segmentation.s_one_set`. accumulator : :class:`~gensim.topic_coherence.text_analysis.WordVectorsAccumulator` or :class:`~gensim.topic_coherence.text_analysis.InvertedIndexAccumulator` Word occurrence accumulator. with_std : bool, optional True to also include standard deviation across topic segment sets in addition to the mean coherence for each topic. with_support : bool, optional True to also include support across topic segments. The support is defined as the number of pairwise similarity comparisons were used to compute the overall topic coherence. Returns ------- list of (float[, float[, int]]) Сosine word2vec similarities per topic (with std/support if `with_std`, `with_support`). Examples -------- .. sourcecode:: pycon >>> import numpy as np >>> from gensim.corpora.dictionary import Dictionary >>> from gensim.topic_coherence import indirect_confirmation_measure >>> from gensim.topic_coherence import text_analysis >>> >>> # create segmentation >>> segmentation = [[(1, np.array([1, 2])), (2, np.array([1, 2]))]] >>> >>> # create accumulator >>> dictionary = Dictionary() >>> dictionary.id2token = {1: 'fake', 2: 'tokens'} >>> accumulator = text_analysis.WordVectorsAccumulator({1, 2}, dictionary) >>> _ = accumulator.accumulate([['fake', 'tokens'], ['tokens', 'fake']], 5) >>> >>> # should be (0.726752426218 0.00695475919227) >>> mean, std = indirect_confirmation_measure.word2vec_similarity(segmentation, accumulator, with_std=True)[0] r__iter__z:%d terms for topic %d are not in word2vec model vocabularyz1%d terms for are not in word2vec model vocabulary) enumeratehasattrappendids_similarityZeroDivisionErrorloggerwarningr) segmented_topics accumulatorwith_std with_supporttopic_coherences total_oov topic_indextopic_segments segment_simsnum_oovw_primew_stars Tlib/python3.11/site-packages/gensim/topic_coherence/indirect_confirmation_measure.pyword2vec_similarityr,sUZI'01A'B'B^^# ^ -  OGV7J// $")6:.. "  ##K$>$>w$O$OPPPP$   1   Q; & NI NNL & & &  6|X| \ \]]]]1}WJIVVV s )A55BBnlrrct||||}g}t||D]\} } t| } tjt | } t | D]/\} \} }|| | f}||| f}t||| | <0|t| |||S)aCalculate the indirect cosine measure. Parameters ---------- segmented_topics: list of lists of (int, `numpy.ndarray`) Output from the segmentation module of the segmented topics. accumulator: :class:`~gensim.topic_coherence.text_analysis.InvertedIndexAccumulator` Output from the probability_estimation module. Is an topics: Topics obtained from the trained topic model. measure : str, optional Direct confirmation measure to be used. Supported values are "nlr" (normalized log ratio). gamma: float, optional Gamma value for computing :math:`W'` and :math:`W^{*}` vectors. with_std : bool True to also include standard deviation across topic segment sets in addition to the mean coherence for each topic; default is False. with_support : bool True to also include support across topic segments. The support is defined as the number of pairwise similarity comparisons were used to compute the overall topic coherence. Returns ------- list List of indirect cosine similarity measure for each topic. Examples -------- .. sourcecode:: pycon >>> from gensim.corpora.dictionary import Dictionary >>> from gensim.topic_coherence import indirect_confirmation_measure, text_analysis >>> import numpy as np >>> >>> # create accumulator >>> dictionary = Dictionary() >>> dictionary.id2token = {1: 'fake', 2: 'tokens'} >>> accumulator = text_analysis.InvertedIndexAccumulator({1, 2}, dictionary) >>> accumulator._inverted_index = {0: {2, 3, 4}, 1: {3, 5}} >>> accumulator._num_docs = 5 >>> >>> # create topics >>> topics = [np.array([1, 2])] >>> >>> # create segmentation >>> segmentation = [[(1, np.array([1, 2])), (2, np.array([1, 2]))]] >>> obtained = indirect_confirmation_measure.cosine_similarity(segmentation, accumulator, topics, 'nlr', 1) >>> print(obtained[0]) 0.623018926945 ) ContextVectorComputerziptuplenpzeroslenr_cossimr r)rrtopicsmeasuregammarrcontext_vectorsr topic_wordsrrirr w_prime_cv w_star_cvs rcosine_similarityr.vsf,GV[%PPO'*63C'D'D^^# ^K(( xN 3 344 $-n$=$= = = A (+)=>J' (; context_vector. Example ------- .. sourcecode:: pycon >>> from gensim.corpora.dictionary import Dictionary >>> from gensim.topic_coherence import indirect_confirmation_measure, text_analysis >>> import numpy as np >>> >>> # create measure, topics >>> measure = 'nlr' >>> topics = [np.array([1, 2])] >>> >>> # create accumulator >>> dictionary = Dictionary() >>> dictionary.id2token = {1: 'fake', 2: 'tokens'} >>> accumulator = text_analysis.WordVectorsAccumulator({1, 2}, dictionary) >>> _ = accumulator.accumulate([['fake', 'tokens'], ['tokens', 'fake']], 5) >>> cont_vect_comp = indirect_confirmation_measure.ContextVectorComputer(measure, topics, accumulator, 1) >>> cont_vect_comp.mapping {1: 0, 2: 1} >>> cont_vect_comp.vocab_size 2 c|dkr t|_ntdt||_t |j|_||_||_i|_ i|_ dS)NrzGThe direct confirmation measure you entered is not currently supported.) _pair_npmi similarity ValueError_map_to_contiguousmappingr$ vocab_sizerr( sim_cachecontext_vector_cache)selfr'r&rr(s r__init__zContextVectorComputer.__init__sw e  [(DOOY[[ [*&11 dl++& $&!!!r/c|j|SN)compute_context_vector)r:idxs r __getitem__z!ContextVectorComputer.__getitem__s*t*C00r/ct||}|j|d}| |||}||j|<|S)aCheck if (segment_word_ids, topic_word_ids) context vector has been cached. Parameters ---------- segment_word_ids: list Ids of words in segment. topic_word_ids: list Ids of words in topic. Returns ------- csr_matrix :class:`~scipy.sparse.csr` If context vector has been cached, then return corresponding context vector, else compute, cache, and return. N)_key_for_segmentr9get _make_seg)r:segment_word_idstopic_word_idskeycontext_vectors rr>z,ContextVectorComputer.compute_context_vectors[ /@@266sDAA  <!^^,z2ContextVectorComputer._make_seg..&s7OOsvsCj1122OOOOOOr/) sps lil_matrixr7r r6r8r3rr(tocsr)r:rErFrHr?pairrNs @rrDzContextVectorComputer._make_segs!(<=='44 3 02 ! J JC<$a(COOOO>NOOO J Jt~-S+/??4AQ+R+RDN4(s###t~d';tz'II####  J ##%%%r/N)__name__ __module__ __qualname____doc__r;r@r>rDr/rrrs[,,\ ' ' '111.&&&&&r/rc4t|gg|ddS)aCompute normalized pairwise mutual information (**NPMI**) between a pair of words. Parameters ---------- pair : (int, int) The pair of words (word_id1, word_id2). accumulator : :class:`~gensim.topic_coherence.text_analysis.InvertedIndexAccumulator` Word occurrence accumulator from probability_estimation. Return ------ float NPMI between a pair of words. Tr)r)rSrs rr2r2/s tfX{D 9 9! <>$ :c??Z__#D EEr/c^tjtj|jdzS)N)r"sqrtsumdata) sparse_vecs rr]r]Fs# 726*/Q.// 0 00r/cni}d}tj|D]}||vr |||<|dz }|S)Nrr) itertoolschain from_iterable) ids_iterableuniq_idsnid_s rr5r5JsPH A,,\:: h  HSM FA Or/cLt|drt|n|}||fS)z:A segment may have a single number of an iterable of them.r)r r!)segmentr* segment_keys rrBrBTs+$+GZ$@$@M%...gK  ##r/)FF)rrFF)rWrgloggingnumpyr" scipy.sparsesparserP2gensim.topic_coherence.direct_confirmation_measurerr getLoggerrTr rr.rr2r%r]r5rBrXr/rrws34hhhhhhhh  8 $ $GGGGTFK