c TdZddlZddlZejeZdZddZdZ d dZ dS) zaThis module contains functions to compute direct confirmation on a pair of words or word subsets.Ng-q=Fcrg}t|j}|D]}g}|D]r\}} || } ||| f} tj| |z tz| |z z } n#t $rd} Ynt $rd} YnwxYw|| s|t||||S)aCalculate the log-conditional-probability measure which is used by coherence measures such as `U_mass`. This is defined as :math:`m_{lc}(S_i) = log \frac{P(W', W^{*}) + \epsilon}{P(W^{*})}`. Parameters ---------- segmented_topics : list of lists of (int, int) Output from the :func:`~gensim.topic_coherence.segmentation.s_one_pre`, :func:`~gensim.topic_coherence.segmentation.s_one_one`. accumulator : :class:`~gensim.topic_coherence.text_analysis.InvertedIndexAccumulator` Word occurrence accumulator from :mod:`gensim.topic_coherence.probability_estimation`. with_std : bool, optional True to also include standard deviation across topic segment sets in addition to the mean coherence for each topic. with_support : bool, optional True to also include support across topic segments. The support is defined as the number of pairwise similarity comparisons were used to compute the overall topic coherence. Returns ------- list of float Log conditional probabilities measurement for each topic. Examples -------- .. sourcecode:: pycon >>> from gensim.topic_coherence import direct_confirmation_measure, text_analysis >>> from collections import namedtuple >>> >>> # Create dictionary >>> id2token = {1: 'test', 2: 'doc'} >>> token2id = {v: k for k, v in id2token.items()} >>> dictionary = namedtuple('Dictionary', 'token2id, id2token')(token2id, id2token) >>> >>> # Initialize segmented topics and accumulator >>> segmentation = [[(1, 2)]] >>> >>> accumulator = text_analysis.InvertedIndexAccumulator({1, 2}, dictionary) >>> accumulator._inverted_index = {0: {2, 3, 4}, 1: {3, 5}} >>> accumulator._num_docs = 5 >>> >>> # result should be ~ ln(1 / 2) = -0.693147181 >>> result = direct_confirmation_measure.log_conditional_probability(segmentation, accumulator)[0] g) floatnum_docsnplogEPSILONKeyErrorZeroDivisionErrorappendaggregate_segment_sims) segmented_topics accumulatorwith_std with_supporttopic_coherencesrs_i segment_simsw_primew_star w_star_countco_occur_countm_lc_is Rlib/python3.11/site-packages/gensim/topic_coherence/direct_confirmation_measure.pylog_conditional_probabilityrs \[)**H^^ " ( (OGV *62 !,Wf_!=.8";w!F>> from gensim.topic_coherence import direct_confirmation_measure >>> >>> segment_sims = [0.2, 0.5, 1., 0.05] >>> direct_confirmation_measure.aggregate_segment_sims(segment_sims, True, True) (0.4375, 0.36293077852394939, 4) >>> direct_confirmation_measure.aggregate_segment_sims(segment_sims, False, False) 0.4375 r)rmeanr stdlentuple)rrrrstatss rr r Ws> 7< D FE+ RVL))***( S&&'''5zzQ 8588E%LL8cg}t|j}|D]}g}|D]\} } || } || } || | f} |r@t| | fgg|d}| |z }|tj|t z z }n/| |z t z}| |z | |z z}tj||z }|||t||||S)a Compute log ratio measure for `segment_topics`. Parameters ---------- segmented_topics : list of lists of (int, int) Output from the :func:`~gensim.topic_coherence.segmentation.s_one_pre`, :func:`~gensim.topic_coherence.segmentation.s_one_one`. accumulator : :class:`~gensim.topic_coherence.text_analysis.InvertedIndexAccumulator` Word occurrence accumulator from :mod:`gensim.topic_coherence.probability_estimation`. normalize : bool, optional Details in the "Notes" section. with_std : bool, optional True to also include standard deviation across topic segment sets in addition to the mean coherence for each topic. with_support : bool, optional True to also include support across topic segments. The support is defined as the number of pairwise similarity comparisons were used to compute the overall topic coherence. Notes ----- If `normalize=False`: Calculate the log-ratio-measure, popularly known as **PMI** which is used by coherence measures such as `c_v`. This is defined as :math:`m_{lr}(S_i) = log \frac{P(W', W^{*}) + \epsilon}{P(W') * P(W^{*})}` If `normalize=True`: Calculate the normalized-log-ratio-measure, popularly knowns as **NPMI** which is used by coherence measures such as `c_v`. This is defined as :math:`m_{nlr}(S_i) = \frac{m_{lr}(S_i)}{-log(P(W', W^{*}) + \epsilon)}` Returns ------- list of float Log ratio measurements for each topic. Examples -------- .. sourcecode:: pycon >>> from gensim.topic_coherence import direct_confirmation_measure, text_analysis >>> from collections import namedtuple >>> >>> # Create dictionary >>> id2token = {1: 'test', 2: 'doc'} >>> token2id = {v: k for k, v in id2token.items()} >>> dictionary = namedtuple('Dictionary', 'token2id, id2token')(token2id, id2token) >>> >>> # Initialize segmented topics and accumulator >>> segmentation = [[(1, 2)]] >>> >>> accumulator = text_analysis.InvertedIndexAccumulator({1, 2}, dictionary) >>> accumulator._inverted_index = {0: {2, 3, 4}, 1: {3, 5}} >>> accumulator._num_docs = 5 >>> >>> # result should be ~ ln{(1 / 5) / [(3 / 5) * (2 / 5)]} = -0.182321557 >>> result = direct_confirmation_measure.log_ratio_measure(segmentation, accumulator)[0] r)rrlog_ratio_measurerrrr r )r r normalizerrrrrrrr w_prime_countrr numerator co_doc_probm_lr_i denominators rr$r$s0t[)**H^^ " ( (OGV'0M&v.L(&9N 9-&0A/B.C[QQRST ,x7 "rvkG.C'D'D&DE,h6'A ,x7L8r2shg  8 $ $ AAAAH&9&9&9RRRRRRRr"