cddZddlmZddlmZddlZddlmZddlZddl m Z ddl m Z ddl mZmZejeZd ZGd d eZGd d eZGddeZdZdZdZdZdZGddeZdS)z@ This module provides classes that deal with term similarities. )array)chainN)sqrt)sparse) corpus2csc)SaveLoad is_corpuszsparse documents must not contain any explicit zero entries and the similarity matrix S must satisfy x^T * S * x >= 0 for any nonzero bag-of-words vector x.c eZdZdZddZdZdS)TermSimilarityIndexa  Base class = common interface for retrieving the most similar terms for a given term. See Also -------- :class:`~gensim.similarities.termsim.SparseTermSimilarityMatrix` A sparse term similarity matrix built using a term similarity index. ct)aKGet most similar terms for a given term. Return the most similar terms for a given term along with their similarities. Parameters ---------- term : str The term for which we are retrieving `topn` most similar terms. topn : int, optional The maximum number of most similar terms to `term` that will be retrieved. Returns ------- iterable of (str, float) Most similar terms along with their similarities to `term`. Only terms distinct from `term` must be returned. )NotImplementedError)selftermtopns ;lib/python3.11/site-packages/gensim/similarities/termsim.py most_similarz TermSimilarityIndex.most_similar)s &"!cddt|D}|jjd|dS)Nz, c3 K|] }d|zV dS)z%s=%sN).0pairs r z.TermSimilarityIndex.__str__..?s&JJtGdNJJJJJJr<>)joinvarsitems __class____name__)rmemberss r__str__zTermSimilarityIndex.__str__>sN))JJtDzz7G7G7I7IJJJJJ>222GGG<._sQ4P4P#/8RbBh4P24P4P4P4P4P4Pr)zipranger.r/)rr5r__r3r4s ` rrz'UniformTermSimilarityIndex.most_similar^s"%eDkk4P4P4P4P37?4P4P4P#Q#Q - - B2t+, , , , , - -rN)r*r$)r!r%r&r'r0rrrrr)r)CsF,////------rr)c,eZdZdZdfd Zd dZxZS) WordEmbeddingSimilarityIndexaz Computes cosine similarities between word embeddings and retrieves most similar terms for a given term. Notes ----- By fitting the word embeddings to a vocabulary that you will be using, you can eliminate all out-of-vocabulary (OOV) words that you would otherwise receive from the `most_similar` method. In subword models such as fastText, this procedure will also infer word-vectors for words from your vocabulary that previously had no word-vector. >>> from gensim.test.utils import common_texts, datapath >>> from gensim.corpora import Dictionary >>> from gensim.models import FastText >>> from gensim.models.word2vec import LineSentence >>> from gensim.similarities import WordEmbeddingSimilarityIndex >>> >>> model = FastText(common_texts, vector_size=20, min_count=1) # train word-vectors on a corpus >>> different_corpus = LineSentence(datapath('lee_background.cor')) >>> dictionary = Dictionary(different_corpus) # construct a vocabulary on a different corpus >>> words = [word for word, count in dictionary.most_common()] >>> word_vectors = model.wv.vectors_for_all(words) # remove OOV word-vectors and infer word-vectors for new words >>> assert len(dictionary) == len(word_vectors) # all words from our vocabulary received their word-vectors >>> termsim_index = WordEmbeddingSimilarityIndex(word_vectors) Parameters ---------- keyedvectors : :class:`~gensim.models.keyedvectors.KeyedVectors` The word embeddings. threshold : float, optional Only embeddings more similar than `threshold` are considered when retrieving word embeddings closest to a given word embedding. exponent : float, optional Take the word embedding similarities larger than `threshold` to the power of `exponent`. kwargs : dict or None A dict with keyword arguments that will be passed to the :meth:`~gensim.models.keyedvectors.KeyedVectors.most_similar` method when retrieving the word embeddings closest to a given word embedding. See Also -------- :class:`~gensim.similarities.levenshtein.LevenshteinSimilarityIndex` Retrieve most similar terms for a given term using the Levenshtein distance. :class:`~gensim.similarities.termsim.SparseTermSimilarityMatrix` Build a term similarity matrix and compute the Soft Cosine Measure. @Nc||_||_||_|pi|_t t |dSr,) keyedvectors thresholdexponentkwargssuperr:r0)rr>r?r@rAr s rr0z%WordEmbeddingSimilarityIndex.__init__sG("  l  *D11::<<<<loggerdebugrrAr?r@)rr5rrr4 similaritys rrz)WordEmbeddingSimilarityIndex.most_similars T& & : LL92 > > > > >94,9bB4dbbVZVabbL". : :J.:z4=89999 : :r)r;r<Nr$)r!r%r&r'r0r __classcell__)r s@rr:r:ds\//`======::::::::rr:c|dkr tjS|dkr tjS|dkr tjStjS)aGet the shortest unsingned integer data-type required for representing values up to a given maximum value. Returns the shortest unsingned integer data-type required for representing values up to a given maximum value. Parameters ---------- max_value : int The maximum value we wish to represent. Returns ------- data-type The shortest unsigned integer data-type required for representing values up to a given maximum value. il)npuint8uint16uint32uint64) max_values r_shortest_uint_dtyperQsJ$4x U y U y 9rct|tsJJt}|dkrtdtd||fd}Jt } tdt| nmtj |dz ksJt j |} td t| t} tj dg|z| rtj || rttd td |tjus|tjurtd n0|tjurtd ntd|zfd} fd} ddlm} n#t&$rd} YnwxYwt)| | D]\}}||ddkrK|}|}|z }|dkr#fd|||Dng}t |}nt ||}|D]J\}}| |||rs| |||#| |||s| |||| |||Ktj| tjtj tjtj t3jff||f}tdd|z|dzz |S)zBuild a sparse term similarity matrix using a term similarity index. Returns ------- matrix : :class:`scipy.sparse.coo_matrix` The sparse term similarity matrix. NrzCDictionary provided to SparseTermSimilarityMatrix must not be emptyz5constructing a sparse term similarity matrix using %scZt|tr|^}}j|}| |fSr,) isinstancetupleidfs) term_index_term_idftfidfs rtfidf_sort_keyz&_create_source..tfidf_sort_keys7 j% ( ( ('NJ:j) :&&rz-iterating over %i columns in dictionary order)keyz)iterating over %i columns in tf-idf orderdtypeQfdz@Dtype %s is unsupported, use numpy.float16, float32, or float64.cr|t|zdkrdS|ksJ|krdSr||fvrdSdS)N?TF)abs) t1_indexr3rGassigned_cellscolumn_nonzero column_sumdominant nonzero_limit symmetrics r cell_fullz!_create_source..cell_fulls|   8,s:>#E 4h'=8888 ( #} 4 4  (H-? 4urc| |||xxdz cc< r||fr|xxt|z cc<dSdS)Nr\)appendaddre) rfr3rGrg column_bufferrhri data_bufferrj row_bufferrls rpopulate_buffersz(_create_source..populate_bufferssX&&&(###:&&&x   A%    5   (3 4 4 4  4 x C OO 3  4 4r)tqdmc|Sr,r)iterables r progress_barz$_create_source..progress_bar sOrrdcFg|]\}}|jvj||fSr)token2id)rrrGr.s r z"_create_source..sH    jz**   & 3   r)r)shapezAconstructed a sparse term similarity matrix with %0.06f%% densitygY@)rTr len ValueErrorrEinfor-keysmaxrVrQrKrzerossetfloat16float32float64ru ImportError enumerateror frombufferrOr coo_matrixgetnnz)indexr.rZrlrjrkr_ matrix_orderr[columnsnonzero_counter_dtypermrtrx column_numberrfr5 num_nonzeronum_rowsrrowsr3rGmatrixrgrqrhrirrrss ````` @@@@@@r_create_sourcers e0 1 1111 !!!z??Lq`^___ KKGOOO%$ '''''  O**++ CS\\RRRR5:,"22222**??? ?WNNN0??XqcL08MNNNN9Xl%888 sJ#JJM  eerz1eCjj "* eCjj [^ccddd444444444444-------      $-\\'-B-B#C#CAA x]+++-(((3 A     !$X.  ;. \ "    $)$6$6r$6$I$I     "   <,''DD,N;;;D$( A A Hjy8Z88  A  8Z@@@@Yx:>> A  8Z@@@  8Z@@@ A- 5999Kz;;;JM-ryAAAM   j--HIR^`lQm n n nF KKK  ,/1 Ms H H! H!c(|s|S|j||d}|dksJt|dkr#|dkr||j|z}t|}|}|dkr||z}|S)aiNormalize a dense vector after a change of basis. Parameters ---------- vector : 1xN ndarray A dense vector. matrix : NxN ndarray A change-of-basis matrix. normalization : {True, False, 'maintain'} Whether the vector will be L2-normalized (True; corresponds to the soft cosine measure), maintain its L2-norm during the change of basis ('maintain'; corresponds to query expansion with partial membership), or kept as-is (False; corresponds to query expansion). Returns ------- vector : ndarray The normalized dense vector. rrr;maintain)Tdot#NON_NEGATIVE_NORM_ASSERTION_MESSAGEr)vectorr normalization vector_normnormalized_vectors r_normalize_dense_vectorr:s*  (,,v&&**62248K # BBBBBB ",{S'8,vx||F+++ {##KS)[( rc|s|Stj|j||jdj}|dksJt |dkr;|tj|j|jdjz}tj|}tj|d|z }tj|}|S)atNormalize a dense corpus after a change of basis. Parameters ---------- corpus : MxN ndarray A dense corpus. matrix : NxN ndarray A change-of-basis matrix. normalization : {True, False, 'maintain'} Whether the vector will be L2-normalized (True; corresponds to the soft cosine measure), maintain its L2-norm during the change of basis ('maintain'; corresponds to query expansion with partial membership), or kept as-is (False; corresponds to query expansion). Returns ------- normalized_corpus : ndarray The normalized dense corpus. r\axisr;rrd) rKmultiplyrrsumminrr nan_to_numcorpusrr corpus_normnormalized_corpuss r_normalize_dense_corpusr_s*  +fhll622FH==AAqAIIKK ??   #HH%HHHH "Er{68VX66:::BBDD '+&&K FC+,=>> &788 rc|s|S|j||jdj}|dksJt |dkr;||j|jdjz}t j|}|tj d|z }d||t j k<|S)aNormalize a sparse corpus after a change of basis. Parameters ---------- corpus : MxN :class:`scipy.sparse.csc_matrix` A sparse corpus. matrix : NxN :class:`scipy.sparse.csc_matrix` A change-of-basis matrix. normalization : {True, False, 'maintain'} Whether the vector will be L2-normalized (True; corresponds to the soft cosine measure), maintain its L2-norm during the change of basis ('maintain'; corresponds to query expansion with partial membership), or kept as-is (False; corresponds to query expansion). Returns ------- normalized_corpus : :class:`scipy.sparse.csc_matrix` The normalized sparse corpus. r\rr;rrdr) rrrrrrrKrr csr_matrixinfrs r_normalize_sparse_corpusrs*  (,,v&&//99==1=EEGK ??   #HH%HHHH "Avx((2266A6>>@@ '+&&K(9# :K(L(LMM56'2612 rc8eZdZdZdddddejfdZd dZdS) SparseTermSimilarityMatrixa[ Builds a sparse term similarity matrix using a term similarity index. Examples -------- >>> from gensim.test.utils import common_texts as corpus, datapath >>> from gensim.corpora import Dictionary >>> from gensim.models import Word2Vec >>> from gensim.similarities import SoftCosineSimilarity, SparseTermSimilarityMatrix, WordEmbeddingSimilarityIndex >>> from gensim.similarities.index import AnnoyIndexer >>> >>> model_corpus_file = datapath('lee_background.cor') >>> model = Word2Vec(corpus_file=model_corpus_file, vector_size=20, min_count=1) # train word-vectors >>> >>> dictionary = Dictionary(corpus) >>> tfidf = TfidfModel(dictionary=dictionary) >>> words = [word for word, count in dictionary.most_common()] >>> word_vectors = model.wv.vectors_for_all(words, allow_inference=False) # produce vectors for words in corpus >>> >>> indexer = AnnoyIndexer(word_vectors, num_trees=2) # use Annoy for faster word similarity lookups >>> termsim_index = WordEmbeddingSimilarityIndex(word_vectors, kwargs={'indexer': indexer}) >>> similarity_matrix = SparseTermSimilarityMatrix(termsim_index, dictionary, tfidf) # compute word similarities >>> >>> tfidf_corpus = tfidf[[dictionary.doc2bow(document) for document in common_texts]] >>> docsim_index = SoftCosineSimilarity(tfidf_corpus, similarity_matrix, num_best=10) # index tfidf_corpus >>> >>> query = 'graph trees computer'.split() # make a query >>> sims = docsim_index[dictionary.doc2bow(query)] # find the ten closest documents from tfidf_corpus Check out `the Gallery `_ for more examples. Parameters ---------- source : :class:`~gensim.similarities.termsim.TermSimilarityIndex` or :class:`scipy.sparse.spmatrix` The source of the term similarity. Either a term similarity index that will be used for building the term similarity matrix, or an existing sparse term similarity matrix that will be encapsulated and stored in the matrix attribute. When a matrix is specified as the source, any other parameters will be ignored. dictionary : :class:`~gensim.corpora.dictionary.Dictionary` or None, optional A dictionary that specifies a mapping between terms and the indices of rows and columns of the resulting term similarity matrix. The dictionary may only be None when source is a :class:`scipy.sparse.spmatrix`. tfidf : :class:`gensim.models.tfidfmodel.TfidfModel` or None, optional A model that specifies the relative importance of the terms in the dictionary. The columns of the term similarity matrix will be build in a decreasing order of importance of terms, or in the order of term identifiers if None. symmetric : bool, optional Whether the symmetry of the term similarity matrix will be enforced. Symmetry is a necessary precondition for positive definiteness, which is necessary if you later wish to derive a unique change-of-basis matrix from the term similarity matrix using Cholesky factorization. Setting symmetric to False will significantly reduce memory usage during matrix construction. dominant: bool, optional Whether the strict column diagonal dominance of the term similarity matrix will be enforced. Strict diagonal dominance and symmetry are sufficient preconditions for positive definiteness, which is necessary if you later wish to derive a change-of-basis matrix from the term similarity matrix using Cholesky factorization. nonzero_limit : int or None, optional The maximum number of non-zero elements outside the diagonal in a single column of the sparse term similarity matrix. If None, then no limit will be imposed. dtype : numpy.dtype, optional The data type of the sparse term similarity matrix. Attributes ---------- matrix : :class:`scipy.sparse.csc_matrix` The encapsulated sparse term similarity matrix. Raises ------ ValueError If `dictionary` is empty. See Also -------- :class:`~gensim.similarities.docsim.SoftCosineSimilarity` A document similarity index using the soft cosine similarity over the term similarity matrix. :class:`~gensim.similarities.termsim.LevenshteinSimilarityIndex` A term similarity index that computes Levenshtein similarities between terms. :class:`~gensim.similarities.termsim.WordEmbeddingSimilarityIndex` A term similarity index that computes cosine similarities between word embeddings. NTFdctj|s*|}|||||||f} t| }tj|sJ||_dSr,)rissparsertocscr) rsourcer.rZrlrjrkr_rargss rr0z#SparseTermSimilarityMatrix.__init__scv&& +E:ui=RWXD#T*F?6** * * *llnn rFFc rs|jjdS|\}}d}||vr"td|||vr"td|t \}t \}|sM|sJt t tjttt} |jj} tjfd| D| tjfd| D| |j| dddf| f } t| |t| |j| } |dur|durtj| d d } | d S|r|s|r|sc||}}||}}d} nd } |jj} t#g|jjd | j|j}tjt|d} ~t tjfd| D| t#|jjd | | ddf |j| dddf| f } t| |t)| || } |dur|durtj| d d } | r| j} | S|jj} t#|rng|jjd | t#|rng|jjd | |j} t+| |t+| |j| } |dur$|dur tj| jd d | _| S)uVGet the inner product(s) between real vectors / corpora X and Y. Return the inner product(s) between real vectors / corpora vec1 and vec2 expressed in a non-orthogonal normalized basis, where the dot product between the basis vectors is given by the sparse term similarity matrix. Parameters ---------- vec1 : list of (int, float) or iterable of list of (int, float) A query vector / corpus in the sparse bag-of-words format. vec2 : list of (int, float) or iterable of list of (int, float) A document vector / corpus in the sparse bag-of-words format. normalized : tuple of {True, False, 'maintain'}, optional First/second value specifies whether the query/document vectors in the inner product will be L2-normalized (True; corresponds to the soft cosine measure), maintain their L2-norm during change of basis ('maintain'; corresponds to query expansion with partial membership), or kept as-is (False; corresponds to query expansion; default). Returns ------- `self.matrix.dtype`, `scipy.sparse.csr_matrix`, or :class:`numpy.matrix` The inner product(s) between `X` and `Y`. References ---------- The soft cosine measure was perhaps first described by [sidorovetal14]_. Further notes on the efficient implementation of the soft cosine measure are described by [novotny18]_. .. [sidorovetal14] Grigori Sidorov et al., "Soft Similarity and Soft Cosine Measure: Similarity of Features in Vector Space Model", 2014, http://www.cys.cic.ipn.mx/ojs/index.php/CyS/article/view/2043/1921. .. [novotny18] Vít Novotný, "Implementation Notes for the Soft Cosine Measure", 2018, http://dx.doi.org/10.1145/3269206.3269317. r;)TFrz${} is not a valid value of normalizec,g|]}|vr|ndSrrriXs rr{z.>*FFFA!q&/!A$$aFFFrr^c,g|]}|vr|ndSrr)rrYs rr{z.?rrNTgrdrFr) num_termsr_r\c,g|]}|vr|ndSrrrs rr{z.Yrr)rr_typerformatr dictrKrr-rrtodenserrrcliprr|nonzerorrdata)rrr normalized normalized_X normalized_Yvalid_normalized_values is_corpus_X is_corpus_Y word_indicesr_rresult transposed expanded_Xs `` r inner_productz(SparseTermSimilarityMatrix.inner_productsJ / /;$))#.. .%/" l"; 6 6 ZCJJ<XXYY Y 6 6 ZCJJ<XXYY Y"1 Q"1 Q< ;< QAQA8F3uQ{{+;+;$<$<==LK%EFFFFFFFeTTTAFFFFFFFeTTTA[aaag!6 !DEMMOOF'6<@@A'6<@@ASWWV__((++Ft# 4 (< 4s33$< + K+  #; #!1+6 [ -9 (< > gfk4== Mr)r)r!r%r&r'rKrr0rrrrrrsbRRf+/ddUZRZ % % % %oooooorr)r'r itertoolsrloggingmathrnumpyrKscipyrgensim.matutilsr gensim.utilsrr getLoggerr!rErr r)r:rQrrrrrrrrrs&&&&&&,,,,,,,,  8 $ $&$!=!=!=!=!=(!=!=!=H-----!4---B?:?:?:?:?:#6?:?:?:D6vvvr"""J!!!H!!!HNNNNNNNNNNr