cWdZddlmZddlZddlZddlmZddlZddl Z ddl m Z ddl mZmZddlmZddlmZejeZd Zd9d Zdejdddfd Zd Zd:dZdZd;dZd;dZd;dZ GddZ!dZ"d;dZ#e#Z$d;dZ%dej&fdZ'GddZ(GddZ)d Z*d!Z+dd+Z4d>d,Z5d>d-Z6d.Z7d/Z8d0Z9 dd1l:m;Z;m$r d2Z;d3Z$rejCwxYw)?zMath helper functions.)with_statementN)utils)entropy)get_blas_funcstriu)get_lapack_funcs)psic2t|f|fdS)aHelper for getting the appropriate BLAS function, using :func:`scipy.linalg.get_blas_funcs`. Parameters ---------- name : str Name(s) of BLAS functions, without the type prefix. ndarray : numpy.ndarray Arrays can be given to determine optimal prefix of BLAS routines. Returns ------- object BLAS function for the needed operation on the given data type. r)r)namendarrays /lib/python3.11/site-packages/gensim/matutils.pyblasrs 4'G: . .q 11Fc|tj|}||j}|dkrgS|r| }||jksttdstj|d|Stj||d|}|tj||S)aEfficiently calculate indices of the `topn` smallest elements in array `x`. Parameters ---------- x : array_like Array to get the smallest element indices from. topn : int, optional Number of indices of the smallest (greatest) elements to be returned. If not given, indices of all elements will be returned in ascending (descending) order. reverse : bool, optional Return the `topn` greatest elements in descending order, instead of smallest elements in ascending order? Returns ------- numpy.ndarray Array of `topn` indices that sort the array in the requested order. Nr argpartition)npasarraysizehasattrargsortrtake)xtopnreverse most_extremes r rr/s( 1 A v qy  B qv~$WR88$z!}}UdU##?1d++ETE2L   RZ|(<(<== > >>rc ||j}||j}||j}n#t$rYnwxYw|rtd|||ddg}}t j|ft j}t j|f|} t|D]q\} } |r%| |zdkrtd| ||t| z} | r t| nggf\||| <| || <| | | }r||ks Jdtj| ||f||f|} n%dggdgf\}} }}t|D]\} } |r$| |zdkrtd| | r t| nggf\}}||| ||t| z }| |||rt#|d znd}t|d z }t j| |} t j|}tj| ||f||f|} | S) a*Convert a streamed corpus in bag-of-words format into a sparse matrix `scipy.sparse.csc_matrix`, with documents as columns. Notes ----- If the number of terms, documents and non-zero elements is known, you can pass them here as parameters and a (much) more memory efficient code path will be taken. Parameters ---------- corpus : iterable of iterable of (int, number) Input corpus in BoW format num_terms : int, optional Number of terms in `corpus`. If provided, the `corpus.num_terms` attribute (if any) will be ignored. dtype : data-type, optional Data type of output CSC matrix. num_docs : int, optional Number of documents in `corpus`. If provided, the `corpus.num_docs` attribute (in any) will be ignored. num_nnz : int, optional Number of non-zero elements in `corpus`. If provided, the `corpus.num_nnz` attribute (if any) will be ignored. printprogress : int, optional Log a progress message at INFO level once every `printprogress` documents. 0 to turn off progress logging. Returns ------- scipy.sparse.csc_matrix `corpus` converted into a sparse CSC matrix. See Also -------- :class:`~gensim.matutils.Sparse2Corpus` Convert sparse format to Gensim corpus format. Nz"creating sparse matrix from corpusrdtypezPROGRESS: at document #%i/%iz:mismatch between supplied and computed number of non-zeros)shaperzPROGRESS: at document #%i) num_termsnum_docsnum_nnzAttributeErrorloggerinforemptyint32 enumeratelenzipappendscipysparse csc_matrixextendmaxr)corpusr!rr"r# printprogressposnowindptrindicesdatadocnodocposnextresult doc_indicesdoc_datas r corpus2cscr>QsF    )(I  'H  %nG      : 8999"l"l'"lQC(G:RX666x %000#F++  JE3 M!6!!; M :E8LLLs3xx'GKN>\c3iiUWY[T\ ;GFGO $d67?&; MM' " " "FF ^^"^^^^(($)@T\H]ej(kk*+BQC&w#F++ # #JE3 @!6!!; @ 7???25$BCII2r( !K NN; ' ' ' KK ! ! ! s3xx G MM' " " " "  ;,3:G q((Iv;;?z$e,,,*W%%(($)@T\H]ej(kk Ms  ++c|dkrd}|dkrd}|j\}}tj|tj||fgtj|||zfggS)azAdd additional rows/columns to `mat`. The new rows/columns will be initialized with zeros. Parameters ---------- mat : numpy.ndarray Input 2D matrix padrow : int Number of additional rows padcol : int Number of additional columns Returns ------- numpy.matrixlib.defmatrix.matrix Matrix with needed padding. r)rrblockzeros)matpadrowpadcolrowscolss r padrGsy$z zJD$ 8 bhf~&&' 64&=) * *+  rCcFtj|tjtj|jz}tj||ztj}|jj |z}||||z | ||S)aGet array aligned at `align` byte boundary in memory. Parameters ---------- shape : int or (int, int) Shape of array. dtype : data-type Data type of array. order : {'C', 'F'}, optional Whether to store multidimensional data in C- or Fortran-contiguous (row- or column-wise) order in memory. align : int, optional Boundary for alignment in bytes. Returns ------- numpy.ndarray Aligned array. r)order) rprodint64ritemsizerAuint8ctypesr7viewreshape)rrrKalignnbytesbuffer start_indexs r zeros_alignedrWs(WU"( + + +bhuoo.F FF XfunBH 5 5 5F=%%-K +{V33 4 9 9% @ @ H HV[ H \ \\rct|tjr |jdkptj|S)zCheck whether `m` is a 2D `numpy.ndarray` or `scipy.sparse` matrix. Parameters ---------- m : object Object to check. Returns ------- bool Is `m` a 2D `numpy.ndarray` or `scipy.sparse` matrix. ) isinstancerr ndimr-r.issparse)ms r ismatrixr^s6 a $ $ 41 P 8M8Ma8P8PPr& .>ct|tjrt|Stj|rt|Sfd|DS)aConvert a numpy.ndarray or `scipy.sparse` vector into the Gensim bag-of-words format. Parameters ---------- vec : {`numpy.ndarray`, `scipy.sparse`} Input vector eps : float, optional Value used for threshold, all coordinates less than `eps` will not be presented in result. Returns ------- list of (int, float) Vector in BoW format. cg|];\}}tj|kt|t|fzany2sparse..s@ I I Igc2r S8H ISXXuRyy ! I I Ir)rZrr dense2vecr-r.r\ scipy2sparsevecrjs `r any2sparserpsk #rz""#c""" |S!!&C%%% I I I I I I IIrc &tj|std|z|dkrtjgS|jddkrt t|j|d}|j ||j |}}tj||dt|gfSg}g}dg}t|} t|jdD]} | | } | | } t | j|d}| j || j |}}||||||dtt||zt!j|}t!j|}tjj|||f|jdt!j|dzfS)aGet the 'topn' elements of the greatest magnitude (absolute value) from a `scipy.sparse` vector or matrix. Parameters ---------- matrix : `scipy.sparse` Input vector or matrix (1D or 2D sparse array). topn : int Number of greatest elements, in absolute value, to return. eps : float Ignored. Returns ------- `scipy.sparse.csr.csr_matrix` Clipped matrix. z"'%s' is not a scipy sparse vector.rr Tr)r)r-r.r\ ValueError csr_matrixrrrdr7r6rr*rangegetrowr,minr concatenateravelcsrr1) matrixrrjbiggestr6r7matrix_indices matrix_data matrix_indptr matrix_absivv_abss r scipy2scipy_clippedrsI$ <  ( (H=FGGG qy+|&&r*** |A! #fk**D$???++G44fk6F6Fw6O6O|&&g3w<<7H'IJJJ  [[ v|A'' N NA a  A%%a((Eej$===GINN733QV[[5I5ITG   t $ $ $  ! !' * * *  r!2SWt5L5L!L M M M M77==??n[117799 |** .- 8<?BF>$:$:Q$>?+   rc|}|jddksJfdt|j|jDS)auConvert a scipy.sparse vector into the Gensim bag-of-words format. Parameters ---------- vec : `scipy.sparse` Sparse vector. eps : float, optional Value used for threshold, all coordinates less than `eps` will not be presented in result. Returns ------- list of (int, float) Vector in Gensim bag-of-words format. rr cg|];\}}tj|kt|t|f.OsF c c cxsCQSQWX[Q\Q\_bQb cSXXuSzz " c c cr)tocsrrr+r6r7rns `r rmrm<sQ" ))++C 9Q<1  c c c c3s{CH3M3M c c ccrc$eZdZdZdZdZdZdS) Scipy2CorpuszConvert a sequence of dense/sparse vectors into a streamed Gensim corpus object. See Also -------- :func:`~gensim.matutils.corpus2csc` Convert corpus in Gensim format to `scipy.sparse.csc` matrix. c||_dS)z Parameters ---------- vecs : iterable of {`numpy.ndarray`, `scipy.sparse`} Input vectors. N)vecs)selfrs r __init__zScipy2Corpus.__init__[s rc#K|jD]?}t|tjrt |V.t |V@dSN)rrZrr full2sparserm)rros r __iter__zScipy2Corpus.__iter__fsc9 ( (C#rz** (!#&&&&&&"3''''''  ( (rc*t|jSr)r*rrs r __len__zScipy2Corpus.__len__ms49~~rN__name__ __module__ __qualname____doc__rrrrbrr rrRsK   (((rrctj|tj}d|D}t|}t ||t |<|S)a_Convert a document in Gensim bag-of-words format into a dense numpy array. Parameters ---------- doc : list of (int, number) Document in BoW format. length : int Vector dimensionality. This cannot be inferred from the BoW, and you must supply it explicitly. This is typically the vocabulary size or number of topics, depending on how you created `doc`. Returns ------- numpy.ndarray Dense numpy vector for `doc`. See Also -------- :func:`~gensim.matutils.full2sparse` Convert dense array to gensim bag-of-words format. rc3XK|]%\}}t|t|fV&dSr)rerf)rgid_val_s r zsparse2full..s7 : :{TCHHeDkk " : : : : : :r)rrAfloat32dictlistvalues)r9lengthr;s r sparse2fullrqs],XfBJ / / /F : :c : : :C s))CSZZ\\**F499 Mrctj|t}tjt ||kd}t t |||S)aOConvert a dense numpy array into the Gensim bag-of-words format. Parameters ---------- vec : numpy.ndarray Dense input vector. eps : float Feature weight threshold value. Features with `abs(weight) < eps` are considered sparse and won't be included in the BOW result. Returns ------- list of (int, float) BoW format of `vec`, with near-zero values omitted (sparse vector). See Also -------- :func:`~gensim.matutils.sparse2full` Convert a document in Gensim bag-of-words format into a dense numpy array. rr)rrrfnonzerordrr+r)rorjnnzs r rrsX, *S & & &C *SXX^ $ $Q 'C C#'' ( ((rc|dkrgStj|t}tjt ||kd}|t t |||d}tt|||S)aLike :func:`~gensim.matutils.full2sparse`, but only return the `topn` elements of the greatest magnitude (abs). This is more efficient that sorting a vector and then taking the greatest values, especially where `len(vec) >> topn`. Parameters ---------- vec : numpy.ndarray Input dense vector topn : int Number of greatest (abs) elements that will be presented in result. eps : float Threshold value, if coordinate in `vec` < eps, this will not be presented in result. Returns ------- list of (int, float) Clipped vector in BoW format. See Also -------- :func:`~gensim.matutils.full2sparse` Convert dense array to gensim bag-of-words format. rrTrr) rrrfrrdrrrr+)rorrjrr}s r full2sparse_clippedrs8 qy *S & & &C *SXX^ $ $Q 'Chhws3xx}}S114FFFGGG GSXXg..// 0 00rc|Rdtj|f|}}t|D]\}}t||dd|f<|dz|ksJn tjfd|D}||S)aConvert corpus into a dense numpy 2D array, with documents as columns. Parameters ---------- corpus : iterable of iterable of (int, number) Input corpus in the Gensim bag-of-words format. num_terms : int Number of terms in the dictionary. X-axis of the resulting matrix. num_docs : int, optional Number of documents in the corpus. If provided, a slightly more memory-efficient code path is taken. Y-axis of the resulting matrix. dtype : data-type, optional Data type of the output matrix. Returns ------- numpy.ndarray Dense 2D array that presents `corpus`. See Also -------- :class:`~gensim.matutils.Dense2Corpus` Convert dense matrix to Gensim corpus format. Nrsrr c0g|]}t|Srb)r)rgr9r!s r rkz corpus2dense..s#!P!P!P#+c9"="=!P!P!Pr)rr'r)r column_stackastype)r2r!r"rr8r;r9s ` r corpus2densers4 RBHi%:%HHHv#F++ ; ;JE3*3 ::F111e8  qyH$$$$$!P!P!P!P!P!P!PQQ ==  rc&eZdZdZddZdZdZdS) Dense2CorpusaTreat dense numpy array as a streamed Gensim corpus in the bag-of-words format. Notes ----- No data copy is made (changes to the underlying matrix imply changes in the streamed corpus). See Also -------- :func:`~gensim.matutils.corpus2dense` Convert Gensim corpus to dense matrix. :class:`~gensim.matutils.Sparse2Corpus` Convert sparse matrix to Gensim corpus format. Tc4|r|j|_dS||_dS)z Parameters ---------- dense : numpy.ndarray Corpus in dense format. documents_columns : bool, optional Documents in `dense` represented as columns, as opposed to rows? N)Tdense)rrdocuments_columnss r rzDense2Corpus.__init__ s$  DJJJDJJJrc#JK|jD]}t|jVdS)zIterate over the corpus. Yields ------ list of (int, float) Document in BoW format. N)rrflat)rr9s r rzDense2Corpus.__iter__s<: ( (Cch'' ' ' ' ' ( (rc*t|jSr)r*rrs r rzDense2Corpus.__len__)s4:rNTrrbrr rrsP   ( ( (rrc,eZdZdZddZdZdZdZdS) Sparse2Corpusa,Convert a matrix in scipy.sparse format into a streaming Gensim corpus. See Also -------- :func:`~gensim.matutils.corpus2csc` Convert gensim corpus format to `scipy.sparse.csc` matrix :class:`~gensim.matutils.Dense2Corpus` Convert dense matrix to gensim corpus. Tc||r||_dS|j|_dS)z Parameters ---------- sparse : `scipy.sparse` Corpus scipy sparse format documents_columns : bool, optional Documents will be column? N)tocscr.rr)rr.rs r rzSparse2Corpus.__init__8s5  + ,,..DKKK ,,..*DKKKrc #Kt|jj|jjddD]H\}}tt|jj|||jj||VIdS)zj Yields ------ list of (int, float) Document in BoW format. r N)r+r.r5rr6r7)rindprevindnows r rzSparse2Corpus.__iter__Hs #4;#5t{7I!""7MNN c cOGVs4;.wv~> @PQXY_Q_@`aabb b b b b c crc&|jjdS)Nr )r.rrs r rzSparse2Corpus.__len__Ts{ ##rcj|j}t|tr^|jj|}|jj|dz}t t |j|||j||S|jtddd|f}t|S)a Retrieve a document vector or subset from the corpus by key. Parameters ---------- key: int, ellipsis, slice, iterable object Index of the document retrieve. Less commonly, the key can also be a slice, ellipsis, or an iterable to retrieve multiple documents. Returns ------- list of (int, number), Sparse2Corpus Document in BoW format when `key` is an integer. Otherwise :class:`~gensim.matutils.Sparse2Corpus`. r N) r.rZrer5rr+r6r7 __getitem__slicer)rkeyr.iprevinows r rzSparse2Corpus.__getitem__Ws  c3   RK&s+E;%cAg.DFN5:6 E$J8OPPQQ Q((%dD*A*A3)GHHV$$$rNr)rrrrrrrrrbrr rr-sb  ++++ c c c$$$%%%%%rrct|dkrdSdtjtd|Dz}|dks Jd|S)zCalculate L2 (euclidean) length of a vector. Parameters ---------- vec : list of (int, number) Input vector in sparse bag-of-words format. Returns ------- float Length of `vec`. r?c3&K|] \}}|dzV dSrYNrbrg_rs r rzveclen..s* : :FAsa : : : : : :r;sparse documents must not contain any explicit zero entries)r*mathsqrtsumrors r veclenrqsd 3xx1}s 49S : :c : : :::;; ;F C<VVVVVV MrcJdkrfd|DSt|S)a#Normalize a vector in L2 (Euclidean unit norm). Parameters ---------- vec : list of (int, number) Input vector in BoW format. length : float Length of vector Returns ------- list of (int, number) L2-normalized vector in BoW format. rc$g|] \}}||z f Srbrb)rgtermidrrs r rkz&ret_normalized_vec..s&>>>;63v&>>>r)rrs `r ret_normalized_vecrs7 }>>>>#>>>>Cyyrr c,d}t|jdkrtj|}|tjt|dzz |z }tjtj||z}tj||z }||z}n|dkrtj|d}|tj|jddzz |z }tjtj||ddtjfzd}tj||z }||ddtjfz }nA|dkr)t|j }|dj |dfStd|z||fS)NY@r rrz'%s' is not a supported axis) r*rrr1logrexpnewaxisret_log_normalize_vecrrt)roaxislog_maxmax_val log_shifttotlog_normks r rrsqG 39~~D&++bfSXX^444w> fRVC)O,,--6#;;* x 19 DfS!nnG"&1);"<"< >??CCCvc{{Y.HBJ//CC QY D%ce,,AQ461Q4< ;dBCC C =rnrm2rscall2cd}||vrtd|d|dtj|r|}|dkr+t jt j|j}|dkr.t j t j|jdz}|dkr|j }|d krKt j |j t j r|t}||z}|r||fS|S|r|d fS|St!|t jr|dkr&t jt j|}|dkr|jd krd }nt'|}|dkrt j|}|d krt j |j t j r|t}|r-t+d |z ||j |fSt+d |z ||j S|r|d fS|S t-t/|}n#t0$r |r|d fcYS|cYSwxYwt!|t2t4frt7|dkr|dkr&tt d |D}|dkr.d t9j t d |Dz}|dkrd t7|z}|d ks Jd|rt;|||fSt;||Std)auScale a vector to unit length. Parameters ---------- vec : {numpy.ndarray, scipy.sparse, list of (int, float)} Input vector in any format norm : {'l1', 'l2', 'unique'}, optional Metric to normalize in. return_norm : bool, optional Return the length of vector `vec`, in addition to the normalized vector itself? Returns ------- numpy.ndarray, scipy.sparse, list of (int, float)} Normalized vector in same format as `vec`. float Length of `vec` before normalization, if `return_norm` is set. Notes ----- Zero-vector will be unchanged. )l1runique'z9' is not a supported norm. Currently supported norms are .rrrYrrrrc3:K|]\}}t|VdSrrdrs r rzunitvec.. s,::FAss3xx::::::rc3&K|] \}}|dzV dSrrbrs r rzunitvec.. s*(D(Dfa(D(D(D(D(D(Drrzunknown input type)rtr-r.r\rrrrdr7rr issubdtyperintegerrrfrZr r blas_nrm2 count_nonzero blas_scalnextiter StopIterationtuplerr*rr)ronorm return_normsupported_normsrfirstrs r unitvecrs0-O ?"vj^b^b^bdsdsdstuuu |S!!iikk 4< .VBF38,,--F 4< 4WRVCHM2233F 8  WF C< }SY 33 (jj'' 6MC F{"  Cx #rz"" 4< )VBF3KK((F 4< (x1} ("3 8  +%c**F C< }SY 33 (jj'' F vs33::39EEvMM vs33::39EEE Cx T#YY   8OOOJJJ  %%'' /CJJ!O / 4< <3::c:::::;;F 4< F49S(D(D(D(D(D%D%DEEEF 8  $3s88^F|ZZZZZZ  3%c622F: :%c622 2-...sI##I:5I:9I:c2t|tc}|rsdSdtjtd|Dz}dtjtdDz}|dkr|dks Jdt t |kr|c}tfd|D}|||zz}|S)aGet cosine similarity between two sparse vectors. Cosine similarity is a number between `<-1.0, 1.0>`, higher means more similar. Parameters ---------- vec1 : list of (int, float) Vector in BoW format. vec2 : list of (int, float) Vector in BoW format. Returns ------- float Cosine similarity between `vec1` and `vec2`. rrc3 K|] }||zV dSrrbrgrs r rzcossim..-&!E!E#)!E!E!E!E!E!Erc3 K|] }||zV dSrrbrs r rzcossim...rrrc3PK|] \}}||dzV!dS)rN)get)rgindexvaluevec2s r rzcossim..2s:OO,%%---OOOOOOr)rrrrrr*items)vec1r vec1lenvec2lenr;s ` r cossimrs$dT$ZZJD$ tsDIc!E!Et{{}}!E!E!EEEFFFGDIc!E!Et{{}}!E!E!EEEFFFG S=iWs]ii,iiii 4yy3t99 4 d OOOO$**,,OOO O OF gF Mrc.tj|r&|} |d\}}t |t |fn$#t$rYdSttf$rYdSwxYwdS)zChecks if a vector is in the sparse Gensim bag-of-words format. Parameters ---------- vec : object Object to check. Returns ------- bool Is `vec` in BoW format. rTF) r-r.r\todensetolistrerf IndexErrorrt TypeError)rorrs r isbowr7s |S!!%kkmm""$$F T C%++ tt  "uu 4s)A11 B>BBcftj|r|}tj|r|}t |rt |rt|$t ||}t ||}||fSt t|t|}t ||}t ||}||fSt|dkr|d}t|dkr|d}||fS)Nr r)r-r.r\toarrayrrr1r*)r r  num_featuresdense1dense2max_lens r _convert_vecrQs |T""||~~ |T""||~~ T{{uT{{  " |44F |44F6> !#d))SYY//G w//F w//F6> ! t99> 7D t99> 7DTzrcLt|||\}}t||S)uQCalculate Kullback-Leibler distance between two probability distributions using `scipy.stats.entropy`. Parameters ---------- vec1 : {scipy.sparse, numpy.ndarray, list of (int, float)} Distribution vector. vec2 : {scipy.sparse, numpy.ndarray, list of (int, float)} Distribution vector. num_features : int, optional Number of features in the vectors. Returns ------- float Kullback-Leibler distance between `vec1` and `vec2`. Value in range [0, +∞) where values closer to 0 mean less distance (higher similarity). rrr)r r rs r kullback_leiblerr js,&dD|DDDJD$ 4  rct|||\}}d||zz}dt||t||zzS)aZCalculate Jensen-Shannon distance between two probability distributions using `scipy.stats.entropy`. Parameters ---------- vec1 : {scipy.sparse, numpy.ndarray, list of (int, float)} Distribution vector. vec2 : {scipy.sparse, numpy.ndarray, list of (int, float)} Distribution vector. num_features : int, optional Number of features in the vectors. Returns ------- float Jensen-Shannon distance between `vec1` and `vec2`. Notes ----- This is a symmetric and finite "version" of :func:`gensim.matutils.kullback_leibler`. r?r)r r ravg_vecs r jensen_shannonr$sN,dD|DDDJD$TD[!G '$((74+A+AA BBrctjrtjrt rt rt t ct ttz}tj dtfd|Dz}|Stj dtj tj z dz z}|S)aCalculate Hellinger distance between two probability distributions. Parameters ---------- vec1 : {scipy.sparse, numpy.ndarray, list of (int, float)} Distribution vector. vec2 : {scipy.sparse, numpy.ndarray, list of (int, float)} Distribution vector. Returns ------- float Hellinger distance between `vec1` and `vec2`. Value in range `[0, 1]`, where 0 is min distance (max similarity) and 1 is max distance (min similarity). r"c3K|]X}tj|dtj|dz dzVYdS)rrYN)rrr)rgrr r s r rzhellinger..s_nn]brwtxxs3344rwtxxs?S?S7T7TTWXXnnnnnnrrY) r-r.r\rrrsetrkeysrrr)r r r6sims`` r hellingerr*s4" |T""||~~ |T""||~~ T{{ uT{{ $ZZd dd499;;''$tyy{{*;*;;<<g #nnnnnfmnnnnn n   gcbgdmmbgdmm;a?DDFFFGG rc tj|r|}tj|r|}t |rt |rt d|Dt d|Dz}t |t |}}d}|D],\}}|t|| |dz }-dt|t|z z St|tj r|}t|tj r|}t|}t|}||z}||z}dtt!|tt!|z z S)aCalculate Jaccard distance between two vectors. Parameters ---------- vec1 : {scipy.sparse, numpy.ndarray, list of (int, float)} Distribution vector. vec2 : {scipy.sparse, numpy.ndarray, list of (int, float)} Distribution vector. Returns ------- float Jaccard distance between `vec1` and `vec2`. Value in range `[0, 1]`, where 0 is min distance (max similarity) and 1 is max distance (min similarity). c3 K|] \}}|V dSrrbrgrweights r rzjaccard..s&33{sFF333333rc3 K|] \}}|V dSrrbr-s r rzjaccard..s&9Y9Y[S&&9Y9Y9Y9Y9Y9Yrrr )r-r.r\rrrrr rxrrfrZrr rr'r*)r r union intersection feature_idfeature_weights r jaccardr4s& |T""||~~ |T""||~~ T{{@uT{{@33d33333c9Y9YTX9Y9Y9Y6Y6YY$ZZdd *.**,, K K &J CS0I0IJJ JLL5&&u555 dBJ ' ' !;;==D dBJ ' ' !;;==D4yy4yyd{ t 5\**++eCJJ.?.????rct||z}|dkrdSdtt||zt|z z S)a]Calculate Jaccard distance between two sets. Parameters ---------- set1 : set Input set. set2 : set Input set. Returns ------- float Jaccard distance between `set1` and `set2`. Value in range `[0, 1]`, where 0 is min distance (max similarity) and 1 is max distance (min similarity). rr)r*rf)set1set2union_cardinalitys r jaccard_distancer9sU"D4K((Ar c$+&&''%0A*B*BB BBr) logsumexpmean_absolute_differencedirichlet_expectationctj|}tjtjtj||z }||z }|S)aLog of sum of exponentials. Parameters ---------- x : numpy.ndarray Input 2d matrix. Returns ------- float log of sum of exponentials of elements in `x`. Warnings -------- For performance reasons, doesn't support NaNs or 1d, 3d, etc arrays like :func:`scipy.special.logsumexp`. )rr1rrr)rx_maxs r r:r: sF$q  F26"&U++,, - - U rcTtjtj||z S)aMean absolute difference between two arrays. Parameters ---------- a : numpy.ndarray Input 1d array. b : numpy.ndarray Input 1d array. Returns ------- float mean(abs(a - b)). )rmeanrd)abs r r;r;!s wrva!e}}%%%rcZt|jdkr2t|ttj|z }nFt|ttj|dddtjfz }||jdS)aExpected value of log(theta) where theta is drawn from a Dirichlet distribution. Parameters ---------- alpha : numpy.ndarray Dirichlet parameter 2d matrix or 1d vector, if 2d - each row is treated as a separate parameter vector. Returns ------- numpy.ndarray Log of expected values, dimension same as `alpha.ndim`. r NF)copy)r*rr rrrrr)alphar;s r r<r<3s u{  q  GZZ#bfUmm"4"44FFZZ#bfUA&6&6"7"72: "FFF}}U[u}555rc\tj|d}|d=~|j\}}tdt |jt d|f\}||dd\}}}}|||dd\}}}}~|dksJt|d|d|f} ||kr|ddd|f}t d|f\} | ||dd\} }}| |||dd\} }}|dks Jd | jj sJ| | fS) aGet QR decomposition of `la[0]`. Parameters ---------- la : list of numpy.ndarray Run QR decomposition on the first elements of `la`. Must not be empty. Returns ------- (numpy.ndarray, numpy.ndarray) Matrices :math:`Q` and :math:`R`. Notes ----- Using this function is less memory intense than calling `scipy.linalg.qr(la[0])`, because the memory used in `la[0]` is reclaimed earlier. This makes a difference when decomposing very large arrays, where every memory copy counts. Warnings -------- Content of `la` as well as `la[0]` gets destroyed in the process. Again, for memory-effiency reasons. rzcomputing QR of %s dense matrix)geqrfrsT)lwork overwrite_aN)orgqrz qr failed) rasfortranarrayrr%debugstrrrflags f_contiguous) larAr]nrGqrtauworkr&rgorgqrqs r qr_destroyrXHst0 "Q%  A 1r 7DAq LL2CLLAAA j1$ / /FE%>>>BT4%adCCCBT4 19 RBQBZA1u 2A2YzB511GFF2s"$???MAtTF2s$q'tDDDMAtT 19!!k!!! 7  a4KrcReZdZdZdZdZdZdZdZe d d Z d Z d Z d S)MmWriteraStore a corpus in `Matrix Market format `_, using :class:`~gensim.corpora.mmcorpus.MmCorpus`. Notes ----- The output is written one document at a time, not the whole matrix at once (unlike e.g. `scipy.io.mmread`). This allows you to write corpora which are larger than the available RAM. The output file is created in a single pass through the input corpus, so that the input can be a once-only stream (generator). To achieve this, a fake MM header is written first, corpus statistics are collected during the pass (shape of the matrix, number of non-zeroes), followed by a seek back to the beginning of the file, rewriting the fake header with the final values. s.%%MatrixMarket matrix coordinate real general c||_|ds|drtdtj|jd|_d|_dS)zf Parameters ---------- fname : str Path to output file. z.gzz.bz2z-compressed output not supported with MmWriterzwb+FN)fnameendswithNotImplementedErrorropenfoutheaders_written)rr\s r rzMmWriter.__init__sf >>%  WENN6$:$: W%&UVV VJtz511 $rc |jtj|dkrMtd|j|jtjdnXtd||||j|jtj|d|d|dd|_ d|_ d S) aWrite headers to file. Parameters ---------- num_docs : int Number of documents in corpus. num_terms : int Number of term in corpus. num_nnz : int Number of non-zero elements in corpus. rzsaving sparse matrix to %sz3 z9saving sparse %sx%s matrix with %i non-zero entries to %s  rsTN) r`writerZ HEADER_LINEr%r&r\rto_utf8 last_docnora)rr"r!r#s r write_headerszMmWriter.write_headerss ,--- Q; Z KK4dj A A A IOOEM/:: ; ; ; ; KKK)Wdj    IOOEM(((IIIwww*WXX Y Y Y#rcd|||fz}t|dkrtd|jttj|jtj|dS)aMWrite "fake" headers to file, to be rewritten once we've scanned the entire corpus. Parameters ---------- num_docs : int Number of documents in corpus. num_terms : int Number of term in corpus. num_nnz : int Number of non-zero elements in corpus. z%i %i %i2z Invalid stats: matrix too large!N) r*rtr`seekrZrfrerrg)rr"r!r#statss r fake_headerszMmWriter.fake_headersszh 7;; u::? A?@@ @ s8/00111  e,,-----rc d|js Jd|j|ksJd|j|fztd|D}|D]=\}}|jt jd|dz|dz|fz>||_|r|ddt|fndS) aiWrite a single sparse vector to the file. Parameters ---------- docno : int Number of document. vector : list of (int, number) Document in BoW format. Returns ------- (int, int) Max word index in vector and len of vector. If vector is empty, return (-1, 0). z:must write Matrix Market file headers before writing data!z,documents %i and %i not in sequential order!c3JK|]\}}t|dk||fVdS)g-q=Nr)rgrws r rz(MmWriter.write_vector..s9DD41aSVVe^DADDDDDDrz %i %i %s r rsr)rsr)rarhsortedr`rerrgr*)rr8vectorrr.s r write_vectorzMmWriter.write_vectors #aa%aaaa&qq(VZ^ZikpYq(qqqqDD6DDDDD$ [ [NFF IOOEM,%!)VaZQW9X*XYY Z Z Z Z/5Br 1 s6{{++7BrFNc :t|}|dddd\}}d\} } g} t|dr|j} ||_|ri} nd}t |D]\} }|r |\}}|| | <n|}| |zdkrt d| |r;|j}|| krd| d<| ||} | | |\}}t|d|z}||z }|rtj | |d z| |_| dz}|p|}||zdkr+t d ||d |z||zz |||z||||||r| Sd S) aSSave the corpus to disk in `Matrix Market format `_. Parameters ---------- fname : str Filename of the resulting file. corpus : iterable of list of (int, number) Corpus in streamed bag-of-words format. progress_cnt : int, optional Print progress for every `progress_cnt` number of documents. index : bool, optional Return offsets? num_terms : int, optional Number of terms in the corpus. If provided, the `corpus.num_terms` attribute (if any) will be ignored. metadata : bool, optional Generate a metadata file? Returns ------- offsets : {list of int, None} List of offsets (if index=True) or nothing. Notes ----- Documents are processed one at a time, so the whole corpus is allowed to be larger than the available RAM. See Also -------- :func:`gensim.corpora.mmcorpus.MmCorpus.save_corpus` Save corpus to disk. rs)rr)rsrsmetadataFrzPROGRESS: saving document #%ir z.metadata.cpicklez*saved %ix%i matrix, density=%.3f%% (%i/%i)rN)rZrirrwr)r%r&r`tellr,rtr1rpicklernclose)r\r2 progress_cntrr!rwmw _num_termsr#r8poslastoffsets orig_metadatadocno2metadatar9bowr7r4max_idrr"s r write_corpuszMmWriter.write_corpussDe__ R$$$# Gw 6: & & "OM&FO $!#H#F++  JE3  T(,u%%|#q( D ;UCCC !W$%"$GBKv&&& __UC88NFFZV44J v GG  , L1D)D E E E+FO19+ i 1 $  KK<)UW_98L%MwX`clXl    )W555   N  rc.|dS)zClose `self.fout` file. Alias for :meth:`~gensim.matutils.MmWriter.close`. Warnings -------- Closing the file explicitly via the close() method is preferred and safer. N)rzrs r __del__zMmWriter.__del__6s rctd|jt|dr|jdSdS)zClose `self.fout` file.z closing %sr`N)r%rLr\rr`rzrs r rzzMmWriter.close@sL \4:... 4   IOO       r)ruFNF) rrrrrfrrirnrt staticmethodrrrzrbrr rZrZus EK % % %$$$8...&CCC2TTT\TlrrZ)MmReader)NF)rHrI)r_)r )rFr)Dr __future__rloggingrgensimrnumpyr scipy.sparser- scipy.statsr scipy.linalgrrscipy.linalg.lapackr scipy.specialr getLoggerrr%rrfloat64r>rGrWr^rprrmrrrrlrrrrrrrrarrayrfrrrrrrr r$r*r4r9gensim._matutilsr:r;r< ImportErrorrXrZgensim.corpora._mmreaderr NO_CYTHONrbrr rs%%%%%% --------000000  8 $ $222&????D"&RZ$PTdeSSSSl:]]]]4QQQ"JJJJ.3 3 3 3 ldddd,>@))))6  !1!1!1!1H.2( ( ( ( V,,,,,,,,^A%A%A%A%A%A%A%A%H*,. D"E222 3 3 D"E222 3 3 ^/^/^/^/B>42.CCCC6D+@+@+@\CCC0A6[[[[[[[[[[[=6=6=60&&&$66666W=6@***ZOOOOOOOOd11111111 /s D D10D1E E