c#dZddlmZddlZddlmZ ddlZn#e$r edwxYwddl m Z ddl m Z ddl mZdd lmZdd lmZGd d ZdS) a This module integrates `NMSLIB `_ fast similarity search with Gensim's :class:`~gensim.models.word2vec.Word2Vec`, :class:`~gensim.models.doc2vec.Doc2Vec`, :class:`~gensim.models.fasttext.FastText` and :class:`~gensim.models.keyedvectors.KeyedVectors` vector embeddings. .. Important:: To use this module, you must have the external ``nmslib`` library installed. To install it, run ``pip install nmslib``. To use the integration, instantiate a :class:`~gensim.similarities.nmslib.NmslibIndexer` class and pass the instance as the `indexer` parameter to your model's `model.most_similar()` method. Example usage ------------- .. sourcecode:: pycon >>> from gensim.similarities.nmslib import NmslibIndexer >>> from gensim.models import Word2Vec >>> >>> sentences = [['cute', 'cat', 'say', 'meow'], ['cute', 'dog', 'say', 'woof']] >>> model = Word2Vec(sentences, min_count=1, epochs=10, seed=2) >>> >>> indexer = NmslibIndexer(model) >>> model.wv.most_similar("cat", topn=2, indexer=indexer) [('cat', 1.0), ('meow', 0.16398882865905762)] Load and save example --------------------- .. sourcecode:: pycon >>> from gensim.similarities.nmslib import NmslibIndexer >>> from gensim.models import Word2Vec >>> from tempfile import mkstemp >>> >>> sentences = [['cute', 'cat', 'say', 'meow'], ['cute', 'dog', 'say', 'woof']] >>> model = Word2Vec(sentences, min_count=1, seed=2, epochs=10) >>> >>> indexer = NmslibIndexer(model) >>> _, temp_fn = mkstemp() >>> indexer.save(temp_fn) >>> >>> new_indexer = NmslibIndexer.load(temp_fn) >>> model.wv.most_similar("cat", topn=2, indexer=new_indexer) [('cat', 1.0), ('meow', 0.5595494508743286)] What is NMSLIB -------------- Non-Metric Space Library (NMSLIB) is an efficient cross-platform similarity search library and a toolkit for evaluation of similarity search methods. The core-library does not have any third-party dependencies. More information about NMSLIB: `github repository `_. Why use NMSIB? -------------- Gensim's native :py:class:`~gensim.similarities.Similarity` for finding the `k` nearest neighbors to a vector uses brute force and has linear complexity, albeit with extremely low constant factors. The retrieved results are exact, which is an overkill in many applications: approximate results retrieved in sub-linear time may be enough. NMSLIB can find approximate nearest neighbors much faster, similar to Spotify's Annoy library. Compared to :py:class:`~gensim.similarities.annoy.Annoy`, NMSLIB has more parameters to control the build and query time and accuracy. NMSLIB often achieves faster and more accurate nearest neighbors search than Annoy. )absolute_importN)openzQNMSLIB not installed. To use the NMSLIB indexer, please run `pip install nmslib`.)utils)Doc2Vec)Word2Vec)FastText) KeyedVectorscbeZdZdZd dZejfdZedZ dZ dZ dZ d Z d ZdS) NmslibIndexeraBThis class allows to use `NMSLIB `_ as indexer for `most_similar` method from :class:`~gensim.models.word2vec.Word2Vec`, :class:`~gensim.models.doc2vec.Doc2Vec`, :class:`~gensim.models.fasttext.FastText` and :class:`~gensim.models.keyedvectors.Word2VecKeyedVectors` classes. Nc|ddddd}|ddi}d|_d|_||_||_||_|rt |jt r|dSt |jttfr| dSt |jtfr| dStddS)a Parameters ---------- model : :class:`~gensim.models.base_any2vec.BaseWordEmbeddingsModel` Model, that will be used as source for index. index_params : dict, optional Indexing parameters passed through to NMSLIB: https://github.com/nmslib/nmslib/blob/master/manual/methods.md#graph-based-search-methods-sw-graph-and-hnsw If not specified, defaults to `{'M': 100, 'indexThreadQty': 1, 'efConstruction': 100, 'post': 0}`. query_time_params : dict, optional query_time_params for NMSLIB indexer. If not specified, defaults to `{'efSearch': 100}`. Ndr)MindexThreadQtyefConstructionpostefSearchzDmodel must be a Word2Vec, Doc2Vec, FastText or KeyedVectors instance)indexlabelsmodel index_paramsquery_time_params isinstancer_build_from_doc2vecrr_build_from_word2vecr _build_from_keyedvectors ValueError)selfrrrs :lib/python3.11/site-packages/gensim/similarities/nmslib.py__init__zNmslibIndexer.__init__fs   ]!$SZ[\\L  2!+S 1    (!2  i$*g.. i((*****DJ8(<== i))+++++DJ88 i--///// !ghhh i ic|dz}|j||j|j|jd}t |d5}t j|||ddddS#1swxYwYdS)aSave this NmslibIndexer instance to a file. Parameters ---------- fname : str Path to the output file, will produce 2 files: `fname` - parameters and `fname`.d - :class:`~nmslib.NmslibIndex`. protocol : int, optional Protocol for pickle. Notes ----- This method saves **only** the index (**the model isn't preserved**). .d)rrrwb)protocolN)r saveIndexrrrr_pickledump)rfnamer% fname_dictdfouts rsavezNmslibIndexer.saves T\  U###!.TE[gkgr s s *d # # 5t LD8 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5sA**A.1A.cB|dz}t|d5}tj|}dddn #1swxYwY|d}|d}|d||}tjdd }||||_|d |_|S) zLoad a NmslibIndexer instance from a file. Parameters ---------- fname : str Path previously used in `save()`. r#rbNrr)rrrhnsw cosinesimilmethodspacer)rr'loadnmslibinit loadIndexrr) clsr)r*fr+rrnmslib_instancers rr5zNmslibIndexer.loadsT\ *d # # q QA               ( 12#D|Whiii 6???  %!"8s 7;;c||jj|jjjdS)z?Build an NMSLIB index using word vectors from a Word2Vec model.N)_build_from_modelrwvget_normed_vectors index_to_keyrs rrz"NmslibIndexer._build_from_word2vecs5 tz}??AA4:=C]^^^^^r!c||jj}|j}|||dS)zBBuild an NMSLIB index using document vectors from a Doc2Vec model.N)rdvr@r=r?)rdocvecsrs rrz!NmslibIndexer._build_from_doc2vecs;*-% w99;;VDDDDDr!ct||j|jjdS)zCBuild an NMSLIB index using word vectors from a KeyedVectors model.N)r=rr?r@rAs rrz&NmslibIndexer._build_from_keyedvectorss1 tz<<>> @WXXXXXr!ctjdd}||||jdtj||j||_||_dS)Nr0r1r2T)print_progress) r6r7addDataPointBatch createIndexrsetQueryTimeParamsrrr)rvectorsrrs rr=zNmslibIndexer._build_from_modelsn 6??? ((( $+DAAA!%)?@@@  r!cj|dd|d\}}fdt||DS)aFind the approximate `num_neighbors` most similar items. Parameters ---------- vector : numpy.array Vector for a word or document. num_neighbors : int How many most similar items to look for? Returns ------- list of (str, float) List of most similar items in the format `[(item, cosine_similarity), ... ]`. r)krc:g|]\}}j|d|z fS)g?)r).0id_distancers r z.NmslibIndexer.most_similar..s-\\\}sHS!3>2\\\r!)r knnQueryBatchreshapezip)rvector num_neighborsids distancess` r most_similarzNmslibIndexer.most_similars_ 11&..B2G2G=1YYZ[\Y]\\\CQZH[H[\\\\r!)NN)__name__ __module__ __qualname____doc__r rPICKLE_PROTOCOLr- classmethodr5rrrr=r[r!rr r _s )i)i)i)iV$)#85555,[*___EEE YYY]]]]]r!r )r_ __future__rpickler' smart_openrr6 ImportErrorgensimrgensim.models.doc2vecrgensim.models.word2vecrgensim.models.fasttextr gensim.modelsr r rbr!rrls'EEP'&&&&&kMMMMkkk +i j jjk))))))++++++++++++&&&&&&J]J]J]J]J]J]J]J]J]J]s*