c:dZddlZddlmZmZejeZGddejZ Gdde Z Gdd ejZ Gd d ejZ dS) aoBasic interfaces used across the whole Gensim package. These interfaces are used for building corpora, model transformation and similarity queries. The interfaces are realized as abstract base classes. This means some functionality is already provided in the interface itself, and subclasses should inherit from these interfaces and implement the missing methods. N)utilsmatutilscFeZdZdZdZfdZdZeddZxZ S) CorpusABCa3Interface for corpus classes from :mod:`gensim.corpora`. Corpus is simply an iterable object, where each iteration step yields one document: .. sourcecode:: pycon >>> from gensim.corpora import MmCorpus # inherits from the CorpusABC class >>> from gensim.test.utils import datapath >>> >>> corpus = MmCorpus(datapath("testcorpus.mm")) >>> for doc in corpus: ... pass # do something with the doc... A document represented in the bag-of-word (BoW) format, i.e. list of (attr_id, attr_value), like ``[(1, 0.2), (4, 0.6), ...]``. .. sourcecode:: pycon >>> from gensim.corpora import MmCorpus # inherits from the CorpusABC class >>> from gensim.test.utils import datapath >>> >>> corpus = MmCorpus(datapath("testcorpus.mm")) >>> doc = next(iter(corpus)) >>> print(doc) [(0, 1.0), (1, 1.0), (2, 1.0)] Remember that the save/load methods only pickle the corpus object, not the (streamed) corpus data itself! To save the corpus data, please use this pattern : .. sourcecode:: pycon >>> from gensim.corpora import MmCorpus # MmCorpus inherits from CorpusABC >>> from gensim.test.utils import datapath, get_tmpfile >>> >>> corpus = MmCorpus(datapath("testcorpus.mm")) >>> tmp_path = get_tmpfile("temp_corpus.mm") >>> >>> MmCorpus.serialize(tmp_path, corpus) # serialize corpus to disk in the MmCorpus format >>> loaded_corpus = MmCorpus(tmp_path) # load corpus through constructor >>> for (doc_1, doc_2) in zip(corpus, loaded_corpus): ... assert doc_1 == doc_2 # no change between the original and loaded corpus See Also -------- :mod:`gensim.corpora` Corpora in different formats. c td)zIterate all over corpus.&cannot instantiate abstract base classNotImplementedErrorselfs 1lib/python3.11/site-packages/gensim/interfaces.py__iter__zCorpusABC.__iter__Ls!"JKKKczddl}|dtt|j|i|dS)agSaves the in-memory state of the corpus (pickles the object). Warnings -------- This saves only the "internal state" of the corpus object, not the corpus data! To save the corpus data, use the `serialize` method of your desired output format instead, e.g. :meth:`gensim.corpora.mmcorpus.MmCorpus.serialize`. rNzcorpus.save() stores only the (tiny) iteration object in memory; to serialize the actual corpus content, use e.g. MmCorpus.serialize(corpus))warningswarnsuperrsave)r argskwargsr __class__s r rzCorpusABC.savePsT   Z    $i#T4V44444rc td)z:Get the corpus size = the total number of documents in it.z2must override __len__() before calling len(corpus)r r s r __len__zCorpusABC.__len__bs!"VWWWrNFc td)awSave `corpus` to disk. Some formats support saving the dictionary (`feature_id -> word` mapping), which can be provided by the optional `id2word` parameter. Notes ----- Some corpora also support random access via document indexing, so that the documents on disk can be accessed in O(1) time (see the :class:`gensim.corpora.indexedcorpus.IndexedCorpus` base class). In this case, :meth:`~gensim.interfaces.CorpusABC.save_corpus` is automatically called internally by :func:`serialize`, which does :meth:`~gensim.interfaces.CorpusABC.save_corpus` plus saves the index at the same time. Calling :func:`serialize() is preferred to calling :meth:`gensim.interfaces.CorpusABC.save_corpus`. Parameters ---------- fname : str Path to output file. corpus : iterable of list of (int, number) Corpus in BoW format. id2word : :class:`~gensim.corpora.Dictionary`, optional Dictionary of corpus. metadata : bool, optional Write additional metadata to a separate too? rr )fnamecorpusid2wordmetadatas r save_corpuszCorpusABC.save_corpusfs<""JKKKr)NF) __name__ __module__ __qualname____doc__rrr staticmethodr __classcell__)rs@r rrs11dLLL55555$XXXLLL\LLLLLrrc,eZdZdZddZdZdZdZdS)TransformedCorpuszQInterface for corpora that are the result of an online (streamed) transformation.Nc |||c|_|_|_|D]\}}t |j||d|_dS)a Parameters ---------- obj : object A transformation :class:`~gensim.interfaces.TransformationABC` object that will be applied to each document from `corpus` during iteration. corpus : iterable of list of (int, number) Corpus in bag-of-words format. chunksize : int, optional If provided, a slightly more effective processing will be performed by grouping documents from `corpus`. FN)objr chunksizeitemssetattrr)r r)rr*rkeyvalues r __init__zTransformedCorpus.__init__sW14VY-$+t~ ,,.. * *JC DHc5 ) ) ) ) rc*t|jS)zGet corpus size.)lenrr s r rzTransformedCorpus.__len__s4;rc#K|jrGtj|j|jD]%}|j|dD]}|V&dS|jD]}|j|VdS)a#Iterate over the corpus, applying the selected transformation. If `chunksize` was set in the constructor, works in "batch-manner" (more efficient). Yields ------ list of (int, number) Documents in the sparse Gensim bag-of-words format. N)r*)r*rgrouperrr) __getitem__)r chunk transformeddocs r rzTransformedCorpus.__iter__s > $t{DNCC & &#'8#7#7#7#N#N&&K%%%%%& & &{ $ $hsm#### $ $rct|jdr|j|j|Stdt |j)aYTransform the document at position `docno` within `corpus` specified in the constructor. Parameters ---------- docno : int Position of the document to transform. Document offset inside `self.corpus`. Notes ----- `self.corpus` must support random indexing. Returns ------- list of (int, number) Transformed document in the sparse Gensim bag-of-words format. Raises ------ RuntimeError If corpus doesn't support index slicing (`__getitem__` doesn't exists). r4z!Type {} does not support slicing.)hasattrrr) RuntimeErrorformattype)r docnos r r4zTransformedCorpus.__getitem__sT. 4; . . ^8DK./ /BII$t{J[J[\\]] ]rN)r r!r"r#r/rrr4rr r'r's`[[(   $$$&^^^^^rr'c eZdZdZdZddZdS)TransformationABCaTransformation interface. A 'transformation' is any object which accepts document in BoW format via the `__getitem__` (notation `[]`) and returns another sparse document in its stead: .. sourcecode:: pycon >>> from gensim.models import LsiModel >>> from gensim.test.utils import common_dictionary, common_corpus >>> >>> model = LsiModel(common_corpus, id2word=common_dictionary) >>> bow_vector = model[common_corpus[0]] # model applied through __getitem__ on one document from corpus. >>> bow_corpus = model[common_corpus] # also, we can apply model on the full corpus c td)aTransform a single document, or a whole corpus, from one vector space into another. Parameters ---------- vec : {list of (int, number), iterable of list of (int, number)} Document in bag-of-words, or streamed corpus. rr )r vecs r r4zTransformationABC.__getitem__""JKKKrNc t|||fi|S)aApply the transformation to a whole corpus and get the result as another corpus. Parameters ---------- corpus : iterable of list of (int, number) Corpus in sparse Gensim bag-of-words format. chunksize : int, optional If provided, a more effective processing will performed. Returns ------- :class:`~gensim.interfaces.TransformedCorpus` Transformed corpus. )r')r rr*rs r _applyzTransformationABC._applys !vyCCFCCCrr>)r r!r"r#r4rFr?rr rArAsJ L L LDDDDDDrrAc*eZdZdZdZdZdZdZdS) SimilarityABCaInterface for similarity search over a corpus. In all instances, there is a corpus against which we want to perform the similarity search. For each similarity search, the input is a document or a corpus, and the output are the similarities to individual corpus documents. Examples -------- .. sourcecode:: pycon >>> from gensim.similarities import MatrixSimilarity >>> from gensim.test.utils import common_corpus >>> >>> index = MatrixSimilarity(common_corpus) >>> similarities = index.get_similarities(common_corpus[1]) # get similarities between query and corpus Notes ----- There is also a convenience wrapper, where iterating over `self` yields similarities of each document in the corpus against the whole corpus (i.e. the query is each corpus document in turn). See Also -------- :mod:`gensim.similarities` Different index implementations of this interface. c td)z Parameters ---------- corpus : iterable of list of (int, number) Corpus in sparse Gensim bag-of-words format. &cannot instantiate Abstract Base Classr )r rs r r/zSimilarityABC.__init__rDrc td)a%Get similarities of the given document or corpus against this index. Parameters ---------- doc : {list of (int, number), iterable of list of (int, number)} Document in the sparse Gensim bag-of-words format, or a streamed corpus of such documents. rJr )r r7s r get_similaritieszSimilarityABC.get_similarities&rDrctj|\}}jr7tj|s#|r d|D}ntj|}|}j|Stddrtj |jStj|rfd|DStj |jS)aGet similarities of the given document or corpus against this index. Uses :meth:`~gensim.interfaces.SimilarityABC.get_similarities` internally. Notes ----- Passing an entire corpus as `query` can be more efficient than passing its documents one after another, because it will issue queries in batches internally. Parameters ---------- query : {list of (int, number), iterable of list of (int, number)} Document in the sparse Gensim bag-of-words format, or a streamed corpus of such documents. Returns ------- {`scipy.sparse.csr.csr_matrix`, list of (int, float)} Similarities given document or corpus and objects corpus, depends on `query`. c6g|]}tj|Sr?)runitvec).0vs r z-SimilarityABC.__getitem__..Ns#@@@QX-a00@@@rNmaintain_sparsityFcDg|]}tj|jSr?)rfull2sparse_clippednum_best)rPrQr s r rRz-SimilarityABC.__getitem__..^s(SSSqH0DMBBSSSr) r is_corpus normalizerismatrixrOrLrVgetattrscipy2scipy_clippedrU)r queryrWresults` r r4zSimilarityABC.__getitem__1s*!?511 5 > 4 $U++ 44@@%@@@EE$,U33E&&u-- = M 4,e 4 4 G/ FF F  V $ $ GSSSSFSSS S/ FF Frc#~K|j}d|_ |jdk}n#t$rd}YnwxYw|rptd|jjd|jD]H}t |jjd||jz}|j||}||D]}|VIn|jD] }||V ||_dS)a Iterate over all documents, compute similarity of each document against all other documents in the index. Yields ------ {`scipy.sparse.csr.csr_matrix`, list of (int, float)} Similarity of the current document and all documents in the corpus. FrN)rXr*AttributeErrorrangeindexshapemin)r normchunking chunk_start chunk_endr5simr7s r rzSimilarityABC.__iter__cs ~ ~)HH   HHH    %Q (8(;T^LL     0 3[4>5QRR  ; #9:;CIIII  z  3is  --N)r r!r"r#r/rLr4rr?rr rHrHsd6 L L L L L L0G0G0Gd+++++rrH) r#logginggensimrr getLoggerr loggerSaveLoadrr'rArHr?rr ros3""""""""  8 $ $kLkLkLkLkLkLkLkL\G^G^G^G^G^ G^G^G^T+D+D+D+D+D+D+D+D\OOOOOENOOOOOr