c DdZddlZddlZddlmZmZmZddlZddl m Z ddl m Z m Z ejeZGdde ZdZdS) aOnline Latent Dirichlet Allocation (LDA) in Python, using all CPU cores to parallelize and speed up model training. The parallelization uses multiprocessing; in case this doesn't work for you for some reason, try the :class:`gensim.models.ldamodel.LdaModel` class which is an equivalent, but more straightforward and single-core implementation. The training algorithm: * is **streamed**: training documents may come in sequentially, no random access required, * runs in **constant memory** w.r.t. the number of documents: size of the training corpus does not affect memory footprint, can process corpora larger than RAM Wall-clock `performance on the English Wikipedia `_ (2G corpus positions, 3.5M documents, 100K features, 0.54G non-zero entries in the final bag-of-words matrix), requesting 100 topics: ====================================================== ============== algorithm training time ====================================================== ============== LdaMulticore(workers=1) 2h30m LdaMulticore(workers=2) 1h24m LdaMulticore(workers=3) 1h6m old LdaModel() 3h44m simply iterating over input corpus = I/O overhead 20m ====================================================== ============== (Measured on `this i7 server `_ with 4 physical cores, so that optimal `workers=3`, one less than the number of cores.) This module allows both LDA model estimation from a training corpus and inference of topic distribution on new, unseen documents. The model can also be updated with new documents for online training. The core estimation code is based on the `onlineldavb.py script `_, by Matthew D. Hoffman, David M. Blei, Francis Bach: `'Online Learning for Latent Dirichlet Allocation', NIPS 2010`_. .. _'Online Learning for Latent Dirichlet Allocation', NIPS 2010: online-lda_ .. _'Online Learning for LDA' by Hoffman et al.: online-lda_ .. _online-lda: https://papers.neurips.cc/paper/2010/file/71f6278d140af599e06ad9bf1ba03cb0-Paper.pdf Usage examples -------------- The constructor estimates Latent Dirichlet Allocation model parameters based on a training corpus .. sourcecode:: pycon >>> from gensim.test.utils import common_corpus, common_dictionary >>> >>> lda = LdaMulticore(common_corpus, id2word=common_dictionary, num_topics=10) Save a model to disk, or reload a pre-trained model .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> >>> # Save model to disk. >>> temp_file = datapath("model") >>> lda.save(temp_file) >>> >>> # Load a potentially pretrained model from disk. >>> lda = LdaModel.load(temp_file) Query, or update the model using new, unseen documents .. sourcecode:: pycon >>> other_texts = [ ... ['computer', 'time', 'graph'], ... ['survey', 'response', 'eps'], ... ['human', 'system', 'computer'] ... ] >>> other_corpus = [common_dictionary.doc2bow(text) for text in other_texts] >>> >>> unseen_doc = other_corpus[0] >>> vector = lda[unseen_doc] # get topic probability distribution for a document >>> >>> # Update the model by incrementally training on the new corpus. >>> lda.update(other_corpus) # update the LDA model with additional documents N)PoolQueue cpu_count)utils)LdaModelLdaStatec\eZdZdZddddddddddd d d d dd d dejffd ZddZxZS) LdaMulticorezAn optimized implementation of the LDA algorithm, able to harness the power of multicore CPUs. Follows the similar API as the parent class :class:`~gensim.models.ldamodel.LdaModel`. NdiF symmetricg?g? 2gMbP?g{Gz?c,|tdtdz n||_||_t |t r|dkrt dtt| ||||||| | | | | ||||||dS)a* Parameters ---------- corpus : {iterable of list of (int, float), scipy.sparse.csc}, optional Stream of document vectors or sparse matrix of shape (`num_documents`, `num_terms`). If not given, the model is left untrained (presumably because you want to call :meth:`~gensim.models.ldamodel.LdaModel.update` manually). num_topics : int, optional The number of requested latent topics to be extracted from the training corpus. id2word : {dict of (int, str), :class:`gensim.corpora.dictionary.Dictionary`} Mapping from word IDs to words. It is used to determine the vocabulary size, as well as for debugging and topic printing. workers : int, optional Number of workers processes to be used for parallelization. If None all available cores (as estimated by `workers=cpu_count()-1` will be used. **Note** however that for hyper-threaded CPUs, this estimation returns a too high number -- set `workers` directly to the number of your **real** cores (not hyperthreads) minus one, for optimal performance. chunksize : int, optional Number of documents to be used in each training chunk. passes : int, optional Number of passes through the corpus during training. alpha : {float, numpy.ndarray of float, list of float, str}, optional A-priori belief on document-topic distribution, this can be: * scalar for a symmetric prior over document-topic distribution, * 1D array of length equal to num_topics to denote an asymmetric user defined prior for each topic. Alternatively default prior selecting strategies can be employed by supplying a string: * 'symmetric': (default) Uses a fixed symmetric prior of `1.0 / num_topics`, * 'asymmetric': Uses a fixed normalized asymmetric prior of `1.0 / (topic_index + sqrt(num_topics))`. eta : {float, numpy.ndarray of float, list of float, str}, optional A-priori belief on topic-word distribution, this can be: * scalar for a symmetric prior over topic-word distribution, * 1D array of length equal to num_words to denote an asymmetric user defined prior for each word, * matrix of shape (num_topics, num_words) to assign a probability for each word-topic combination. Alternatively default prior selecting strategies can be employed by supplying a string: * 'symmetric': (default) Uses a fixed symmetric prior of `1.0 / num_topics`, * 'auto': Learns an asymmetric prior from the corpus. decay : float, optional A number between (0.5, 1] to weight what percentage of the previous lambda value is forgotten when each new document is examined. Corresponds to :math:`\kappa` from `'Online Learning for LDA' by Hoffman et al.`_ offset : float, optional Hyper-parameter that controls how much we will slow down the first steps the first few iterations. Corresponds to :math:`\tau_0` from `'Online Learning for LDA' by Hoffman et al.`_ eval_every : int, optional Log perplexity is estimated every that many updates. Setting this to one slows down training by ~2x. iterations : int, optional Maximum number of iterations through the corpus when inferring the topic distribution of a corpus. gamma_threshold : float, optional Minimum change in the value of the gamma parameters to continue iterating. minimum_probability : float, optional Topics with a probability lower than this threshold will be filtered out. random_state : {np.random.RandomState, int}, optional Either a randomState object or a seed to generate one. Useful for reproducibility. Note that results can still vary due to non-determinism in OS scheduling of the worker processes. minimum_phi_value : float, optional if `per_word_topics` is True, this represents a lower bound on the term probabilities. per_word_topics : bool If True, the model also computes a list of topics, sorted in descending order of most likely topics for each word, along with their phi values multiplied by the feature length (i.e. word count). dtype : {numpy.float16, numpy.float32, numpy.float64}, optional Data-type to use during calculations inside model. All inputs are also converted. Nr autozFauto-tuning alpha not implemented in LdaMulticore; use plain LdaModel.)corpus num_topicsid2word chunksizepassesalphaetadecayoffset eval_every iterationsgamma_threshold random_stateminimum_probabilityminimum_phi_valueper_word_topicsdtype) maxrworkersbatch isinstancestrNotImplementedErrorsuperr __init__)selfrrrr$rrr%rrrrrrrrrr r!r" __class__s :lib/python3.11/site-packages/gensim/models/ldamulticore.pyr*zLdaMulticore.__init__msN3:Ns1ikkAo...w  eS ! ! pevo p%&noo o lD!!**jyuRU:*+,\o/X] +     c N  t|nC#t$r6tdt d|DYnwxYwdkrtddSjxjz c_jrd}nd}jj zj pd t z}tdz }t d |jj|jj |jzd krtd t%d j z }t%fdd f d }t dj t'j t(|f}t+jD]Bdgdc} t-jjjjt5j|j|} t9| D]\} | t z } || jfddxxdz cc<t d| | jzt zdn #t<j$r |YnwxYw|ddkr|dddk| krtAdD|!dS)aSTrain the model with new documents, by EM-iterating over `corpus` until the topics converge (or until the maximum number of allowed iterations is reached). Train the model with new documents, by EM-iterating over the corpus until the topics converge, or until the maximum number of allowed iterations is reached. `corpus` must be an iterable. The E step is distributed into the several processes. Notes ----- This update also supports updating an already trained model (`self`) with new documents from `corpus`; the two models are then merged in proportion to the number of old vs. new documents. This feature is still experimental for non-stationary input streams. For stationary input (no topic drift in new documents), on the other hand, this equals the online update of `'Online Learning for LDA' by Hoffman et al.`_ and is guaranteed to converge for any `decay` in (0.5, 1]. Parameters ---------- corpus : {iterable of list of (int, float), scipy.sparse.csc}, optional Stream of document vectors or sparse matrix of shape (`num_documents`, `num_terms`) used to update the model. chunks_as_numpy : bool Whether each chunk passed to the inference step should be a np.ndarray or not. Numpy can in some settings turn the term IDs into floats, these will be converted back into integers in inference, which incurs a performance hit. For distributed computing it may be desirable to keep the chunks as `numpy.ndarray`. z4input corpus stream has no len(); counting documentsc3K|]}dVdS)r N).0_s r- z&LdaMulticore.update..s"..!A......r.rz1LdaMulticore.update() called with an empty corpusNr%onliner zrunning %s LDA training, %s topics, %i passes over the supplied corpus of %i documents, updating every %i documents, evaluating every ~%i documents, iterating %ix with a convergence threshold of %frzxtoo few updates, training might not converge; consider increasing the number of passes or iterations to improve accuracy)maxsizec`tjzjjz zj S)N)powr num_updatesrr)pass_r+sr-rhoz LdaMulticore.update..rhos0t{U*d.>.OPSWS]R]^^ ^r.Fc d}sMdxxdzcc<d}M|r|r ddks j krk  dkdkr,|s j z zdkr dSdSdSdS)z Clear the result queue, merging all intermediate results, and update the LDA model if necessary. Frr T) total_docsN)emptymergegetnumdocsdo_mstepresetr:log_perplexity) force merged_newchunkr lencorpusotherr; queue_size result_queuer<r+ updateafters r-process_result_queuez1LdaMulticore.update..process_result_queue sB J"((** " L,,..///1 " ! #((** "  E* EA!); ER]A] E cceeUEAI666 >EuE1AK1OS]0]ab0bE'')'DDDDDEE E EEEr.z%training LDA model using %i processes)as_numpyT)blockz[PROGRESS: pass %i, dispatched chunk #%i = documents up to #%i/%i, outstanding queue size %i)rFzIinput corpus size changed during training (don't use generators as input)F)"len TypeErrorloggerwarningsumstaterBr%rr$rminr#inforrrrrr worker_e_steprangerrsstatsshapergrouper enumerateputqueueFull RuntimeError terminate)r+rchunks_as_numpy updatetype evalafterupdates_per_pass job_queuerNpoolreallen chunk_streamchunk_norHrrIrJr;rKrLr<rMs` @@@@@@@@@r-updatezLdaMulticore.updates: /F II / / / NNQ R R R..v.....III / >  NNN O O O F i' : 8 J#KK!J.4<7K_)  : #;<< q)k"9::  ? i t(<     dk )B .  NN]    !dl"2333 ww  _ _ _ _ _ _ E E E E E E E E E E E E E E E$  ;T\JJJDL-)\41PQQ4;'' p pE#$#q JTXtz'8'>??E =/ZZZL#,\#:#: ' '%3u::% / /! x &C5 QQQ"1 *  8!8X-FU-SU^`jkl`m  :///-,...../ /%$&&&&Q-!# 1$$40000Q-!# 1)# p"#nooo p s"=AA>A,J++KKrQ) __name__ __module__ __qualname____doc__npfloat32r*rn __classcell__)r,s@r-r r hs#sD$kSRB!&Tt#'bj S S S S S S j@@@@@@@@r.r c.td td|\}}}td|t|||_||j||~td||jd|_td)akPerform E-step for each job. Parameters ---------- input_queue : queue of (int, list of (int, float), :class:`~gensim.models.lda_worker.Worker`) Each element is a job characterized by its ID, the corpus chunk to be processed in BOW format and the worker responsible for processing it. result_queue : queue of :class:`~gensim.models.ldamodel.LdaState` After the worker finished the job, the state of the resulting (trained) worker model is appended to this queue. worker_lda : :class:`~gensim.models.ldamulticore.LdaMulticore` LDA instance which performed e step z#worker process entering E-step loopTzgetting a new jobz$processing chunk #%i of %i documentsz#processed chunk, queuing the resultNz result put) rTdebugrArRrW sync_staterDdo_estepr`) input_queuerL worker_ldarmrHw_states r-rZrZEs LL6777 # ()))#.??#4#4 % ;Xs5zzRRR"    E"""  :;;;)***  \""" #r.)rrloggingramultiprocessingrrrnumpyrsgensimrgensim.models.ldamodelrr getLoggerrorTr rZr1r.r-rsQQf 222222222255555555  8 $ $ZZZZZ8ZZZz#####r.