cdZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z ejeZdZGddZedkr ejdej ejedd ej ZeZed d eje jdededejdSdS)aWorker ("slave") process used in computing distributed Latent Semantic Indexing (LSI, :class:`~gensim.models.lsimodel.LsiModel`) models. Run this script on every node in your cluster. If you wish, you may even run it multiple times on a single machine, to make better use of multiple cores (just beware that memory footprint increases linearly). How to use distributed LSI -------------------------- #. Install needed dependencies (Pyro4) :: pip install gensim[distributed] #. Setup serialization (on each machine) :: export PYRO_SERIALIZERS_ACCEPTED=pickle export PYRO_SERIALIZER=pickle #. Run nameserver :: python -m Pyro4.naming -n 0.0.0.0 & #. Run workers (on each machine) :: python -m gensim.models.lsi_worker & #. Run dispatcher :: python -m gensim.models.lsi_dispatcher & #. Run :class:`~gensim.models.lsimodel.LsiModel` in distributed mode: .. sourcecode:: pycon >>> from gensim.test.utils import common_corpus, common_dictionary >>> from gensim.models import LsiModel >>> >>> model = LsiModel(common_corpus, id2word=common_dictionary, distributed=True) Command line arguments ---------------------- .. program-output:: python -m gensim.models.lsi_worker --help :ellipsis: 0, -3 N)lsimodel)utilscXeZdZdZejdZejejdZe j ddZ eje j ddZ eje j ddZ ejdZd S) Workercd|_dS)zPartly initialize the model. A full initialization requires a call to :meth:`~gensim.models.lsi_worker.Worker.initialize`. N)modelselfs 8lib/python3.11/site-packages/gensim/models/lsi_worker.py__init__zWorker.__init__Ms  c tj|_d|_||_||_d|_td|tj di||_ dS)aFully initialize the worker. Parameters ---------- myid : int An ID number used to identify this worker in the dispatcher object. dispatcher : :class:`~gensim.models.lsi_dispatcher.Dispatcher` The dispatcher responsible for scheduling this worker. **model_params Keyword parameters to initialize the inner LSI model, see :class:`~gensim.models.lsimodel.LsiModel`. rFzinitializing worker #%sN) threadingLock lock_updatejobsdonemyid dispatcherfinishedloggerinforLsiModelr)r rr model_paramss r initializezWorker.initializeUsa%>++  $  -t444&6666 r c|jtdd}|F|js? |j|j}n#t j$rY;wxYw||j?|\t d|j|j | ||j |jdSt d|jdS)aRequest jobs from the dispatcher, in a perpetual loop until :meth:`~gensim.models.lsi_worker.Worker.getstate` is called. Raises ------ RuntimeError If `self.model` is None (i.e. worker not initialized). Nz0worker must be initialized before receiving jobszworker #%s received job #%iz#worker #%i stopping asking for jobs) r RuntimeErrorrrgetjobrQueueEmptyrrr processjobjobdone)r jobs r requestjobzWorker.requestjobls : SQRR R $-  o,,TY77;     $-   J KK5ty$- P P P OOC O # #DI . . . . . KK=ty I I I I IsAAArc,|j||xjdz c_tr`|jtzdkrOtjtjd}|j |dSdSdS)zIncrementally process the job and potentially logs progress. Parameters ---------- job : iterable of list of (int, float) Corpus in BoW format. rzlsi_worker.pklN) r add_documentsr SAVE_DEBUGospathjointempfile gettempdirsave)r r#fnames r r!zWorker.processjobs   %%%   #$-*49 #GLL!4!6!68HIIE JOOE " " " " " # # # #r ctd|j|jt |jjtjsJd|_ |jjS)zLog and get the LSI model's current projection. Returns ------- :class:`~gensim.models.lsimodel.Projection` The current projection. z,worker #%i returning its state after %s jobsT) rrrr isinstancer projectionr Projectionrr s r getstatezWorker.getstatesO  BDIt}]]]$*/1DEEEEE z$$r ctd|j|jj|j_d|_dS)z4Reset the worker by deleting its current projection.zresetting worker #%iFN)rrrrr2 empty_likerr s r resetz Worker.resetsA  *DI666 $ 5 @ @ B B  r cntd|jtjddS)zTerminate the worker.zterminating worker #%irN)rrrr)_exitr s r exitz Worker.exits,  ,di888  r N)__name__ __module__ __qualname__r Pyro4exposeronewayr$r synchronousr!r4r7r:rr r rrLs \77\7, \ \JJ\\J6U}%% # #&% # \U}%% % %&%\ % \U}%%&%\  \\r r__main__z)%(asctime)s - %(levelname)s - %(message)s)formatleveliy) descriptionformatter_classz running %s zgensim.lsi_workerT) random_suffixzfinished running %s)__doc__r)sysloggingargparserr,queuerr> gensim.modelsrgensimr getLoggerr;rr(r basicConfigINFOArgumentParserRawTextHelpFormatterparser parse_args_rr+argv pyro_daemonprogrr r r[s//b  """"""  8 $ $ iiiiiiiiX z 4GJRYR^____ $X $$QYQn o o oFA KK chhsx00111E)66884HHHH KK%v{33333 4 4r