o Nrf!@sUddlmZddlZddlZddlZddlZddlZddlZddlZddl m Z m Z ddl m Z ddlmZddlmZddlZddlmZddlmZmZmZdd lmZmZdd lmZdd lmZdd l m!Z!d dZ"ee#e$j%e"eej&ej'dZ(ej)Z*ddZ+Gddde,Z-iZ.de/d<d0ddZ0zddl1Z2e2j34ddZ5Wne6yddZ5d1ddZYnwd d!Z7d"Z8d#d$Z9    %   d2d3d*d+Z:d,d-Z;d1d.d/ZcCs|Srrr<rrr_pack_tracebackfsr@cCsdt|S)N)joinr! format_tbr?rrrr@krcCst||}|r)r>)r7r<rrrr ns r c Csrt\}}}t|}z |||f}W|Sty8}zt\}}}t|}|||f}WYd}~|Sd}~wwr)sysexc_infor@ BaseException)edumpsexc_type exc_value exc_tracebackr<resultrrrpack_exceptionssrMzThe 'multiprocessing.context' configuration option will be ignored on Python 2 and on Windows, because they each only support a single context. cCs6tdd}tjdkr|dkrttttSt|S)z+Return the current multiprocessing context.zmultiprocessing.contextZspawnwin32) rgetrDplatformr_CONTEXT_UNSUPPORTED UserWarningr get_context)Z context_namerrrrSs    rSTdskrkeysSequence[Key] | Keyc  KsZ|ptdd}|ptdd}|ptdd}|p!tddp!t}|durFtjddvr3d tjd<t} tt|d }t|| |d }d } n|durNt d t |t j j rYt|}d} t|}t||\} } |rqt| || \}} n| }|p|tddp|t}|ptddpt}zt|j|j||ft||tt|d| }W| r||S| r|ww)aMultiprocessed get function appropriate for Bags Parameters ---------- dsk : dict dask graph keys : object or list Desired results from graph num_workers : int Number of worker processes (defaults to number of cores) func_dumps : function Function to use for function serialization (defaults to cloudpickle.dumps) func_loads : function Function to use for function deserialization (defaults to cloudpickle.loads) optimize_graph : bool If True [default], `fuse` is applied to the graph before computation. pool : Executor or Pool Some sort of `Executor` or `Pool` to use initializer: function Ignored if ``pool`` has been set. Function to initialize a worker process before running any tasks in it. chunksize: int, optional Size of chunks to use when dispatching work. Defaults to 5 as some batching is helpful. If -1, will be computed to evenly divide ready work across workers. chunksizepoolNzmultiprocessing.initializer num_workersZPYTHONHASHSEED)N0Z6640Zuser_initializer)Z mp_context initializerTzThe ``initializer`` argument is ignored when ``pool`` is provided. The user should configure ``pool`` with the needed ``initializer`` on creation.F func_loads func_dumps)Zget_idrHloadsrMZraise_exceptionrW)rrOrosenvironrSrinitialize_worker_processrr isinstancerrYZPoolr rr r _loads_dumpsr submitZ _max_workersrrMr shutdown)rTrUrZr^r_Zoptimize_graphrYr]rWkwargscontextcleanupZdsk2 dependenciesZdsk3r`rHrLrrrrOsb&    rOcCs&tjd}|dur|jdSdS)Nnumpy)rDmodulesrOrandomseed)nprrrdefault_initializers rrcCst|dur |dSdS)zE Initialize a worker process before running any tasks in it. N)rrr\rrrrcs rc)r7r8r9r8r)NNNTNNN)rTrrUrV)= __future__rcopyregrZmultiprocessing.poolrapicklerDr!collections.abcrrconcurrent.futuresr functoolsrwarningsrZ cloudpickleZdaskrZ dask.localr r r Zdask.optimizationr r Z dask.systemrZ dask.typingrZ dask.utilsrrr*r(unionrHHIGHEST_PROTOCOLrfr`rerr8rr6__annotations__r>Ztblib.pickling_supportZtblibZpickling_supportinstallr@ ImportErrorrMrQrSrOrrrcrrrrs`               j