0Fie*VddlmZddlZddlmZmZddlmZddlm Z ddl m Z ddl Z ddlZn #e$rdZYnwxYwGddeZGd d e ZGd d e Zeejneja da d ZdZdZdZdZdZdZdZdZdZ dZ!dZ"dZ#dZ$dZ%dZ&dS)) annotationsN)IntEnumauto)uname) NamedTuple)parseceZdZeZ eZ eZ eZ eZ eZ dS) NVMLStateN) __name__ __module__ __qualname__r UNINITIALIZED INITIALIZEDDISABLED_PYNVML_NOT_AVAILABLEDISABLED_CONFIGDISABLED_LIBRARY_NOT_FOUND DISABLED_WSL_INSUFFICIENT_DRIVERRY[[3PPrc,dtjvS)zCheck if we are in Windows Subsystem for Linux; some PyNVML queries are not supported there. Taken from https://www.scivision.dev/python-detect-wsl/ zmicrosoft-standard)rreleaserrr_in_wslr*@s 577? 22rcttjtjtjtjhvrdSttjkrttj krdSttj kr-tj dstjadSttjkrttj ksttj kr tjn9#tjtjtjf$rtjaYdSwxYwt'rVt)tjt)t.krtjadSddlm}tjatj a|dSt5dtdtd)zIdempotent (per-process) initialization of PyNVML Notes ----- Modifies global variables NVML_STATE and NVML_OWNER_PIDNzdistributed.diagnostics.nvmlradd_gpu_metricsz+Unhandled initialisation state (NVML_STATE=z, NVML_OWNER_PID=))r#r rrrrrr$r%r&rdaskconfiggetpynvmlnvmlInitNVMLError_LibraryNotFoundNVMLError_DriverNotLoadedNVMLError_Unknownr* parse_versionnvmlSystemGetDriverVersiondecodeMINIMUM_WSL_VERSIONdistributed.workerr- RuntimeErrorr,s r init_oncer=Gs/!,2   y, , ,29;;1N1N y. . .t{&88 .. i+++")++0M0M y. . .  O      ,  ,  $     #=J FF   99   - / / 6 6 8 8  - . .//#CJ F : : : : : :#.JY[[N O      Pz P Pn P P P   s4D2D>=D>cdttsdStjS)Nr)r=r'r2nvmlDeviceGetCountrrrdevice_get_countr@}s- KKK   +q(***rc pt}ttjkrt dttjkrt dttjkrt dtdttjkrt d|dkrt d ttttj dd d }n#t $rd}YnwxYwt#j|S) Nz8NVML monitoring requires PyNVML and NVML to be installedz$PyNVML is installed, but NVML is notz3Outdated NVIDIA drivers for WSL, please upgrade to z or newerzKPyNVML monitoring disabled by 'distributed.diagnostics.nvml' config settingrzNo GPUs availableCUDA_VISIBLE_DEVICES,)r@r#r rr<rrr:rnextmapintr%environr1split ValueErrorr2nvmlDeviceGetHandleByIndex)countgpu_idxs r_pynvml_handlesrNs:   EY<<<UVVV y; ; ;ABBB yA A A ." . . .    y0 0 0     !./// C(>CCII#NNOOGG   GGG 0999s>AD D! D!ctttdrtj|}ntj|}t d|DS)a)Check whether the current process is same as that of handle Parameters ---------- handle : pyvnml.nvml.LP_struct_c_nvmlDevice_t NVML handle to CUDA device Returns ------- out : bool Whether the device handle has a CUDA context on the running process. 'nvmlDeviceGetComputeRunningProcesses_v2c3NK|] }tj|jkV!dSN)r%r&pid).0procs r z+_running_process_matches..s/EE4ry{{dh&EEEEEEr)r=hasattrr2rP$nvmlDeviceGetComputeRunningProcessesany)handlerunning_processess r_running_process_matchesr\sdKKKv@AAP"J6RR"GOO EE3DEEE E EErc ttr\ttD]?}t j|} t j|\}}n!#tj$rtj}YnwxYw|tj krtt j |D]v} t j ||}n#tj $rY*wxYwt|r9t j|}tdt!|||ccSwt|r6t j|}tdt!||cSAtdS)aKCheck whether the current process already has a CUDA context created. Returns ------- out : CudaContext Object containing information as to whether the current process has a CUDA context created, and in the positive case containing also information about the device the context belongs to. T)rrr)r r!rrF)r )r=r'ranger@r2rKnvmlDeviceGetMigModeNVMLError_NotSupportedNVML_DEVICE_MIG_DISABLENVML_DEVICE_MIG_ENABLEnvmlDeviceGetMaxMigDeviceCount#nvmlDeviceGetMigDeviceHandleByIndexNVMLError_NotFoundr\nvmlDeviceGetUUIDrr)indexrZmig_current_modemig_pending_moder mig_handlers rhas_cuda_contextrlsKKK+--..  E6u==F B5;5PQW5X5X2 "2"20 B B B#)#A    B6#@@@!&v'LV'T'T!U!UI!%+%O"I&& "4!!! !0 ;;%7 CC*(,(6%))))   ",F33!3F;;D&$($25$Q$Q$Q 5 ) ) ))s$A))BB>CC&%C&ct t|}tj|}tj|}ns#t $rft |tr|nt|d}tj|}tj |}tj|}YnwxYwt||S)a`Get both device index and UUID from device index or UUID Parameters ---------- device : int, bytes or str An ``int`` with the index of a GPU, or ``bytes`` or ``str`` with the UUID of a CUDA (either GPU or MIG) device. Returns ------- out : CudaDeviceInfo Object containing information about the device. Examples -------- >>> get_device_index_and_uuid(0) # doctest: +SKIP {'device-index': 0, 'uuid': b'GPU-e1006a74-5836-264f-5c26-53d19d212dfe'} >>> get_device_index_and_uuid('GPU-e1006a74-5836-264f-5c26-53d19d212dfe') # doctest: +SKIP {'device-index': 0, 'uuid': b'GPU-e1006a74-5836-264f-5c26-53d19d212dfe'} >>> get_device_index_and_uuid('MIG-7feb6df5-eccf-5faa-ab00-9a441867e237') # doctest: +SKIP {'device-index': 0, 'uuid': b'MIG-7feb6df5-eccf-5faa-ab00-9a441867e237'} utf-8r^) r=rGr2rKrgrJ isinstancebytesnvmlDeviceGetHandleByUUIDnvmlDeviceGetIndexr)devicer device_handler uuid_handles rget_device_index_and_uuidrvs2KKK 56{{ 9,GG ' 66 555#FE22Nvvfg8N8N6t<< 0== ' 44 5 t, ? ? ??s7AA-B87B8cZt t|}tj|}nK#t$r>t |t r|nt |d}tj|}YnwxYw tj|S#tj $rddgcYSwxYw)aXGet MIG mode for a device index or UUID Parameters ---------- device: int, bytes or str An ``int`` with the index of a GPU, or ``bytes`` or ``str`` with the UUID of a CUDA (either GPU or MIG) device. Returns ------- out : list A ``list`` with two integers ``[current_mode, pending_mode]``. rnr) r=rGr2rKrJrorprqr`ra)rsrrZrs rget_device_mig_moderx sKKK86{{ 2<@@ 888#FE22Nvvfg8N8N1$778*6222  (1v s"#4AA<;A<BB*)B*cb tj|jS#tj$rYdSwxYwrR)r2nvmlDeviceGetUtilizationRatesgpurahs r_get_utilizationr~(s@3A66::  (tt ..cb tj|jS#tj$rYdSwxYwrR)r2nvmlDeviceGetMemoryInfousedrar|s r_get_memory_usedr/s@-a0055  (ttrcb tj|jS#tj$rYdSwxYwrR)r2rtotalrar|s r_get_memory_totalr6s@-a0066  (ttrc tj|S#t$rtj|cYStj$rYdSwxYwrR)r2nvmlDeviceGetNamer9AttributeErrorrar|s r _get_namer=sr'**11333 +++'*****  (tts%(AAAc\t}t|t|dS)N) utilizationz memory-used)rNr~rr|s r real_timerFs2A'**'**  rc\t}t|t|dS)N)z memory-totalname)rNrrr|s rone_timerNs0A)!,,!   r)' __future__rr%enumrrplatformrtypingrpackaging.versionrr7r/r2 ImportErrorr rrrrr#r$r:r'r*r=r@rNr\rlrvrxr~rrrrrrrrrsX"""""" 444444 MMMM FFF W W W W W W W W!!!!!Z!!! .....*...~++   #=QQQ 3333 3 3 l+++::::FFF*******Z&@&@&@R6s /99