0Fie UdZddlmZddlZddlmZmZddlmZddl m Z ddl m Z ddl mZmZmZmZdd lmZdd lmZddlZdd lmZdd lmZmZerdd lmZdZde d<GddeZ!e!deee!deee!deee!dej"ej#dZ$de d<ee%5ddl&Z& e&j"e'dn#e($r e%dwxYwe!de&j"e&j#e$d<e$de$d<dddn #1swxYwYee%5ddl)Z)ee)j*edkr e%dddl+Z)e!de)j,j"e e)j,j#de$d<e$de$d<dddn #1swxYwYee%5ddl-Z-ee-j*ed kr e%d!d"Z.d#Z/e!d$e.e/e$d$<dddn #1swxYwYdFd)Z0dGd/Z1ej2d0d1d1d2d3dd4dHd>Z3ej2d?dIdEZ#dS)JzY Record known compressors Includes utilities for determining whether or not to compress ) annotationsN)CallableIterable)suppress)partial)randint) TYPE_CHECKINGAnyLiteral NamedTuple)parse)identity) context_meter)ensure_memoryviewnbytes) TypeAliasbytes | bytearray | memoryviewrAnyBytesc.eZdZUded<ded<ded<dS) Compressionz None | strnamezCallable[[AnyBytes], AnyBytes]compress decompressN)__name__ __module__ __qualname____annotations__@lib/python3.11/site-packages/distributed/protocol/compression.pyrrs6,,,,......rrzlib)NFautor!z.dict[str | None | Literal[False], Compression] compressionsrzNeed snappy >= 0.5.3snappyr"z0.23.1zNeed lz4 >= 0.23.1lz4T)return_bytearrayz0.9.0zNeed zstandard >= 0.9.0ctjtjdtjd}||S)Nzdistributed.comm.zstd.levelzdistributed.comm.zstd.threads)levelthreads) zstandardZstdCompressordaskconfiggetr)datazstd_compressors r zstd_compressr1[sS#2+//"?@@KOO$CDD   ''---rcRtj}||SN)r*ZstdDecompressorr)r/zstd_decompressors r zstd_decompressr6bs%%688 ++D111rzstdkeystrreturn str | Nonec tj|} t|jS#t $r>ddtD}td|d|d|dwxYw)zFetch and validate compression settings, with a nice error message in case of failure. This also resolves 'auto', which may differ between different hosts of the same cluster. ,c34K|]}t|VdSr3)repr).0ns r z+get_compression_settings..rs(77Qa777777rzInvalid compression setting =z. Valid options are .)r,r-r.r#rKeyErrorjoin ValueError)r8rvalids r get_compression_settingsrIis ;??3  D D!&&    77,77777 S3 S S S S5 S S S    s 3AA;b memoryviewsizeintrAc|dkr|dksJ|dks|dkrtdSg}|j|z }td|}t|dz D]D}td|}t ||z|}|||||}E|||||z|dkr|dStd|S)zSample a bytestring from many locations Parameters ---------- b : full memoryview size : int target size of each sample to collect (may be smaller if samples collide) n : int number of samples to collect rr)rKrrrangeminappendrF) rJrLrAparts max_startstart_ next_startends r byte_samplerYxs 199a qyyAFF# E4I Ay ! !E 1q5\\Q ** %$, ++ QuSy\""" LL554<'()))AvvQx#((5//***rri'gffffff?)min_size sample_sizensamples min_ratio compressionpayloadr[r\r]r^floatr_str | None | Literal[False]tuple[str | None, AnyBytes]ct|}|jsd|fS|t|cxkrdksnd|fSt|}t |||}t ||||jzkr9||} t | ||jzkr |j| fSd|fS)aMaybe compress payload 1. Don't compress payload if smaller than min_size 2. Sample the payload in spots, compress those, and if it doesn't compress to at least to the original, return the original 3. Then compress the full original; it doesn't compress at least to , return the original 4. Return the compressed output Returns ------- - Name of compression algorithm used - Either compressed or original payload Nl)r#rrrrYlenr) r`r[r\r]r^r_compmvsample compresseds r maybe_compressrjs0  $D 9W} w 0 0 0 05 0 0 0 0W} 7 # #B[( 3 3F 4== !!Y%>>>]]2&& z??i")3 3 39j( ( =rrheaderdict[str, Any]framesIterable[AnyBytes]list[AnyBytes]cBdt|d|DS)z8Decompress frames according to information in the headercTg|]%\}}t||&Sr)r#r)r@rframes r zdecompress..s?    D% T%%e,,   rr_)zip)rkrms r rrs2  vm4f==   r)r8r9r:r;)rJrKrLrMrArMr:rK)r`rr[rMr\rMr]rMr^rar_rbr:rc)rkrlrmrnr:ro)4__doc__ __future__rr!collections.abcrr contextlibr functoolsrrandomrtypingr r r r packaging.versionr parse_versiontlzrr,distributed.metricsrdistributed.utilsrrtyping_extensionsrrrrrrr# ImportErrorr$rK TypeErrorr% __version__ lz4.blockblockr*r1r6rIrYmeterrjrrr rs@ #""""" ........::::::::::::444444 ------77777777,++++++76666/////*/// +dHh / / ;tXx 0 0 Kh 1 1 K t ? ? @@ Xk22MMM2 3(((( 222k01112)[6?FDUVVL'1L222222222222222"Xk//JJJ}S_%% h(?(???k.///%+    $t<<<  L(.L'///////////////,XkOO}Y*++mmG.D.DDDk3444...222';v}oNNL%OOOOOOOOOOOOOOO*     ++++@Z  /5*****! *Z\""#"sP<D!CD!C--(D!!D%(D%7A+F..F25F2AHHH