0Fie ddlmZddlZddlZddlmZmZmZddlm Z m Z ddl Z ddl m Z e je jdZddDZd ed <d ed < ddlZd1dZn#e$rd1dZYnwxYwd2dZefd3dZd4dZd4dZe d d dd5d%Ze d d&d6d)Zd d dd*Zd7d-Zejd.zZd8d0ZdS)9) annotationsN) CollectionIterableSequence)Literaloverloadnbyteszdistributed.comm.shardci|]}d|zd S)z max_%s_leni).0xs :lib/python3.11/site-packages/distributed/protocol/utils.py rs,&'\A )strbinarraymapextFstrict_map_keyrawnintreturn memoryviewc:tj|fdjS)Nu1)dtype)numpyemptydatars r host_arrayr$s{A4t,,,11rc:tt|SN)r bytearrayr#s rr$r$#s)A,,'''rbuffers(Iterable[bytes | bytearray | memoryview]cd|D}ttd|D}d}|D].}|d||||jz<||jz }/|S)Nc,g|]}t|Sr )r)r bufs r z+host_array_from_buffers..*s . . .s:c?? . . .rc3$K|] }|jV dSr&r )r mvs r z*host_array_from_buffers..+s$11r111111rrB)r$sumcastr )r(mvsoutoffsetr/s rhost_array_from_buffersr7's / .g . . .C S11S11111 2 2C F+-773<<FVbi' '(") Jrframebytes | memoryviewlist[memoryview]c|pt}tj|krgSjjz}|jzfdt d|DS)z Split a frame into a list of frames of maximum size This helps us to avoid passing around very large bytestrings. Examples -------- >>> frame_split_size([b'12345', b'678'], n=3) # doctest: +SKIP [b'123', b'45', b'678'] c*g|]}||zSr r )r ir8items_per_shards rr-z$frame_split_size..Is( V V VqE!a/)) * V V Vrr)BIG_BYTES_SHARD_SIZErr itemsizerange)r8rnitemsr>s` @rframe_split_sizerC3su ! !A u  E |qw \U^ +F5>)O V V V V VE!V_4U4U V V VVrframes*Collection[bytes | bytearray | memoryview]bytescxt|}tt|}tjd|d|g|RS)NQ)lenrr structpack)rDnframes nbytes_framess rpack_frames_preluderNLs>&kkG''M ;~7~~~w ? ? ? ??rcLdt|g|S)zPack frames into a byte-like object This prepends length information to the front of the bytes-like object See Also -------- unpack_frames r)joinrN)rDs r pack_framesrQRs' 88(00:6: ; ;;r) remainderpartialbbytes | bytearray | memoryviewrRboolrSLiteral[False]cdSr&r rTrRrSs r unpack_framesrZ^ Cr)rR Literal[True]"tuple[list[memoryview], list[int]]cdSr&r rYs rrZrZhr[rct|}d}tj|}tj||\}tj||||}g}|d|zz}|j} d} g} |D]Z} |r2|| kr,| |t |dn%|| z} |||| | }[| | ksJ|r|||d|r|| fS|S)aUnpack bytes into a sequence of frames This assumes that length information is at the front of the bytestring, as performed by pack_frames Parameters ---------- b: packed frames, as returned by :func:`pack_frames` remainder: If True, return one extra frame at the end which is the continuation of a stream created by concatenating multiple calls to :func:`pack_frames`. This last frame will be empty at the end of the stream. partial: If True, allow for b to contain less frames than what the preamble indicates; return a tuple of ([frames so far], [lengths of missing frames]) See Also -------- pack_frames rHrN)rrJcalcsize unpack_fromr extendrIappend) rTrRrSfmtfmt_sizen_frameslengthsrDstartnbendmissing_lengthslengths rrZrZrs2, 1 A Cs##H$S!,,KX H!3c!3!3QAAG F H %E B CO  u{{  " "73v;;==#9 : : : Efn ac l### "9999! ai   && rr4Sequence[memoryview]c n|sttSt|dkr|dS|d}t|tst dt ||j}|j}d}d}t|D]\}}t|ts!t |dt ||j dkrH|j|urt|d|jd||j st|d|j dkrt|d|j d |j|krt|d |jd|t|}|dkr|}n(||z} || krtd |d | d d|| z d||j z }|dkrt|dksJ|S|dks Jdt|d} t| } || z } | | | |z|S)a Zero-copy "concatenate" a sequence of contiguous memoryviews. Returns a new memoryview which slices into the underlying buffer to extract out the portion equivalent to all of ``mvs`` being concatenated. All the memoryviews must: * Share the same underlying buffer (``.obj``) * When merged, cover a continuous portion of that buffer with no gaps * Have the same strides * Be 1-dimensional * Have the same format * Be contiguous Raises ValueError if these conditions are not met. r`rzExpected memoryview; got z: expected memoryview; got z#: memoryview has different buffer: z vs z: memoryview non-contiguousz: memoryview has z dimensions, not 1z: inconsistent format: z memoryview z2 does not start where the previous ends. Expected rz , starts z byte(s) away.z#Underlying buffer is null pointer?!r1)rr'rI isinstance TypeErrortypeobjformat enumerater ValueError contiguousndimaddress_of_memoryviewr3) r4firstrsrtfirst_start_addrr r=r/ start_addr expected_addrbase_mvbase_start_addr start_indexs rmerge_memoryviewsrs" ')++&&& 3xx1}}1v FE eZ ( (CADKKAABBB )C \F F32"j)) IqGGT"XXGGHH H 9>>  6  NNNNsNN } @>>>?? ? 7a<<OOBGOOOPP P 9  QQ")QQQQRR R*2.. q )  ,v5M]** e!ee -dee:D}:Teee ") {{5zzQ q "G oo""3''G+G44O"_4K ;v!55 6 ; ;F C CCrr`r/c t|}tj|S#t$rYnwxYw ddl}n!#t $rtd|dwxYw||j ddS)z Get the pointer to the first byte of a memoryview's data. If the memoryview is read-only, NumPy must be installed. rNz+Cannot get address of read-only memoryview z since NumPy is not installed.r") one_byte_carr from_bufferctypes addressofrqr ImportErrorrvasarray__array_interface__)r/carrnps rryrys &((,,%%%            \" \ \ \     ::b>> -f 5a 88s0 ==AA$)rrrr)r(r)rr)r8r9rrrr:)rDrErrF)rTrUrRrVrSrWrr:)rTrUrRrVrSr\rr])r4rnrr)r/rrr) __future__rrrJcollections.abcrrrtypingrrdaskdistributed.utilsr utils parse_bytesconfiggetr? msgpack_optsr r$rr7rCrNrQrZrc_byterryr rrrsf"""""" ::::::::::$$$$$$$$ $$$$$$z--dkoo>V.W.WXX+P "'  U ( LLL22222((((((((((    )=WWWWW2@@@@ < < < < #    #(33333lGDGDGDGDT !  9 9 9 9 9 9s4A== B B