>ieddlmZddlZddlZddlmZmZddlmZm Z ddl m Z ddl m Z ddlmZmZ d d Zd ZdS)) annotationsN)OpenFileget_fs_token_paths)infer_compression read_block)tokenize)delayed) is_integer parse_bytesF128 MiB10 kiBc t|ttttjfst dt|d|\}} t| dkrtd|z|Qt|trt|}t|st dt|}|*dggt| z} dggt| z} nvg} g} | D]ndkrt} n} | td d } | td | dkr+| g| g| |zr| |kr | | |zz }n|}d}dg}g}| |z |d zd z krZ||z }|t|||d|dz | |z |d zd z kZ|| |dz |rd |d<|dxxd zcc<| || |pt#t$g}t'| | | D]q\}}t)||fd|D}fdt'|||D}||r|r|durd}t|trt|}t-| d5}||}n]||} ||}|sn-|vr#||d dzz}n||z}E|}dddn #1swxYwY|r||| fS||fS)aU Given a path or paths, return delayed objects that read from those paths. The path may be a filename like ``'2015-01-01.csv'`` or a globstring like ``'2015-*-*.csv'``. The path may be preceded by a protocol, like ``s3://`` or ``hdfs://`` if those libraries are installed. This cleanly breaks data by a delimiter if given, so that block boundaries start directly after a delimiter and end on the delimiter. Parameters ---------- urlpath : string or list Absolute or relative filepath(s). Prefix with a protocol like ``s3://`` to read from alternative filesystems. To read from multiple files you can pass a globstring or a list of paths, with the caveat that they must all have the same protocol. delimiter : bytes An optional delimiter, like ``b'\n'`` on which to split blocks of bytes. not_zero : bool Force seek of start-of-file delimiter, discarding header. blocksize : int, str Chunk size in bytes, defaults to "128 MiB" compression : string or None String like 'gzip' or 'xz'. Must support efficient random access. sample : int, string, or boolean Whether or not to return a header sample. Values can be ``False`` for "no sample requested" Or an integer or string value like ``2**20`` or ``"1 MiB"`` include_path : bool Whether or not to include the path with the bytes representing a particular file. Default is False. **kwargs : dict Extra options that make sense to a particular storage connection, e.g. host, port, username, password, etc. Examples -------- >>> sample, blocks = read_bytes('2015-*-*.csv', delimiter=b'\n') # doctest: +SKIP >>> sample, blocks = read_bytes('s3://bucket/2015-*-*.csv', delimiter=b'\n') # doctest: +SKIP >>> sample, paths, blocks = read_bytes('2015-*-*.csv', include_path=True) # doctest: +SKIP Returns ------- sample : bytes The sample header blocks : list of lists of ``dask.Delayed`` Each list corresponds to a file, and each delayed object computes to a block of bytes from that file. paths : list of strings, only included if include_path is True List of same length as blocks, where each item is the path to the file represented in the corresponding block. z3Path should be a string, os.PathLike, list or tuplerb)modestorage_optionsrz%s resolved to no filesNzblocksize must be an integerinferzHCannot do chunked reads on compressed files. To read, set blocksize=NonesizezfBacking filesystem couldn't determine file size, cannot do chunked reads. To read, set blocksize=None.c g|] }d|d S)z read-block--).0otokens /lib/python3.11/site-packages/dask/bytes/core.py zread_bytes..s*:::a)a))%)):::c Xg|]&\}}}t|||'S) compression) dask_key_name)r) rrkeylr# delayed_read delimiterfspaths rrzread_bytes..s`    3 LT{;;;!        r Tr r") isinstancestrlisttupleosPathLike TypeErrorrlenOSErrorr r intr ValueErrorinfoappendr read_block_from_fileziprukeyrreadsplit)urlpathr(not_zero blocksizesampler# include_pathkwargsfs_tokenpathsoffsetslengthscompr blocksize1placeofflengthoutoffsetkeysvaluesf sample_buffnewr'r)r*rs ` ` @@@@r read_bytesrSsD gT5"+> ? ?OMNNN,W4QWXXXB% 5zzQ/'9::: i % % /#I..I)$$ <:;; ; NN 3%#e**$6(SZZ'* '* 'Dg%%(.." 2774==(D| E r"""r"""")#+y(8(8!%):!;JJ!*JcUlj1n%999Z'EJJs5zz***MM#b'CG"3444Ulj1n%999 dSWn---#CF1IIINIIIs###v&&&&/00L C #E7G < <  ff9dBGGDMM;PVWW::::6:::         !v66     6 % T>>F fc " " ) ((F b%( < < < % ffVnn  4&&..C C'''#))Iq*A*A!*DDyP$"-"3K 4%! % % % % % % % % % % % % % % %""sE!! 3;sA6OOOctj|5}|dkr"| |cdddSt||||cdddS#1swxYwYdS)Nr)copyr;r) lazy_filerJbsr(rPs rr8r8s 9  1 !88 668811111111!S"i00111111111111111111sAAA #A )NFr r NF) __future__rrUr/ fsspec.corerr fsspec.utilsrr dask.baser dask.delayedr dask.utilsr r rSr8rr rr^s"""""" 4444444466666666 ........   mmmm`11111r