>ie+RddlmZddlZddlZddlmZddlmZddlZ ddl m Z ddl m ZddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZmZdd lmZ ddZdZ ej!deddddddddde j"ddf dZ" ddZ#dZ$dZ%dS)) annotationsN)partial) zip_longest) open_files)compute) read_bytes)flatten) PANDAS_GE_200PANDAS_VERSION)dataframe_creation_dispatch) from_delayed)insert_meta_param_description make_meta)delayedrecordsTutf-8strictc N ||dk}|dkr|rtd| d<|o|dk d<t|df||| |j|d|pi} fdt| |D} |r)| t } t t| i| S| S) aWrite dataframe into JSON text files This utilises ``pandas.DataFrame.to_json()``, and most parameters are passed through - see its docstring. Differences: orient is 'records' by default, with lines=True; this produces the kind of JSON output that is most common in big-data applications, and which can be chunked when reading (see ``read_json()``). Parameters ---------- df: dask.DataFrame Data to save url_path: str, list of str Location to write to. If a string, and there are more than one partitions in df, should include a glob character to expand into a set of file names, or provide a ``name_function=`` parameter. Supports protocol specifications such as ``"s3://"``. encoding, errors: The text encoding to implement, e.g., "utf-8" and how to respond to errors in the conversion (see ``str.encode()``). orient, lines, kwargs passed to pandas; if not specified, lines=True when orient='records', False otherwise. storage_options: dict Passed to backend file-system implementation compute: bool If true, immediately executes. If False, returns a set of delayed objects, which can be computed at a later time. compute_kwargs : dict, optional Options to be passed in to the compute method compression : string or None String like 'gzip' or 'xz'. name_function : callable, default None Function accepting an integer (partition index) and producing a string to replace the asterisk in the given filename globstring. Should preserve the lexicographic order of partitions. Nr;Line-delimited JSON is only available withorient="records".orientlineswt)encodingerrors name_functionnum compressioncTg|]$\}}tt||%S)rwrite_json_partition).0outfiledkwargss 6lib/python3.11/site-packages/dask/dataframe/io/json.py zto_json..[sD    GQ &$%%a&99   ) ValueErrorr npartitionszip to_delayeddictlist dask_compute)dfurl_pathrrstorage_optionsrrrrcompute_kwargsrr$outfilespartss ` r%to_jsonr5s h })# u L   F83) 3F7O  # N    b  H    h 88   E  !!VVNL%:>::;;; r'c|5}|j|fi|dddn #1swxYwYtj|jSN)r5ospathnormpath)r/openfiler$fs r%r r gs  Q 1               7  HM * **s ""pandasiinferFc  dkdkrrtd|rdksstd|pi} durd d t trGts%tdttd t t j  |rt|d f||| d |} ra|\}}} |d }t j fd|Dt fdt||D}n |\}}d}d}dt|} t| |  t    fdt||D}nFt|df|d|}t j fd|D  fd|D}t!| S)a Create a dataframe from a set of JSON files This utilises ``pandas.read_json()``, and most parameters are passed through - see its docstring. Differences: orient is 'records' by default, with lines=True; this is appropriate for line-delimited "JSON-lines" data, the kind of JSON output that is most common in big-data scenarios, and which can be chunked when reading (see ``read_json()``). All other options require blocksize=None, i.e., one partition per input file. Parameters ---------- url_path: str, list of str Location to read from. If a string, can include a glob character to find a set of file names. Supports protocol specifications such as ``"s3://"``. encoding, errors: The text encoding to implement, e.g., "utf-8" and how to respond to errors in the conversion (see ``str.encode()``). orient, lines, kwargs passed to pandas; if not specified, lines=True when orient='records', False otherwise. storage_options: dict Passed to backend file-system implementation blocksize: None or int If None, files are not blocked, and you get one partition per input file. If int, which can only be used for line-delimited JSON files, each partition will be approximately this size in bytes, to the nearest newline character. sample: int Number of bytes to pre-load, to provide an empty dataframe structure to any blocks without data. Only relevant when using blocksize. encoding, errors: Text conversion, ``see bytes.decode()`` compression : string or None String like 'gzip' or 'xz'. engine : callable or str, default ``pd.read_json`` The underlying function that dask will use to read JSON files. By default, this will be the pandas JSON reader (``pd.read_json``). If a string is specified, this value will be passed under the ``engine`` key-word argument to ``pd.read_json`` (only supported for pandas>=2.0). include_path_column : bool or str, optional Include a column with the file path where each row in the dataframe originated. If ``True``, a new column is added to the dataframe called ``path``. If ``str``, sets new column name. Default is ``False``. path_converter : function or None, optional A function that takes one argument and returns a string. Used to convert paths in the ``path`` column, for instance, to strip a common prefix from all the paths. $META Returns ------- dask.DataFrame Examples -------- Load single file >>> dd.read_json('myfile.1.json') # doctest: +SKIP Load multiple files >>> dd.read_json('myfile.*.json') # doctest: +SKIP >>> dd.read_json(['myfile.1.json', 'myfile.2.json']) # doctest: +SKIP Load large line-delimited JSON files using partitions of approx 256MB size >> dd.read_json('data/file*.csv', blocksize=2**28) NrrzSJSON file chunking only allowed for JSON-linesinput (orient='records', lines=True).Tr9c|Sr7r)xs r%zread_json..s1r'zYPandas>=2.0 is required to pass a string to the `engine` argument of `read_json` (pandas=z is currently installed).)engine ) blocksizesampler include_pathrc3.K|]}|VdSr7r)r!ppath_converters r% zread_json..s-,N,N1^^A->->,N,N,N,N,N,Nr'c3VK|]#\}}|gt|zV$dSr7)len)r!rIchunkrJs r%rKzread_json..sP!!5=Q""#c%jj0!!!!!!r'r7c bg|]+\}}tt||  ,S)meta)rread_json_chunk) r!rNr9rrCrinclude_path_columnr$rQ path_dtypes r%r&zread_json..sc    t %GO $ $#       r'rt)rrrc38K|]}|jVdSr7)r9)r!r<rJs r%rKzread_json..s/(O(OA)?)?(O(O(O(O(O(Or'c tg|]4}tt||j5Sr)rread_json_filer9) r!r<rCrSr$rrrJrTs r%r&zread_json.. sc     $GN # ##qv&&      r'rP)r( isinstancestrr r rpd read_jsonrCategoricalDtyper r*rRrrrr )r0rrr1rErFrrrrQrCrSrJr$b_outfirstchunkspaths first_path flat_paths flat_chunksr4filesrTs `` `` ````` @r%r\r\ms?v })# u L    f ))) 4   &+Od""$$&#6 J~..JJJ  f555H     #,       #( E65'a11J,,N,N,N,N,N,N,NNNJ !!!!ADUFASAS!!!JJ"ME6J JJfoo <"#  D            +; CC       #      ((O(O(O(O(O(O(OOO                 D ) ) ))r'c tj|||} | d|| fddd|} | | jr|S|rt | |||} | S)NrrTrr)ioStringIOdecodeseekemptyadd_path_column) rNrrrC column_namer9rTr$rQsr/s r%rRrR1s ELL62233AFF1III  :)4 : :6 : :B BH @ RdJ ? ? Ir'cz|5}||f||d|} dddn #1swxYwY|rt| |||} | S)Nrg)rm) r<rrrCrnr9rTr$ open_filer/s r%rXrX@s Ei VI DfE D DV D DEEEEEEEEEEEEEEE@ RdJ ? ? Is   c ||jvrtd|d|jdi|tj|gt |z|iS)Nz(Files already contain the column name: 'z^', so the path column cannot use this name. Please set `include_path_column` to a unique name.)dtyper)columnsr(assignr[SeriesrM)r/rnr9rss r%rmrmHsrbj   {      29 O O RYvB/?u%M%M%MN O OOr') rNNTrrNNNr7)& __future__rrhr8 functoolsr itertoolsrr=r[ fsspec.corer dask.baserr. dask.bytesr dask.corer dask.dataframe._compatr r dask.dataframe.backendsr dask.dataframe.io.ior dask.dataframe.utilsrr dask.delayedrr5r register_inplacer\rRrXrmrr'r%rs"""""" !!!!!!""""""------!!!!!!@@@@@@@@??????------IIIIIIII     OOOOd+++ .-h77      <***87*FRV    PPPPPr'