o NrfN@sddlmZddlZddlmZmZddlmZddlZddl Z ddl m Z ddl mZzddlmZWn ey?dZYnwddZd d Zdd d Zd dZddZddZ    dddZeGdddeZeGdddeZdS)) annotationsN)Protocolruntime_checkable)uuid4)LocalFileSystem)VersioncCs|o t|tp t|S)z'Check if an fsspec file-system is local) isinstancer_is_local_fs_pyarrowfsr `/var/www/html/software/conda/envs/catlas/lib/python3.10/site-packages/dask/dataframe/io/utils.py _is_local_fss rcCs0|rt|dr t|jSt|dr|jdkSdS)z-Check if a pyarrow-based file-system is localr type_namelocalF)hasattrr r rr r r r r s     r c Cs(|dkrddlm}|j}ndd}|jduod|jv}|r@t|jdd}d d |d gD}d d |d gD}ni}i} tt|D]<} || } | j |vrp| j |vrjt j gd dj || j j} n|| j } nz|| j} Wn tyYqJw| | | j <qJ|r|D]} d| | <q| S)z4Convert a pyarrow.Schema object to pandas dtype dictZnumpy_nullabler)PYARROW_NULLABLE_DTYPE_MAPPINGcSs|SN)Zto_pandas_dtype)tr r r .sz%_get_pyarrow_dtypes..Nspandasutf8c Ss&i|]}|d|dd|dqS) field_namenameNZ numpy_typeget.0cr r r 4sz'_get_pyarrow_dtypes..columnsc SsBi|]}|ddvr|dr|d|dd|dddqS)Z pandas_type)datetimeZ datetimetzmetadatarrNtimezonerrr r r r8szM8[ns]dtypecategory)Zdask.dataframe.io.parquet.arrowrrr!jsonloadsdecoderangelenrpdSeriesdtZ tz_localizer$typeNotImplementedError)Zschema categoriesZ dtype_backendrZ type_mapperZhas_pandas_metadataZpandas_metadataZpandas_metadata_dtypestzZdtypesifieldZ numpy_dtypecatr r r _get_pyarrow_dtypes's@         r5csfdd|Dfdd|pgD}t|dkrd}nt|dkr3|d}|ddkr2|d|_ntjj||d }tj|d }|rH||j_|S) aNGet the final metadata for the dask.dataframe Parameters ---------- to_read_columns : list All the columns to end up with, including index names file_dtypes : dict Mapping from column name to dtype for every element of ``to_read_columns`` index_cols : list Subset of ``to_read_columns`` that should move to the index column_index_names : list The values for df.columns.name for a MultiIndex in the columns, or df.index.name for a regular Index in the columns Returns ------- meta : DataFrame c s$i|]}|tjg|ddqS)Zint64r#)r+r,rr) file_dtypesr r rrsz%_meta_from_dtypes..csg|]}|qSr )popr)datar r usz%_meta_from_dtypes..rNZ__index_level_0__)names)index)r*rr+Z MultiIndexZ from_arraysZ DataFramerr;)Zto_read_columnsr6Z index_colsZcolumn_index_namesZindexesr<dfr )r8r6r _meta_from_dtypes]s     r>cCstjS)z0Simple utility function to get random hex string)rhexr r r r _guidsr@cCs|dur|S||S)z5Helper function to place an object on a context stackN) enter_context)objstackr r r _set_contexts rDc  sdurfdd|DS|pi}|dd}|dkrbdurbtsbttjtdkrb|ddpAdgt|}dd }|d krTtd |d fd dt ||DSdurqfd d|DSfdd|DS)aReturn a list of open-file objects given a list of input-file paths. WARNING: This utility is experimental, and is meant for internal ``dask.dataframe`` use only. Parameters ---------- paths : list(str) Remote or local path of the parquet file fs : fsspec object, optional File-system instance to use for file handling context_stack : contextlib.ExitStack, Optional Context manager to use for open files. open_file_func : callable, optional Callable function to use for file opening. If this argument is specified, ``open_file_func(path, **kwargs)`` will be used to open each file in ``paths``. Default is ``fs.open``. precache_options : dict, optional Dictionary of key-word arguments to use for precaching. If ``precache_options`` contains ``{"method": "parquet"}``, ``fsspec.parquet.open_parquet_file`` will be used for remote storage. **kwargs : Key-word arguments to pass to the appropriate open function Ncs"g|] }t|fiqSr )rDrpath) context_stackkwargsopen_file_funcr r r9sz%_open_input_files..methodparquetz 2021.11.0 row_groups cache_typepartsz<'parts' `cache_type` required for 'parquet' precaching, got .cs.g|]\}}ttj|f|dqS))r rL)rDfsspec_parquetZopen_parquet_file)rrFZrgsrGr rHr r r9s cs$g|]}tj|fiqSr rDopenrErQr r r9s$cs"g|] }tt|fiqSr rRrE)rGrHr r r9s") copyr7rrfsspec __version__updater* ValueErrorzip) pathsr rGrIZprecache_optionsrHZprecacherLrMr )rGr rHrIr _open_input_filess4#     r[c@s,eZdZdZeddZddZddZdS) DataFrameIOFunctionzkDataFrame IO function with projectable columns Enables column projection in ``DataFrameIOLayer``. cCt)z$Return the current column projectionr/selfr r r rszDataFrameIOFunction.columnscCr])zUReturn a new DataFrameIOFunction object with a new column projection r^)r`rr r r project_columnssz#DataFrameIOFunction.project_columnscOr])z Return a new DataFrame partitionr^)r`argsrHr r r __call__szDataFrameIOFunction.__call__N)__name__ __module__ __qualname____doc__propertyrrarcr r r r r\s   r\c@s eZdZdddZdddZdS) SupportsLockreturnobjectcCdSrr r_r r r acquirezSupportsLock.acquirecCrlrr r_r r r releasernzSupportsLock.releaseN)rjrk)rdrerfrmror r r r ris rir)NNNN) __future__rr&typingrruuidrrUZpandasr+Zfsspec.implementations.localrZpackaging.versionrZfsspec.parquetrKrP ImportErrorrr r5r>r@rDr[r\rir r r r s8      6*  N