+dopddlmZddlmZmZddlmZmZGddeZdZ Gdd eZ d S) )Version) DataSourceSchema) RemoteSource get_partitioncteZdZdZdZdZfdZdZdZdZ dZ d Z e d Z e d ZxZS) RemoteDataFramezDataframe on an Intake serverremote_dataframe dataframec jtt|j||fi||d|_t |d|_|d|_|d|_|dd|_ t|j|j|j|j|_ d|_ dS)N npartitionsshapemetadatadtypeverifyFrextra_metadatarr) superr __init__rtuplerrrgetrr_schemar )selfurlheaderskwargs __class__s :lib/python3.11/site-packages/intake/container/dataframe.pyrzRemoteDataFrame.__init__s-ot$$-c7EEfEEE!-06'?++ z* G_ jj511 $*:4=`d`jrvr|}}} cXddlm}ddljfdtjD_t jt dkr |j_n&|jj _j S)Nrc g|]=}tjjjj|>S)delayedrrr _source_id container).0idaskrs r z2RemoteDataFrame._load_metadata..#sVTTTvw5$,,}55dh do_c_mopqqTTTr z2.5.0) verify_meta) dask.dataframer dask.delayedrangerpartsr __version__ from_delayedrr)rddr)s` @r_load_metadatazRemoteDataFrame._load_metadatas###### > VTTTTT|ABFBR|S|STTTDJt'((77+;+;; V!#!r _data_to_source)sourcepathrdfs r_persistzRemoteDataFrame._persist9s_ !!BB"   BBB .r4BB6BBBs 88c ddlm}t|st ddlm}n#t $rt dwxYwt|ds||d}|j |fi|||i}|S)Nr) ParquetSourcezPPlease install intake-parquet to use persistence on dataframe container sources.rr)r)meta) r,r is_dataframe_likerIintake_parquetrP ImportErrorhasattr from_pandas to_parquet)rMrLrr2rPrKs rrJzRemoteDataFrame._data_to_sourceNs###### $$ &% % u 4 4 4 4 4 4 4 u u ustt t ur=)) 322B d%%f%%%t"--- s%?)__name__ __module__ __qualname____doc__namer&rr3r9r>rBrF staticmethodrNrJ __classcell__rs@rr r s'' DI   '''(((CC\C(  \     r r cttfddDo+tfddDotd S)z>Looks like a Pandas DataFrame Copied from dask.utils c38K|]}t|VdSr5rU)r'r\typs r z$is_dataframe_like..es-SSdwsD!!SSSSSSr )groupbyheadmergemeanc38K|]}t|VdSr5rb)r'r\rMs rrdz$is_dataframe_like..esQ\G\Grv\cdfhl\m\m\G\G\G\G\G\Gr )dtypesr)typeallrU)rMrcs`@rrRrR_s r((C SSSS.RSSS S S eX[\G\G\G\G{F\G\G\GYGYG ePWX[]dPePeLeer cJeZdZdZdZdZd fd ZdZdZdZ d Z d Z xZ S) GenericDataFrameaCreate partitioned dataframe from a set of files and any reader This data-source allows you to specify any reader to create dataframes. The reader must take an open file-like object as input, and output a pandas.DataFrame. Parameters ---------- urlpath: str Location of data. May be local files or remote with a protocol specifier. May be a list of files or a glob pattern to be expanded. reader: func f(open_file, **kwargs) -> pandas.DataFrame storage_options: dict Passed to the file-system backend to open files; typically includes credentials for remote storage kwargs: Passed to reader function generic_dataframer Nc ||_||_|pi|_|}t |di||_d|_dS)Nr)r) rreaderstorage_optionscopyrrpoprr )rurlpathrqrrrrs rrzGenericDataFrame.__init__sf .4" &**Z"<"<=== r cddlm}ddlddlm}|jfij_djfdjD_| j_jj _ dtjj f_ jj_t#j jjj _jS)Nr) open_filescd|5}||fi|}|j|d<|cdddS#1swxYwYdS)NrL)rL) open_filerqrofrMs r read_a_filez4GenericDataFrame._load_metadata..read_a_files bVB))&))&^6                   s %))cdg|],}|jj-Sr#)r$rqr)r'ryr)r{rs rr*z3GenericDataFrame._load_metadata..s9uuu]f3$,,{33It{DKXXuuur r)r,r r-fsspecrwrrrfilesr/r1rlencolumnsrrjto_dictrrrr)rr2rwr)r{s` @@rr3zGenericDataFrame._load_metadatas!######%%%%%%ZAAD,@AA     > BuuuuuujnjtuuuDJ__TZ88DN#~9D DN$: ; ;t}dhdnvzwABBBDL|r ch||j|Sr5r6r8s rr9zGenericDataFrame._get_partitionr:r c\||jSr5r<r=s rr>zGenericDataFrame.readr?r c8||jSr5rAr=s rrBzGenericDataFrame.to_daskrCr cd|_dSr5rEr=s rrFzGenericDataFrame._closerGr r5) rXrYrZr[r\r&rr3r9r>rBrFr^r_s@rrnrnhs( DI,'''(((r rnN) packaging.versionrintake.source.baserrbaserrr rRrnr#r rrs&%%%%%11111111--------NNNNNlNNNbeeeDDDDDzDDDDDr