HR-eL x d Z ddlZddlZddlZddlmZ ddlmZ ddl m
Z
dZ g Z d Z
dd Zdd
Zd Zd Zd
ZdS )z
This package contains functions for reading and writing Parquet
tables that are not meant to be used directly, but instead are
available as readers/writers in `astropy.table`. See
:ref:`astropy:table_io` for more details.
N)
minversion)AstropyUserWarning)NOT_OVERWRITING_MSGs PAR1c |\ | }n# t $ r Y dS w xY w| d }| | |t k S || d S dS )a Checks if input is in the Parquet format.
Parameters
----------
origin : Any
filepath : str or None
fileobj : `~pyarrow.NativeFile` or None
*args, **kwargs
Returns
-------
is_parquet : bool
True if 'fileobj' is not None and is a pyarrow file, or if
'filepath' is a string ending with '.parquet' or '.parq'.
False otherwise.
NF )z.parquetz.parq)tellAttributeErrorreadseekPARQUET_SIGNATUREendswith)originfilepathfileobjargskwargspos signatures 7lib/python3.11/site-packages/astropy/io/misc/parquet.pyparquet_identifyr s " ,,..CC 55 LLOO S --- !6777us
''Fc
%&' t \ }}}t | t t j f s t | d st
d | | }|j $d |j D } ni } ddl
m }
m } m
} m}
i }d| v rJ| d d }| | }d |v r|d }nd}d
|j D }d }d |v r,d
}|d }|D ] }t' || D ]&||&< t) | '|' | |' | t1 t2 'fd| D 'g }'D ]7&&fd| D }| | 8|st9 d |rIt1 |d D ]!}|'vr|d | "|s | | || }|j }nd}g }|D ]&| & j! }d}t ||j" r|j# } |j$ f}n t ||j% r|j# } n|} | |& |' fvrT|(|( | ) n)|( | ) |f d& } | | v rtU | | } nc|r!d} tW j, d| dtZ n@t] d |& D } tW j, d| d| dtZ | |& k rd| nd| } ||( | |( | |f |r@t_ j0 dt1 tc || } | | | }!n | | }!tc || D ]\ &%|& 2 }"| & j! }||& |' fv r|"3 % }"nt ||j" rti |" dk r_t_ j5 |" }"|j# |& |' fv r|"3 %d }"nt_ j0 d% }"npt ||j% r[|j# |& |' fv r,t_ j6 %fd |"D t^ j7 }"|!8 |
&|" |dd |d! D }#|!j9 D ]8}"d"D ]3}$|$|#|"j: v r"tw |"|$|#|"j: |$ 49|
< |! }!|!S )#a
Read a Table object from a Parquet file.
This requires `pyarrow `_
to be installed.
The ``filters`` parameter consists of predicates that are expressed
in disjunctive normal form (DNF), like ``[[('x', '=', 0), ...], ...]``.
DNF allows arbitrary boolean logical combinations of single column
predicates. The innermost tuples each describe a single column predicate.
The list of inner predicates is interpreted as a conjunction (AND),
forming a more selective and multiple column predicate. Finally, the most
outer list combines these filters as a disjunction (OR).
Predicates may also be passed as List[Tuple]. This form is interpreted
as a single conjunction. To express OR in predicates, one must
use the (preferred) List[List[Tuple]] notation.
Each tuple has format: (``key``, ``op``, ``value``) and compares the
``key`` with the ``value``.
The supported ``op`` are: ``=`` or ``==``, ``!=``, ``<``, ``>``, ``<=``,
``>=``, ``in`` and ``not in``. If the ``op`` is ``in`` or ``not in``, the
``value`` must be a collection such as a ``list``, a ``set`` or a
``tuple``.
For example:
.. code-block:: python
('x', '=', 0)
('y', 'in', ['a', 'b', 'c'])
('z', 'not in', {'a','b'})
Parameters
----------
input : str or path-like or file-like object
If a string or path-like object, the filename to read the table from.
If a file-like object, the stream to read data.
include_names : list [str], optional
List of names to include in output. If not supplied, then
include all columns.
exclude_names : list [str], optional
List of names to exclude from output (applied after ``include_names``).
If not supplied then no columns are excluded.
schema_only : bool, optional
Only read the schema/metadata with table information.
filters : list [tuple] or list [list [tuple] ] or None, optional
Rows which do not match the filter predicate will be removed from
scanned data. See `pyarrow.parquet.read_table()` for details.
Returns
-------
table : `~astropy.table.Table`
Table will have zero rows and only metadata information
if schema_only is True.
r
z5pyarrow can only open path-like or file-like objects.Nc f i | ].\ }}| d | d /S zUTF-8)decode.0kvs r
z&read_table_parquet.. s6
W
W
Wtq!ahhw '!2!2
W
W
W r )ColumnTablemeta serializetable_meta_yaml
r# c i | ]}||S r( )r names r r z&read_table_parquet.. s >>>$>>>r F__serialized_columns__Tc g | ]}|v | S r( r( )r x use_namess r
z&read_table_parquet.. s PPPQi q r c & g | ]
\ }}|k |S r( r( )r ncolr) s r r. z&read_table_parquet.. s" KKKvq#ts{{{{{r z3No include_names specified were found in the table.)columnsfilters table::len::
zNo z1 found in metadata. Guessing {strlen} for schema.c 3 X K | ]%}t | V &d S )N)lenas_py)r rows r z%read_table_parquet.. s2 DDcSYY[[))DDDDDDr z* found in metadata. Using longest string (z
characters).Uz|S)dtype)datar# )r# c : g | ]}| S r( astype)r r9 dts r r. z&read_table_parquet.. s# #B#B#BsCJJrNN#B#B#Br )r) r= c i | ] }|d | S )r) r( )r r, s r r z&read_table_parquet..& s BBBqy!BBBr datatype) descriptionformatunitr# )= get_pyarrow
isinstancestrosPathLikehasattr TypeError read_schemametadataitems
astropy.tabler! r" r# r$ popsplitget_header_from_yamlnames
_get_namessetvaluesintersection_updatedifference_updatelistdictfromkeysextend
ValueErrorkeys
read_tablenum_rowsfieldtypeFixedSizeListType
value_type list_sizeListTypestringbinaryappendto_pandas_dtypeintwarningswarnr maxnpzeroszipto_numpyr@ r7 stackarrayobject_
add_columnr2 r) setattr _construct_mixins_from_columns)(input
include_names
exclude_names schema_onlyr3 paparquet_schemamdr! r" r# r$ meta_dict meta_yamlmeta_hdrfull_table_columnshas_serialized_columnsserialized_columnsscol
names_to_readrU pa_tablerb r< tshaperf md_namestrlenstrnamer= tabler1 header_colsattrrA r) r- s( @@@r read_table_parquetr ; s v !]]NB ec2; / 0 0 U uf%% U STT T
'
'F "
W
Wv?T?T?V?V
W
W
W
<<<<<<<<<<<< IB FF,--33D99 ,,Y77 X (I ?> >>> " 9,,!% &'? @& 0 0D"#5d#;<<
0
0+/"4((
0 &--//00I %%m444 ##M222
PPPP"4";";"="=PPPQQ I M $ $KKKK!3!9!9!;!;KKK U#### PNOOO >#;<AACCDD > >D9$$2377=== %%e]G%TT$
E 4+ 4+ LL # a- . . J[NEE
2;
'
' JJ J biikk299;; 7 7 7 } Z 7 7 9 9:::: j 8 8 : :E BCCC ('' b==G%%FF
F MRgRRR"
DDXd^DDDDDF M+g + ++ + +"
#- ";";,f,,, f = LL ! ! ! ! LL'5) * * * * (:xc-&?&?!@!@AAA4i000
9%%%M511 : :HD"4.))++C T""'ARYY[["))++...jjnnAr 344
V s88a<< (3--C| RYY[['AAA
"jjA / / (1B///CCAr{++
V `_
to be installed.
Parameters
----------
table : `~astropy.table.Table`
Data table that is to be written to file.
output : str or path-like
The filename to write the table to.
overwrite : bool, optional
Whether to overwrite any existing file without warning. Default `False`.
Notes
-----
Tables written with array columns (fixed-size or variable-length) cannot
be read with pandas.
Raises
------
ValueError
If one of the columns has a mixed-type variable-length array, or
if it is a zero-length table and any of the columns are variable-length
arrays.
r )r# r$ )serialize_context_asz.`output` should be a string or path-like, not r Nr& z$Cannot serialize mixed-type column (z) with parquet.)rf z7Cannot serialize zero-length table with object column ()rf rg r r4 r% c f i | ].\ }}| d | d /S r )encoder s r r z'write_table_parquet.. sC 15A188G,, r )rO )version g | ]}| d S )Fcopyr? )r r9 dt_news r r. z'write_table_parquet.. s) TTT#szz&uz55TTTr Fr )rd )r )3rQ r# r$ astropy.utils.data_infor rG rH rI rJ rK rM represent_mixins_as_columnsget_yaml_from_tablejoinr< rU rd rq rw r7 r_ list_from_numpy_dtyper subdtypeprodrk r2 rP itemsizestr_bytes_rp r pathexistsremoveOSErrorr rE
ParquetWriter newbyteorderrS ravelr@ baserv rc r" from_arrays write_table) r output overwriter# r$ r r r writer_version encode_tabler
meta_yaml_str type_listr) rA obj_dtyper9
arrow_typerO r1 r r metadata_encoder writerarraysvalr r s @r write_table_parquetr 3 s @ .-------<<<<<<"--- B fsBK 0 1 1 SQQQRRR
i ( ( D D < B B &
!A+/A A A ]]Q
..r{1~/BCC '"(++ " JJ ,,RW55J$
+,,,,H!)//11 4 4 c
IN9%
?? T"1%+0ABG||qBI~~ '- A AC "8SY-? @ @HH <<.1(a-.@.@H*D** + +
")^^.1(mmH*D** +&3"## 9A 9I9I O YYy?Y
;
;F w~~f > > If -4V<<== =
vv~ F F %& &, G GD #D)B __S))Fw"*$$TTTT dASTTTRX ""|$$q(( ($T*002299&+E9RRL)) CC
CC"4(//U/CC MM"((3V\\$-?-?-D(EE F F F F 8''v'>>8$$$9 % % % % % % % % % % % % % % % % % %s% BB B1E3T11T58T5c g }| D ]X\ }}t |t r#| t | =|dk r| | Y|S )a" Recursively find the names in a serialized column dictionary.
Parameters
----------
_dict : `dict`
Dictionary from astropy __serialized_columns__
Returns
-------
all_names : `list` [`str`]
All the column names mentioned in _dict and sub-dicts.
r) )rP rH r\ r^ rV rk )_dict all_namesr r s r rV rV sw I
1 a Z]] + + + +
&[[ Q r c ddl m} ddlm} | d|t
| d|t | d|t dS )z,
Register Parquet with Unified I/O.
r )registry)r" r N)
astropy.ior rQ r" register_readerr register_writerr register_identifierr ) io_registryr" s r register_parquetr sy 322222###### 52DEEE 52EFFF##Iu6FGGGGGr c dd l } ddl m} n # t $ r t d w xY wt | d rd}nd}| ||fS )Nr )r z3pyarrow is required to read and write parquet filesz6.0.0z2.4z2.0)pyarrowr ImportError Exceptionr )r r r s r rG rG s O ####### O O OMNNNO "g
w &&s
')NNFN)F)__doc__rJ rn numpyrq
astropy.utilsr astropy.utils.exceptionsr astropy.utils.miscr r __all__r r r rV r rG r( r r r s
$ $ $ $ $ $ 8 7 7 7 7 7 2 2 2 2 2 2
D OSu u u up_% _% _% _%D , H H H ' ' ' ' 'r