o NrfV@sddlmZddlZddlZddlmZddlZddlm Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZmZmZmZmZmZmZmZdd l m!Z!dd l"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0m1Z1m2Z2m3Z3m4Z4e5de6fiZ7ddZ8ddZ9e%e.ddddddZ:ddZ;ddZGd$d%d%Z Gd&d'd'e Z?dS)() annotationsN)Integral)is_datetime64_any_dtype)Rolling) normalize_arg)tokenize)BlockwiseDepDict)methods)check_axis_keyword_deprecation)Scalar_Frame_get_divisions_map_partitions_get_meta_map_partitions_maybe_from_pandasapply_and_enforce new_dd_objectpartitionwise_graph) from_pandas)_maybe_align_partitions)insert_meta_param_descriptionis_dask_collectionis_dataframe_likeis_series_like)unpack_collections)HighLevelGraph) no_default)Mapply derived_fromfuncname has_keywordCombinedOutputcCsd}|durt|tr|jd|krt||dur*t|tr*|jd|kr*t|dd|||fD}t|}t||durCt|nd|durNt|fSdfS)NzqPartition size is less than overlapping window size. Try using ``df.repartition`` to increase the partition size.rcSsg|]}|dur|qSN).0pr#r#_/var/www/html/software/conda/envs/catlas/lib/python3.10/site-packages/dask/dataframe/rolling.py 9sz#_combined_parts..) isinstancershapeNotImplementedErrorr concatr!len)Z prev_partZ current_partZ next_partbeforeaftermsgpartscombinedr#r#r&_combined_parts*s& r2c Osdd|D}|d\}}}dd|D}||i|} |dur"d}t|tjr*|}d} |jddkr=| jd|jd} |rE| rE|| 9}|durP| j|dSt|tjrX|}|r`| r`|| 9}| j|| S)NcSg|] }t|tr|qSr#r(r!r$dfr#r#r&r'Fz!overlap_chunk..rcSs"g|] }t|tr |dn|qSrr4r$argr#r#r&r'Is")r(datetime timedeltar)iloc) funcr-r.argskwargsdfsr1Zprev_part_lengthZnext_part_lengthoutZ expansionr#r#r& overlap_chunkEs(  rCT)metaenforce_metadatatransform_divisionsalign_dataframesc st|st|rt|st|dn|}|f|}ttr"tttr,tttj s8ttj rDt |j j j sCtdnttrVdkrVttrVdksZtd| dd} | dd} t|slJ| dur~tg|Ri| } ndt|} t|g|Ri| } | d | } |rt|}zt|}Wnty} zt| d | d} ~ wwd d |D}t|||| | td d|Dr| dft|tdd |Df| fi}tj| ||d}t|| Sg}g}t|||||| fdd}|D]7}t|tr"||}| || |q t!|}t"|\}}|r;| ||#|q | |q i}d}| $D]\}}t!|}t"|\}}|#||||<|rfd}qJt%|drddt&ddD}|'dt(||fdd}|rt)t*| |g|R|t+d|}n|r| n|}t)t+| |g|Ri|d|i}tj| ||d}t,|| S)a] Apply a function to each partition, sharing rows with adjacent partitions. Parameters ---------- func : function The function applied to each partition. If this function accepts the special ``partition_info`` keyword argument, it will receive information on the partition's relative location within the dataframe. df: dd.DataFrame, dd.Series args, kwargs : Positional and keyword arguments to pass to the function. Positional arguments are computed on a per-partition basis, while keyword arguments are shared across all partitions. The partition itself will be the first positional argument, with all other arguments passed *after*. Arguments can be ``Scalar``, ``Delayed``, or regular Python objects. DataFrame-like args (both dask and pandas) will be repartitioned to align (if necessary) before applying the function; see ``align_dataframes`` to control this behavior. enforce_metadata : bool, default True Whether to enforce at runtime that the structure of the DataFrame produced by ``func`` actually matches the structure of ``meta``. This will rename and reorder columns for each partition, and will raise an error if this doesn't work, but it won't raise if dtypes don't match. before : int, timedelta or string timedelta The rows to prepend to partition ``i`` from the end of partition ``i - 1``. after : int, timedelta or string timedelta The rows to append to partition ``i`` from the beginning of partition ``i + 1``. transform_divisions : bool, default True Whether to apply the function onto the divisions and apply those transformed divisions to the output. align_dataframes : bool, default True Whether to repartition DataFrame- or Series-like args (both dask and pandas) so their divisions align before applying the function. This requires all inputs to have known divisions. Single-partition inputs will be split into multiple partitions. If False, all inputs must have either the same number of partitions or a single partition. Single-partition inputs will be broadcast to every partition of multi-partition inputs. $META See Also -------- dd.DataFrame.map_overlap zMMust have a `DatetimeIndex` when using string offset for `before` and `after`rz*before and after must be positive integerstokenN parent_metazoverlap--zx. If you don't want the partitions to be aligned, and are calling `map_overlap` directly, pass `align_dataframes=False`.cSr3r#)r(r r5r#r#r&r'r7zmap_overlap..css|]}t|tVqdSr")r(r r9r#r#r& szmap_overlap..cSsg|]}|jdfqSr8)_namer9r#r#r&r's dependenciesc si}t|\}}||t|\}}||dt|}tt|||D]\}\}} } ||f} t|| | f|| <q*tj |||gd} t | |S)Nzoverlap-concat-rN) _get_previous_partitionsupdate_get_nexts_partitionsr enumeratezipZ __dask_keys__r2rfrom_collectionsr) r:dskZprevs_parts_dskprevsZnexts_parts_dsknextsname_aiprevcurrentnextkeygraph)r.r- divisionsrDr#r&_handle_frame_arguments   z+map_overlap.._handle_frame_argumentTFpartition_infocSsi|] \}}|f||dqS))numberdivisionr#)r$rZrdr#r#r& s zmap_overlap..cs|i|d|iS)Nrbr#)rbr?r@) orig_funcr#r&r>szmap_overlap..func)rOZ_func_metarO)-rrrrr(strpdZ to_timedeltar;r<rindex_meta_nonemptyZ inferred_type TypeErrorr ValueErrorpopcallablerrrrrallrtuplerrUr r r appendrrextenditemsr rSinsertrrrrCr)r>r6r-r.rDrErFrGr?r@namerJrIerAlayerr_Zargs2rOrar:Zarg2 collectionsZkwargs3simplekvrbrVZkwargs4r#)r.r-r`rDrgr& map_overlap`sB                         r~c Csi}|j}d}dt||}|r.)rMrr(rrrrtailrsr;r<rjrr`rr=rmax_tail_timedelta)r6r-rVrYrWrZr^ZdivsrZpt_zZpt_iZlbfirstjr#rr&rPbsX 1         rPcCs||j|j|kS)zReturn rows of ``next_`` whose index is before the last observation in ``current`` + ``after``. Parameters ---------- current : DataFrame next_ : DataFrame after : timedelta Returns ------- overlapped : DataFrame )rkr)r\Znext_r.r#r#r&rsrcstfdd|D}|S)a4Return the concatenated rows of each dataframe in ``prevs`` whose index is after the first observation in ``current`` - ``before``. Parameters ---------- current : DataFrame prevs : list of DataFrame objects before : timedelta Returns ------- overlapped : DataFrame cs$g|]}||jjkqSr#)rkmin)r$r[r-r\r#r&r's$z#_tail_timedelta..)r r+)rWr\r-selectedr#rr&rsrc@sHeZdZdZddddefddZddZedd Ze d d Z d d Z e e ddZe e ddZe e ddZe e ddZe e ddZe e ddZe e ddZe e d.ddZe e d.dd Ze e d!d"Ze e d#d$Ze e d%d&Ze e   '   d/d(d)Ze e d*d+ZeZd,d-ZdS)0rz%Provides rolling window calculations.NFcCsn||_||_||_||_||_||_|jjdi|t |jt r%dnd|_ |jdvr5t dtdSdS)Nfreq)rkrHrowszTUsing axis=1 in Rolling has been deprecated and will be removed in a future version.r#)objwindow min_periodscenteraxiswin_typerhrolling_rolling_kwargsr(int _win_typewarningswarn FutureWarning)selfrrrrrrr#r#r&__init__s  zRolling.__init__cCs.|j|j|j|jd}|jtur|j|d<|S)N)rrrrr)rrrrrrrr@r#r#r&rs  zRolling._rolling_kwargscCs,|jdvpt|jto|jdkp|jjdkS)zm Indicator for whether the object has a single partition (True) or multiple (False). )rHcolumnsrH)rr(rrrrrr#r#r&_has_single_partitions  zRolling._has_single_partitioncOsJt|jdi|}Wdn1swYt|||i|S)Nr#)r rgetattr)r6rolling_kwargsrwr?r@rr#r#r&pandas_rolling_methodszRolling.pandas_rolling_methodcOs|}|j|jj||g|Ri|}|jr*|jj|j||g|R||d|S|jr:|jd}|j|d}n|jdkrHt |j}d}n|jd}d}t |j|j||||g|R||d|S)N)rIrDrHrr) rrrrlrZmap_partitionsrrrrjZ Timedeltar~)r method_namer?r@rrDr-r.r#r#r& _call_methodsX      zRolling._call_methodcC |dS)Ncountrrr#r#r&r/ z Rolling.countcCr)Ncovrrr#r#r&r3rz Rolling.covcCr)Nsumrrr#r#r&r7rz Rolling.sumcCr)Nmeanrrr#r#r&r;rz Rolling.meancCr)Nmedianrrr#r#r&r?rzRolling.mediancCr)Nrrrr#r#r&rCrz Rolling.mincCr)Nrrrr#r#r&rGrz Rolling.maxrHcC|jdddS)NstdrHddofrrrr#r#r&rKz Rolling.stdcCr)NvarrHrrrr#r#r&rOrz Rolling.varcCr)Nskewrrr#r#r&rSrz Rolling.skewcCr)Nkurtrrr#r#r&rWrz Rolling.kurtcCs |d|S)Nquantiler)rrr#r#r&r[s zRolling.quantilecythonc Cs(|pi}|pd}|jd||||||dS)Nr#r)rawengine engine_kwargsr?r@r)rr>rrrr?r@r#r#r&r_s z Rolling.applycOs|jd|g|Ri|S)Naggr)rr>r?r@r#r#r& aggregateuszRolling.aggregatec CsJdd}|}|j|d<|j|d<ddddt||d DS) NcSs |\}}dddddd}||S)NrrHrrrrrrr#)itemr|r}_orderr#r#r&order|szRolling.__repr__..orderrrz Rolling [{}],css*|]\}}|dur|d|VqdS)N=r#)r$r|r}r#r#r&rLs z#Rolling.__repr__..)r^)rrrformatjoinsortedru)rrrr#r#r&__repr__{s   zRolling.__repr__)rH)FrNNN)__name__ __module__ __qualname____doc__rrrpropertyr staticmethodrrr pd_Rollingrrrrrrrrrrrrrrrrr#r#r#r&rsb   (               rcsTeZdZ     d fdd ZfddZedddd d Zfd d ZZS)RollingGroupbyNFrc s|j|_|j|_|j}|jdur7t|jtr|jg}nt|j}t|jtr-||jn| |j||}t j ||||||ddS)Nr) _groupby_kwargsZ_slice_groupby_slicerr(rilistZbyrsrtsuperr) rgroupbyrrrrrrZ sliced_plus __class__r#r&rs&        zRollingGroupby.__init__cs(t}|dddvr|d|S)Nr)rrk)rrgetrorrr#r&rs  zRollingGroupby._rolling_kwargsgroupby_kwargs groupby_slicec OsH|jdi|}|r||}|jdi|}t|||i|jddS)Nrf)levelr#)rrrZ sort_index) r6rrwrrr?r@rrr#r#r&rs  z$RollingGroupby.pandas_rolling_methodcs$tj|g|R|j|jd|S)Nr)rrrr)rrr?r@rr#r&rszRollingGroupby._call_method)NNFNr) rrrrrrrr __classcell__r#r#rr&rs !r)@ __future__rr;rnumbersrZpandasrjZpandas.api.typesrZpandas.core.windowrrZdask.array.corerZ dask.baserZdask.blockwiserZdask.dataframer Zdask.dataframe._compatr Zdask.dataframe.corer r r rrrrrZdask.dataframe.iorZdask.dataframe.multirZdask.dataframe.utilsrrrrZ dask.delayedrZdask.highlevelgraphrZ dask.typingrZ dask.utilsrrrrr typerrr!r2rCr~rRrPrrrr#r#r#r&sJ         (    ]%AL