d-WddlmZddlZddlZddlZddlmZmZmZm Z ddl m Z m Z m Z ddlZddlmZddlmZddlmZddlmZe rdd lmZd Zd1dZd2dZd3dZd4dZdZd5dZ d6d"Z!d7d(Z" d8d9d0Z#dS):) annotationsN)HashableIterableMappingSequence) TYPE_CHECKINGAnyCallable)align) DataArray)Datasetis_dask_collection)T_Xarrayct|SN)zip)iterables 4lib/python3.11/site-packages/xarray/core/parallel.pyunziprs >ar bc,|}|}t|jt|jD]1}|j||j|krt d|d2dS)NzChunk sizes along dimension z are not equal.) unify_chunkssetchunks intersection ValueError)rrdims rassert_chunks_compatibler!s A A18}}))#ah--88TT 8C=AHSM ) )RCRRRSS S *TTrresultDataArray | DatasetexpectedMapping[str, Any]kindstrc |dkrd}n|dkrd}||tt||z }|rtd|d|dtt||||z }|rtd|d|ddS) Ncoords coordinate data_varsdataz4Result from applying user function does not contain z variables .z2Result from applying user function has unexpected )rgetattrr)r"r$r&nice_strmissingextras rcheck_result_variablesr2!s x   tns764#8#8999G  / / /$+ / / /    %% & &$ 7E   - - -$) - - -     robjreturnr ct|tstdt|t |jdkrtdt t|jS)NzExpected Dataset, got zGTrying to convert Dataset with more than one data variable to DataArray) isinstancer TypeErrortypelenr+nextitervalues)r3s rdataset_to_dataarrayr>8s c7 # #><c<<=== 3=A U    S]))++,, - --rcf|j|}n|}|Sr)name_to_temp_dataset to_dataset)r3datasets rdataarray_to_datasetrDDs3 x&&((.."" Nrct|tr|}t|}nt|trd}n|Sddlm}t}|jD].\}}||j|j }|j ||j f||</|j |_ | |j }|t|S|S)zIf obj is a DataArray or Dataset, return a new object of the same type and with the same variables and dtypes, but where all variables have size 0 and numpy backend. If obj is neither a DataArray nor Dataset, return it unaltered. Nr)meta_from_array)ndim)r7r rDr dask.array.utilsrF variablesitemsr,rGdimsattrs set_coordsr)r>)r3 obj_arrayrFmetar@variablemeta_objs r make_metarROs #y!! "3'' C ! !  000000 99D---//??h"?8=x}EEEmXx~>T DJ ??3: & &D#D))) KrfuncCallable[..., T_Xarray]rcd|f|zD} ||i|}n"#t$r}td|d}~wwxYwt|ttfst dt ||S)zz"infer_template..os999C3999rzsCannot infer object returned from running user provided function. Please supply the 'template' kwarg to map_blocks.NzIFunction must return an xarray DataArray or Dataset. Instead it returned ) Exceptionr7r r r8r9)rSr3argskwargs meta_argstemplatees rinfer_templaterbks:9C6D=999I4-f--  @    h) 4 5 5  H~~     Os :5:xdict[Hashable, Any]ct|tr|}d|jDS)zWMap variable name to numpy(-like) data (Dataset.to_dict() is too complicated). c$i|] \}}||jSrW)r,)rYkvs r zmake_dict..s 6 6 6$!QAqv 6 6 6r)r7r rArIrJrcs r make_dictrksI!Y!    6 6!+"3"3"5"5 6 6 66rr r chunk_indexr chunk_boundsc||vr3||}t||||||dzStdS)Nr6)slice)r rlrm which_chunks r_get_chunk_slicerrqsL k!#& \#&{3\#5F{UV5WXXX ;;rrWr] Sequence[Any]r^Mapping[str, Any] | Noner`DataArray | Dataset | Nonec )*+,-./01234dAd }Atttfs%td t jd t|t std|i}n$t|tstd|D] }t|rtd!t|s ||g|Ri|S dd l *dd l *ddl m }n#t$rYnwxYw|gt|z}d|D1d|D} t!1fdt#|D\} } 1fdt#|D} t%| ddi} t'd| D} t!t)tt+| | | zd\}}t-|dj/t-|dj0| dd D]L}t3|d|/|j0|jMt7|| dg|Ri|t9j}|t90z}|t90z }0fd|D--fd|D/fdjD3n,t-j-j33st?d 3D]q}|/vrktA/|tA3|kr?t?d!|d"tA/|d#tA3|d$rttrd%}j!}"n7ttrd&}ntd't i,tGj$t,}d(%*j&'|*j()|d||+d)/*D}d*/*D.d+3*D2dB*fd1 4tWj,|D]Z}t-t+|-|))+,.4fd2t+1|D}i}3fd3)*D|d4<t9j.-|d5<t9j/-|d6<)-2fd7-D|d8<+f|z}||||| |f,|<i}j0*D]Q\}}|-vr |d9+} | ||<| f}!|jD]}|)vr |!)|fz }!|!d:z }!tbj2||f|| |!<R\|3+,*fd;|D<}"|i|"j4|i|"j5+fd=|-D<}"td>-*Dj6?}#|#jD]2}$|$j6|#|$_6|$j7|#|$_73|*D]\}} |j}%g}&|%D]w}|3vr|&83|"||#jvr"|&8|#j9|fM|jvr!|&8j9|fx*j:;|"| |&|j<@}'|%|'|j6f|#|<|j7|#|_7|#=j>}#|rt|#}(||(_!|(S|#S)CaApply a function to each block of a DataArray or Dataset. .. warning:: This function is experimental and its signature may change. Parameters ---------- func : callable User-provided function that accepts a DataArray or Dataset as its first parameter ``obj``. The function will receive a subset or 'block' of ``obj`` (see below), corresponding to one chunk along each chunked dimension. ``func`` will be executed as ``func(subset_obj, *subset_args, **kwargs)``. This function must return either a single DataArray or a single Dataset. This function cannot add a new chunked dimension. obj : DataArray, Dataset Passed to the function as its first argument, one block at a time. args : sequence Passed to func after unpacking and subsetting any xarray objects by blocks. xarray objects in args must be aligned with obj, otherwise an error is raised. kwargs : mapping Passed verbatim to func after unpacking. xarray objects, if any, will not be subset to blocks. Passing dask collections in kwargs is not allowed. template : DataArray or Dataset, optional xarray object representing the final result after compute is called. If not provided, the function will be first run on mocked-up data, that looks like ``obj`` but has sizes 0, to determine properties of the returned object such as dtype, variable names, attributes, new dimensions and new indexes (if any). ``template`` must be provided if the function changes the size of existing dimensions. When provided, ``attrs`` on variables in `template` are copied over to the result. Any ``attrs`` set by ``func`` will be ignored. Returns ------- A single DataArray or Dataset with dask backend, reassembled from the outputs of the function. Notes ----- This function is designed for when ``func`` needs to manipulate a whole xarray object subset to each block. Each block is loaded into memory. In the more common case where ``func`` can work on numpy arrays, it is recommended to use ``apply_ufunc``. If none of the variables in ``obj`` is backed by dask arrays, calling this function is equivalent to calling ``func(obj, *args, **kwargs)``. See Also -------- dask.array.map_blocks, xarray.apply_ufunc, xarray.Dataset.map_blocks xarray.DataArray.map_blocks Examples -------- Calculate an anomaly from climatology using ``.groupby()``. Using ``xr.map_blocks()`` allows for parallel operations with knowledge of ``xarray``, its indices, and its methods like ``.groupby()``. >>> def calculate_anomaly(da, groupby_type="time.month"): ... gb = da.groupby(groupby_type) ... clim = gb.mean(dim="time") ... return gb - clim ... >>> time = xr.cftime_range("1990-01", "1992-01", freq="M") >>> month = xr.DataArray(time.month, coords={"time": time}, dims=["time"]) >>> np.random.seed(123) >>> array = xr.DataArray( ... np.random.rand(len(time)), ... dims=["time"], ... coords={"time": time, "month": month}, ... ).chunk() >>> array.map_blocks(calculate_anomaly, template=array).compute() array([ 0.12894847, 0.11323072, -0.0855964 , -0.09334032, 0.26848862, 0.12382735, 0.22460641, 0.07650108, -0.07673453, -0.22865714, -0.19063865, 0.0590131 , -0.12894847, -0.11323072, 0.0855964 , 0.09334032, -0.26848862, -0.12382735, -0.22460641, -0.07650108, 0.07673453, 0.22865714, 0.19063865, -0.0590131 ]) Coordinates: * time (time) object 1990-01-31 00:00:00 ... 1991-12-31 00:00:00 month (time) int64 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 Note that one must explicitly use ``args=[]`` and ``kwargs={}`` to pass arguments to the function being applied in ``xr.map_blocks()``: >>> array.map_blocks( ... calculate_anomaly, ... kwargs={"groupby_type": "time.year"}, ... template=array, ... ) # doctest: +ELLIPSIS dask.array<-calculate_anomaly, shape=(24,), dtype=float64, chunksize=(24,), chunktype=numpy.ndarray> Coordinates: * time (time) object 1990-01-31 00:00:00 ... 1991-12-31 00:00:00 month (time) int64 dask.array rSr r]listr^dict arg_is_arrayIterable[bool]r$c dt||D}||i|}t|dt|jz }|rtd|d|jD]\}} ||dvrM|j||d|kr0td|d|j|d|d|d||d vr<|d |} | | std |d | d | d t||dt|trt||dt|S)z Wrapper function that receives datasets in args; converts to dataarrays when necessary; passes these to the user function `func` and checks returned objects for expected shapes/sizes/etc. c:g|]\}}|rt|n|SrW)r>)rYis_arrayrZs rr[z0map_blocks.._wrapper..s?   #*2 :  % % %s   rshapesz Dimensions z missing on returned object.zReceived dimension z of length z. Expected length r-indexeszExpected index  to be . Received instead.r)r+) rrsizesr_indexesrJequalsr2r7r rk) rSr]r^rxr$converted_argsr"missing_dimensionsr@indexexpected_indexs r_wrapperzmap_blocks.._wrappers  !$\4!8!8    ~000!(!344s6<7H7HH  N0NNN  "?0022  KD%x)))<%();D)AAA$GdGGdASGG+3H+=d+CGGGx ***!))!4T!:||N33$h$hhhhV[hhh vx::: fg & & B "68[ A A A   rNz2template must be a DataArray or Dataset. Received rz7args must be a sequence (for example, a list or tuple).z.kwargs must be a mapping (for example, a dict)zgCannot pass dask collections in kwargs yet. Please compute or load values before passing to map_blocks.r)HighLevelGraphcFg|]}t|ttfSrW)r7r r rXs rr[zmap_blocks..Is'KKK3C'9!566KKKrc8g|]}t|tSrW)r7r rXs rr[zmap_blocks..Js"???s 3 **???rc36K|]\}}| ||fVdSrrWrYrrZ is_xarrays r zmap_blocks..MsI((#i>N(  ((((((rc.g|]\}}| ||fSrWrWrs rr[zmap_blocks..Ps;#)EBR  rjoinexactc3bK|]*}t|trt|n|V+dSr)r7r rDrXs rrzmap_blocks..VsS &0Y%?%?HS!!!Src|dS)NrrWrjs rzmap_blocks..\s aPQdr)keyr6c"i|] }|| SrWrW)rYr input_indexess rrizmap_blocks..ms HHHs3 c*HHHrc,i|]}|j|SrW)r)rYrgr`s rrizmap_blocks..ns#EEEA8,Q/EEErc*i|]}|v||SrWrW)rYr input_chunkss rrizmap_blocks..os1= = = '*sl?R?RCc"?R?R?RrzjProvided template has no dask arrays. Please construct a template with appropriately chunked dask arrays.z~map_blocks requires that one block of the input maps to one block of output. Expected number of output chunks along dimension rrzP instead. Please provide template if not provided, or fix the provided template.TFz.func output must be DataArray or Dataset; got z{}-{}cNi|]"\}}|tt|#SrW)ranger:rYr chunks_vs rrizmap_blocks..s, R R RMCc5X'' R R RrcDi|]\}}|tjd|zSrnpcumsumrs rrizmap_blocks..s9,9CRYth ' 'rcDi|]\}}|tjd|zSrrrs rrizmap_blocks..s9,9CRYth ' 'rgraphgnamer'rCr cg}g}t}|jD]\}} | jrZ| } | jD]} | | } |d|d| df|z} t| j| | jgf|| <nhfd| jD} | | }|d|dj |f|z} t|j||jgf|| <||j vr| || g| || gtt|ft|f|jfS)z Creates a task that subsets an xarray dataset to a block determined by chunk_index. Block extents are determined by input_chunk_bounds. Also subtasks that subset the constituent variables of a dataset. -rc4i|]}|t|SrWrq)rYr rlinput_chunk_boundss rriz?map_blocks..subset_dataset_to_block..s9*3 =OPPr)tupler=rIrJrr, __dask_keys__rKrLiselbasetokenize _coord_namesappendr rw)rrrCrrlr+r) chunk_tupler@rPchunkr chunk_variable_task subsettersubsetdasks `` rsubset_dataset_to_blockz+map_blocks..subset_dataset_to_blocks K..0011 %/5577! >! >ND(&&x}55  ..00#=44C!+c"23EE*.'C'C'C'Cq'C'C&E &S#]E8>:.)**'} "y11BBeBBdi&8&8&@&@BB'' #[&&,7.)* w+++ t%89::::  $(;!<====$ *T6NGMJJrc <g|]\}}|r|n|SrWrW)rYisxrrZrlrrrrs rr[zmap_blocks..sQ   c  # #E5#7I; W W W   rc<i|]\}}|v |||SrWrW)rYrgrh output_chunkss rrizmap_blocks..s:   '+q!a=FXFXA}Q"FXFXFXrr}r+r)c Li|] }||t|!SrWr)rYr rlr~output_chunk_boundss rrizmap_blocks..sB    /[BUVVW   rr~rrc>g|]}||SrWr)rYrZrs rr[zmap_blocks..s,LLLct/F/Fs/K/KLcLLLr) dependenciesci|]}|hSrWrW)rYr@rs rrizmap_blocks..s;;;teW;;;rc>i|]\}}||SrW)to_pandas_index)rYrgidxs rrizmap_blocks..&s*GGGVQ3&&((GGGr)r)rL)r@rdtype) rSr r]rvr^rwrxryr$rw)rrwrr'rCr )@r7r r r8r9__name__rrr=rr dask.arraydask.highlevelgraphr ImportErrorrvr enumerater rsortedrrwrrr!updaterbrrK chunksizesrr:r@rA collections defaultdictformatutilsfuncnamerrrJ itertoolsproductkeysr+r)rIoperatorgetitemfrom_collectionslayersrrLencodingrrarrayArrayrrMrr>)5rSr3r]r^r`rvaluerall_argsr|xarray_indices xarray_objsothersaligned_npargsrZtemplate_indexespreserved_indexes new_indexesr result_is_array template_name new_layersichunkr blocked_argsr$ from_wrapper var_key_mapr@rPgname_lrhlgr"rrK var_chunksr,darlrrrr~rrrrrrrs5 ` @@@@@@@@@@@@r map_blocksrs P.!.!.!.!`Jx)W9M$N$N chAX c c c    dH % %SQRRR ~  ( (JHIII e $ $ <   c " "*tC)$)))&)))  6666666      utDzz!HKK(KKKI??h???H#((((('0':':(((##NK'0':':F [/w//GK tC 4455>NNSSSIAv q ())L+,,M122++ C000CJ'''S\****!$ DTDDDVDDx011,s=/A/AA&]););; HHHH6GHHHEEEEEEEFFF= = = = .6m= = = x()) +  W   ,  3|C'8#9#9SsAS=T=T#T#T-DG--RUVbcfVgRhRh-- c 233--- (I&&   ,,.. Hg & &  MT(^^ M M   E?J?V @@J NN D!!49#5#5fQiv#N#N  E S R\=O=O=Q=Q R R RF=I=O=O=Q=Q=J=P=P=R=R4K4K4K4K4K4Kn!(&--//:4N4N 3v{{}}k::;;         !F33        /:/@/@/B/B   !$H$6$;$;$=$= > > !5!5!7!788          x+- '|VXxXl,. &06688 N NND(w''''G 'K $+:C}  +%%K,..CC4KCC)1(8,'MJw  $ $+ N.  ) )  LLLLVLLL *  C .$3:$$  ;;;;):):;;;    CGGw}}GGGnF ::&uo3u !)%!9u $**,,88 g~"  : :Cm##!!-"45555''!!6<#4"67777 %%!!8>##6"8999z gj8L   dHTN$89t ( 7t    x4 5 5F !& ) ) Ms3D DD)rr rr )r"r#r$r%r&r')r3r r4r )r3r r4r )rSrTr3r#r4r)rcr#r4rd)r rrlrrmr)rWNN) rSrTr3r#r]rrr^rsr`rtr4r)$ __future__rrrrcollections.abcrrrrtypingrr r numpyrxarray.core.alignmentr xarray.core.dataarrayr xarray.core.datasetr xarray.core.pycompatrxarray.core.typesrrr!r2r>rDrRrbrkrqrrWrrrs""""""AAAAAAAAAAAA//////////''''''++++++''''''333333+******TTTT    . . . . .8.7777'++/ sssssssr