tfs@ddlmZddlZddlZddlmZddlmZmZddl m Z m Z m Z ddl Z ddlZddlZddlZddlmZddlmZdd lmZdd lmZdd lmZmZdd lmZdd l m!Z!m"Z"gdZ#ee$Z%dZ&gdZ' d dZ(GddeZ) d! d"dZ*d#dZ+ d$dZ,d%dZ-d&dZ. d' d(dZ/ d) d*dZ0GddeZ1 d+ d,dZ2 d+ d-dZ3 d) d.dZ4y)/) annotationsN) bisect_right) OrderedDict defaultdict)AnyIteratorLiteral) get_logger) MapFunctor)__format_version_mcool__)Cooler) ContactBinnercreate)lock)GenomeSegmentationparse_cooler_uri) merge_coolerscoarsen_coolerzoomify_cooler) iiii'iaiPiii i@Bi%&i@KLicttj|dj}tt |D] }|||z } d}|d}dg}dg}d} t |t ||z||dz } | |k(r| dz } |j| |j|| || |k(rn| }|| }^tj|}tj|}tj|} | |kDj} | dkDr-tj| d|d| jdd||fS) a Given ``k`` bin1_offset indexes, determine how to partition the data from ``k`` corresponding pixel tables for a k-way merge. The paritition is a subsequence of bin1 IDs, defining the bounds of chunks of data that will be loaded into memory from each table in a single "epoch" of merging data. The bounds are calculated such that no single epoch will load more than ``bufsize`` records into memory. However, the ``bufsize`` condition is not guaranteed and a warning will be raised if it cannot be satisfied for one or more epochs (see Notes). Parameters ---------- indexes : sequence of 1D array-like of equal length Offset arrays that map bin1 IDs to their offset locations in a corresponding pixel table. bufsize : int Maximum number of pixel records loaded into memory in a single merge epoch. Returns ------- bin1_partition : 1D array Bin1 IDs defining where to partition all the tables for merging. cum_nrecords : 1D array Cumulative number of records (from all pixel tables combined) that will be processed at each epoch. Notes ----- The one exception to the post-condition is when a single bin1 increment in a table contains more than ``bufsize`` records. r)lor z/ merge epochs will require buffering more than z pixel records, with as many as g.)npzerosshaperangelenrminappendarraydiffsumwarningswarnmax) indexesbufsizecombined_indexicombined_start combined_nnzbin1_partition cum_nrecordsrhinrecords_per_epochn_overs W/var/www/html/software/conda/envs/higlass/lib/python3.12/site-packages/cooler/reduce.pymerge_breakpointsr6.suVXXgaj../N 3w< %'!*$%N!"%LSN3L B    (, 7    8 !GBb!N2./ "  -  '+) ,XXn-N88L)L. 7* / / 1F z hEgYO..@.D.D.Fq-I L < ''c4eZdZdZ d ddZddZy) CoolerMergerz, Implementation of cooler merging. Nc$t||_||_|dgn||_|jDcic]}|dc}|_||jj ||dj }|t|Dchc]}|j c}dkDr td|dj}tdt|D]2} tj|| j|k(r)tdy|djgddd} tdt|D]W} || jgddd} t| t| k7stj| | k(rNtdycc}wcc}w) Ncountr&rr z%Coolers must have the same resolutionz&Coolers must have the same chromosomeschromstartendz$Coolers must have same bin structure)listcoolersmergebufcolumnsaggupdatebinsizer! ValueError chromsizesr rallbins) selfrArBrCrDcolrFcrHr-rJbins2s r5__init__zCoolerMerger.__init__snG}   $+Oy *.,,73CJ7 ? HHOOC !*$$  w/!AII/014 !HII ..J1c'l+ Ovvgaj33zAB$%MNN O1:??$%>?BD1c'l+ M )*CDQGJ#d)+BFF5D=4I$%KLL M80s F;F c #VK|jDcgc]}|jdd}}t||j\}}t j |}t jdt|t jd|t jd||jDcgc]}t|j}}t jd|dgt|jz}|ddD]}|D cgc]} | | } } tjt|j|| D cgc]!\}} } | | z dkDr|j| | #c} } }dd } | jd d gd j|j j#}|j%Dcic]\}}||j&c}}t jd| | }ycc}wcc}wcc} wcc} } }wcc}}ww)Nrindexes/bin1_offsetzn_merge_epochs: zbin1_partition: znrecords_per_merge_epoch: znnzs: rr T)axis ignore_indexbin1_idbin2_idsortzrecords consumed: )rAopenr6rBrr%loggerinfor!debugpixelspdconcatzipgroupby aggregaterD reset_indexitemsvalues)rKrMr*r0r1r3nnzsstartsrUindexstopsr>stopcombineddfkvs r5__iter__zCoolerMerger.__iter__s?C||L!166#;45LL(9$--'P$ WW\2 &s+='>&?@A ''789 12D1EFG)-6AAHHJ66 fTFO$s4<<((%ab) G189U7^9E9yy+.dllFE*J&5$u )HHJuT* !H  )Y!7d C488$  ,.88:641a1ahh;6 6 KK,UG4 5F1 M7 : 7sAH)H BH); H?H) H&,H)&H8AH)H#*?H))NN)rAz list[Cooler]rBintrClist[str] | NonerDdict[str, Any] | NonereturnzIterator[dict[str, np.ndarray]])__name__ __module__ __qualname____doc__rOror7r5r9r9sD%)%) MMM" M # M<(r7r9c "tjdjdj||Dcgc] }t |}}|D cgc]} | j dk(} } t | rd} nt| sd} n td|dg}tt} |D]Z} | jj} |D]9}|| vrtd |d | jd | |j| |;\|i}|D]}||vstj | |||<!|d j#gd dd}|d jj%dd}t'||||}t)|||f|||| d|ycc}wcc} w)a Merge multiple coolers with identical axes. The merged cooler is stored at ``output_uri``. .. versionadded:: 0.8.0 Parameters ---------- output_uri : str Output cooler file path or URI. input_uris : list of str List of input file path or URIs of coolers to combine. mergebuf : int Maximum number of pixels processed at a time. columns : list of str, optional Specify which pixel value columns to include in the aggregation. Default is to use all available value columns. dtypes : dict, optional Specific dtypes to use for value columns. Default is to propagate the current dtypes of the value columns. agg : dict, optional Functions to use for aggregating each value column. Pass the same kind of dict accepted by ``pandas.DataFrame.groupby.agg``. Default is to apply 'sum' to every value column. kwargs Passed to ``cooler.create``. Notes ----- The default output file mode is 'w'. If appending output to an existing file, pass `mode='a'`. See also -------- cooler.coarsen_cooler cooler.zoomify_cooler z Merging: {} symmetric-upperTFz1Cannot merge symmetric and non-symmetric coolers.Nr;Pixel value column '' not found in input ''.rr<zgenome-assembly)rBrCrD)rCdtypesassemblysymmetric_upper)rZr[formatjoinr storage_moderIanyrGrr@r]rfilenamer#r result_typerJgetr9r) output_uri input_urisrBrCrrDkwargspathclrsclris_symmr dtype_map pixel_dtypesrLrJriterators r5rrsb KK%%dii &;<=%/ 0TF4L 0D 0@DEs#44EGE 7| \LMM)D!I 9zz|**  9C,& *3%0!ll^2/ #%%l3&78 9 9~: f ..)C.9F3K: 7<<>3 4Q 7DAw|| 148HD8W#NH   '   I 1Es FF c tj|}t|dk\r|ddk(sJtjdtjtj |f}t dttj|d|z Dcgc]}||z }}|j|dtjtj||}||Scc}w)zGiven an integer interval partition ``edges`` from 0..nnz, prune the edges to make the new subintervals roughly ``maxlen`` in length. rr) rasarrayr!r_cumsumr%r rpceilr#unique searchsorted)edgesmaxlencumlenr-cutsidxs r5_greedy_prune_partitionr8s JJu E u:?uQx1}, , UU1bii// 0F %aRWWVBZ&5H-I)J K L1FQJ LD LKKr  ))BOOFD1 2C : Ms C)c||z}t|}tj||z }ttjt j |}|S)aH Number of zoom levels for a quad-tree tiling of a genomic heatmap. At the base resolution, we need N tiles, where N is the smallest power of 2 such that the tiles fully cover the 1D data extent. From that starting point, determine the number of zoom levels required to "coarsen" the map up to 1 tile. )r&mathrrprlog2)rH base_binsize bins_per_tiletile_length_bptotal_bpn_tiles n_zoom_levelss r5get_quadtree_depthrEsP#\1N:Hii>12G "'''"234M r7c#TKt|t|}}| ||z}| w)z Generate a geometric progression of integers. Beginning with integer ``start``, yield an unbounded geometric progression with integer ratio ``mul``. rpr>muls r5geomprogr`s4USX3E K     s&(c#RKt|}| dD] }||z |dz}w)z Generate a nice progression of integers. Beginning with integer ``start``, yield a sequence of "nicely" spaced integers: an unbounded geometric progression with ratio 10, interspersed with steps of ratios 2 and 5. )r rrrs r5niceprogros@ JE K  C#+     s%'c||kDrgS|dk(r t|d}n |dk(r t|}ntd|dtg} t|}||kDr |S|j |%)a Return a sequence of integers with a "preferred" stepping pattern. Parameters ---------- start : int Starting value in the progression. stop : int Upper bound of progression, inclusive. Values will not exceed this. style : {'nice', 'binary'} Style of progression. 'nice' gives geometric steps of 10 with 2 and 5 in between. 'binary' gives geometric steps of 2. Returns ------ list of int Examples -------- For certain values of `start` (n * 10^i), nice stepping produces familiar "preferred" sequences [1]_: Note denominations in Dollars (1-2-5) >>> preferred_sequence(1, 100, 'nice') [1, 2, 5, 10, 20, 50, 100] Coin denominations in Cents >>> preferred_sequence(5, 100, 'nice') [5, 10, 25, 50, 100] .. [1] https://en.wikipedia.org/wiki/Preferred_number#1-2-5_series binaryrnicez1Expected style value of 'binary' or 'nice'; got 'r)rrrGnextr#)r>rjstylegenseqns r5preferred_sequencersR t|  ua  &uoFugRPQ 9+C  I t8  J 1 r7c@| t|h}n t|}tjt |j |}tj t|t }tj t|t }tt|dddD]7\}}|dz }|dk\s|||zdk(r|||<|||z||<-|dz}|dk\r'9t|D]'\}}|dk(s |||vstd||d|d|||fS) a" From a set of target resolutions and one or more base resolutions deduce the most efficient sequence of integer multiple aggregations to satisfy all targets starting from the base resolution(s). Parameters ---------- resolutions: sequence of int The target resolutions bases: sequence of int, optional The base resolutions for which data already exists. If not provided, the smallest resolution is assumed to be the base. Returns ------- resn: 1D array Resolutions, sorted in ascending order. pred: 1D array Index of the predecessor resolution in `resn`. A value of -1 implies that the resolution is a base resolution. mult: 1D array Multiplier to go from predecessor to target resolution. N)dtyperr rz Resolution z. cannot be derived from the base resolutions: r) r"setrr$sorteduniononesr!rpr@ enumeraterG) resolutionsbasesresnpredmultr-targetps r5get_multiplier_sequencersB8 }[!"E  88F5;;{34 5D GGCIS ) )D GGCIS ) )D)D/*4R40 6 E1fQ1$Q DG+QQ 1f$1 7tAwe+d1gY')).q2  t r7cpeZdZdZef ddZe d dZd dZd dZ d dZ y) CoolerCoarsenerz/ Implementation of cooler coarsening. c\||_||_||_t|tr|dkDsJ||_t ||_ddg|_t||_ |j|jz|_ |jDcic]}|dc}|_ ||jj|t|} | j} | j|_| j#d|_| j#d|_|j d|_n|j |z|_| j+gddd} |j-| | ||_t1| |j.|_g} |j2j4j7D]G\} }|j$|}|j$|dz}| j9|j&|||I| j;|j&dt=| |j |_ycc}w) Nr rUrVr&zindexes/chrom_offsetrRr<r) _map source_uri batchsize isinstancerpfactor chunksize index_columnsr@ value_columnsrCrDrErrHrF old_binsize _load_dsetold_chrom_offsetold_bin1_offset new_binsizerJ coarsen_binsnew_binsrgsidmaprdextendr#rr)rKrrrrCrDrmaprLrrHold_binsr_chromr-c0c1s r5rOzCoolerCoarsener.__init__s $"&#&6A:55 Y'3!']))D,>,>> *.*<*<=3CJ= ? HHOOC Z ^^ ;; #/E F"~~.CD    ##D #//&8D 88:78;))(JG $Z? ,,. =IFA&&q)B&&q1u-B LL--bFl; <  =  T))"-.,UDNNC A>s H)cfd}|jddgdj|jdS)Nc|ddgjjdd}|djdz dj}t|t|kr"tj ||j f}||d<|S)Nr=r>r?r )copyilocrer!rrname)groupoutr?rHrs r5_eachz+CoolerCoarsener.coarsen_bins.._each9s'*+00277&AC,##FQJ$8&$89@@C3x#c("eeCEJJ!778CJJr7r=Tobservedr<)drop)raapplyrc)rrHrrs `` r5rzCoolerCoarsener.coarsen_bins5s>   WWtW ,-F H U5\ [d[ #  r7c|\}}t|j}|jdd|j}|||}tj |d||j j}|j j}|j j} |j j} |dj} |dj} |dj} |dj}|M| | | z}| | |z}tj| |d d z |d <tj| |d d z |d <nltj| |z jt }tj||z jt }|| |z|d <|| |z|d <|j#|j$dj'|j(j+S)NTF)r convert_enum chrom1chrom2start1start2right)sider rUrVrW)rrr]rCrZr[rrFchrom_binoffset chrom_abspos start_absposrerrfloorastyperprarrbrDrc)rKspanrr2rtablechunkrFrrr chrom_id1 chrom_id2rr abs_start1 abs_start2rel_bin1rel_bin2s r5 _aggregatezCoolerCoarsener._aggregateHsBT__% 5 9$,,Gb  rd!B4L!''//''11ww++ ww++ (O** (O** x''x'' ?%i069J%i069J jwG!K )  jwG!K ) xx 0188=Hxx 0188=H.y9HDE) .y9HDE)  MM$,,4M 8 Ytxx [] r7cv |j|}|S#t$r}tt||d}~wwxYwN)r  MemoryError RuntimeErrorstr)rKrres r5rbzCoolerCoarsener.aggregateps? .OOD)E  .s1v&A - .s 838c#K|j}tt|jdd|jdd}t dt ||D]} |dkDrt j|j|j||||z}|dkDrt j |D]2}|jDcic]\}}||jc}}4y#|dkDrt jwwxYwcc}}ww)Nrr r) rr@r`rr r!racquirerrbreleaserdre)rKrspansr-resultsrlrmrns r5rozCoolerCoarsener.__iter__wsNN SCR$**QR.9:q#e*i0 ;A #q=LLN))DNNE!a)m4LMq=LLN ;/1xxz:tq!q!(({:: ; ; q=LLN!;s*AD ;C'2D D  D 'D D N)rrrrprrprC list[str]rDrrrrprr )r pd.DataFramerH pd.Seriesrrprtr)rztuple[int, int]rtrrs) rurvrwrxrrO staticmethodrr rbroryr7r5rrs6D6D6D 6D  6D # 6D6D6Dp  ,5 ?B   $& P ;r7rc xt|} t|}|dg}|i}| jj} |D]7} | | vrt d| d| j d|j | | | 9 |dkDr+tj|} |j dtt|||||||dkDr  jnt} | j}|j d d t||| f|| jd k(d ||dkDr jyy#|dkDr jwwxYw) aP Coarsen a cooler to a lower resolution by an integer factor *k*. This is done by pooling *k*-by-*k* neighborhoods of pixels and aggregating. Each chromosomal block is coarsened individually. Result is a coarsened cooler stored at ``output_uri``. .. versionadded:: 0.8.0 Parameters ---------- base_uri : str Input cooler file path or URI. output_uri : str Input cooler file path or URI. factor : int Coarsening factor. chunksize : int Number of pixels processed at a time per worker. nproc : int, optional Number of workers for batch processing of pixels. Default is 1, i.e. no process pool. columns : list of str, optional Specify which pixel value columns to include in the aggregation. Default is to use all available value columns. dtypes : dict, optional Specific dtypes to use for value columns. Default is to propagate the current dtypes of the value columns. agg : dict, optional Functions to use for aggregating each value column. Pass the same kind of dict accepted by ``pandas.DataFrame.groupby.agg``. Default is to apply 'sum' to every value column. kwargs Passed to ``cooler.create``. See also -------- cooler.zoomify_cooler cooler.merge_coolers Nr;r}r~rr r)rCrDrrr#Tr|)rr)rrpr]rrGr setdefaultmpPoolrrrrrrclose)base_urirrrnprocrCrrDrr input_dtypesrLpoolrrs r5rrsdl  C [F) ~::<&&L6 l "&se+Ecll^SUV    c<#4 5 6 19775>D   fd +"   !AI3 $$(D)    ,,0AA     19 JJL 519 JJL s 7BD!!D9c t|tr|g}i} i} t} |D]} t| \} }t | |}|j dn |j }| |f| |<|j ddjddjj| |<| j|t|| \}}}t|}tjd|d|dg}| D]}tjd t|z| |\} }tj | d 5}tj |d 5}d |}|j#|d z||d z|j#|dz||dzddgt%|D]!}|j#|d|z||d|z#|j#|dz||dz||j&j)||j&ddddddt+|D][}||dk(r |||}|||z}tjd|d|dt-|d|z|d|z|||f||||dd|]tj |d5}|j&j)dt.ddddy#1swYxYw#1swYxYw#1swYyxYw)a Generate multiple cooler resolutions by recursive coarsening. Result is a "zoomified" or "multires" cool file stored at ``outfile`` using the MCOOL v2 layout, where coolers are stored under a hierarchy of the form ``resolutions/`` for each resolution ``r``. .. versionadded:: 0.8.0 Parameters ---------- base_uris : str or sequence of str One or more cooler URIs to use as "base resolutions" for aggregation. outfile : str Output multires cooler (mcool) file path. resolutions : list of int A list of target resolutions to generate. chunksize : int Number of pixels processed at a time per worker. nproc : int, optional Number of workers for batch processing of pixels. Default is 1, i.e. no process pool. columns : list of str, optional Specify which pixel value columns to include in the aggregation. Default is to use only the column named 'count' if it exists. dtypes : dict, optional Specific dtypes to use for value columns. Default is to propagate the current dtypes of the value columns. agg : dict, optional Functions to use for aggregating each value column. Pass the same kind of dict accepted by ``pandas.DataFrame.groupby.agg``. Default is to apply 'sum' to every value column. kwargs Passed to ``cooler.create``. See also -------- cooler.coarsen_cooler cooler.merge_coolers Nr r=Trz$Copying base matrices and producing z new zoom levels.r;z Bin size: rQwz /resolutions/z/chromsz/binsrUrVz/pixels/z/indexesrzAggregating from z to rz::resolutions/r+)r rCrrDmodez HDF5::MCOOL)rzformat-version)rrrrrrFrJrasizer)addrr!rZr[h5pyFilerr@attrsrEr rr ) base_urisoutfilerrr rCrrDr parsed_urisn_bins_longest_chrombase_resolutions input_uriinfileingrouprrrrrn_zoomssrcdestprefixrLr- prev_binsizerFfws r5rrs8j)S!K Ku + *95VW%KK/qS[[ %+W$5 L! HHJqM WWtW , TV SU \* \* +/{ ?!9=tG}= ..xu--  HHWz)4*1D E L   % %c'l&8&8 9 : : ::(7^  7b= DG} a( ' ~T'!DE |n5 5 wi0 0 G     & 7D ! R $8P Q   I : : : :H  s16K B/J:<K#K:K ?KK Kc t|\}}t||}t|j|jt }d} t jdtj|jt jd|jt jd|t jdd|zt jd|d|dzd zt} t|} |j} t jd t| zd zt| ztj|d 5} tj|d 5}| j||t| | | | <ddddddt|dz ddD]}| | z} t|dz}t|} t jdt| zd zt| zt!|dzt|z|dzt| z| |||| | | <tj|d5}|j"j%d|i||j"d<|j"j%| ddd|| fS#1swYxYw#1swYxYw#1swY|| fSxYw)zL Quad-tree tiling using legacy MCOOL layout (::0, ::1, ::2, etc.). rztotal_length (bp): z binsize: z n_zooms: zquad tile cover: zCopying base matrix to level z and producing z new zoom levels zcounting down to 0...z Zoom level: z bin size: rQr$Nr rzAggregating at zoom level: z::)rr rr%zmax-zoomz max-zooms)rrrrHrFHIGLASS_TILE_DIMrZr[rr&rrr)r*rr rr+rE)r1r-r rrr2r3rr4r zoom_levels zoomLevelrFr5r6r- prevLevelr9s r5legacy_zoomifyr?rs'y1OFG  !C >NOG F KK%bffS^^&<%=>? KK)CKK=)* KK)G9%& KK#AL>23 KK' I_WI-> ? @ ! " -KG IkkG KKY/-?#g,NO 63 )3 7C )$( $I/!( I))7Q;B ')6AJ F  ))n  'l    dNS^ + dNS^ +   ") I)), 7D !%R W-. '  $ % K G)))):% K s1J++"J J+ AJ8J( #J++J58K)r*zlist[np.ndarray | h5py.Dataset]r+rprtztuple[np.ndarray, np.ndarray])NNN)rrrrrBrprCrqrrrrDrrrtNone)r np.ndarrayrrprtrA)rHrrrprrprtrp)r>rprrprt Iterator[int])r>rprtrB)r)r>rprjrprzLiteral['binary', 'nice']rt list[int]r )rrCrzlist[int] | Nonertz)tuple[np.ndarray, np.ndarray, np.ndarray])r NNN)rrrrrrprrpr rprCrqrrrrDrrrtr@)r,zstr | list[str]r-rrrCrrpr rprCrqrrrrDrrrtr@) r1rr-rr rprrprtztuple[int, dict[str, int]])5 __future__rrr'bisectr collectionsrrtypingrrr r) multiprocessrnumpyrpandasr^_loggingr _typingr _versionr apirrrparallelrutilrr__all__rurZr; ZOOMS_4DNr6r9rrrrrrrrrrr?ryr7r5rSs" 0))  .)6 ? H   "U( ,U( U(#U(pL=Lf!%$(!% ```` ` " `  ` `F  6 ((.: : : %: :~#77 7/7tM;mM;j $$(!%hhh h h  h  h "h h h` $$(!%} } } }  }  }  }  "}  }  } BDH@ @  @ ),@ 9<@ @ r7