tf2 8dZddlmZddlZddlZddlmZddlmZm Z ddl m Z ddl m Z mZmZddlZddlZddlZddlmZd d lmZd d lmZd d lmZmZmZmZm Z m!Z!ed Z"Gdde#Z$dddddddddd dddddddddd dZ%dZ& d0 d1dZ' d2dZ( d3dZ)dZ* d4d Z+ d5 d6d!Z,Gd"d#Z-Gd$d%e-Z.Gd&d'e-Z/Gd(d)e-Z0Gd*d+e-Z1Gd,d-e-Z2Gd.d/e-Z3y)7z Contact Binners ~~~~~~~~~~~~~~~ Binners are iterators that convert input data of various flavors into a properly sorted, chunked stream of binned contacts. ) annotationsN) bisect_left)Counter OrderedDict)partial)AnyCallableIterator)is_integer_dtype) get_logger) MapFunctor)GenomeSegmentationbalanced_partition check_binsget_chromsizes partitionrlencodez cooler.createc eZdZy) BadInputErrorN)__name__ __module__ __qualname___/var/www/html/software/conda/envs/higlass/lib/python3.12/site-packages/cooler/create/_ingest.pyrr%srrTFreflectchromstart)rrend12) decode_chroms is_one_based tril_action chrom_field anchor_field sided_fieldssuffixessortvalidatepos)rr-)bg2pairsc  |rtjtj|d|jdj } tjtj|d|jdj } na|dj } |dj } | rAd| jfd| jffD]"\} }t|rtd| ddz| dk| dkz}tj|r |}| |} | |} ||j}t|s g|d <g|d <|Stj|||dz}tj|||d z}|r |d z}|d z}| r|j|jfD]}t|rtd |dk|dkz}tj|r=||}td j|jjd|j j | }|j j | }||kD||kDz}tj|r=||}tdj|jjd|1| | kD| | k(||kDzz}tj|r |dk(r| || |c| |<| |<||||c||<||<|D]\}|j"|||d zf|j"|||dzf c|j"|||dzf<|j"|||d zf<^n|dk(r+|}| |} | |} ||}||}||j}nQ|dk(r=||}tdj|jjdt%d|d|j&}|j(}||j*}|j,}g}g}t/| || |D]\} }!}"}#|| }$|| d z}%|j1|$tj2||$|%|| |!zdzd z ||"}$||"d z}%|j1|$tj2||$|%||"|#zdzd z ||d <||d <n|| ||zz|d <|| ||zz|d <| r|j5d d g}|S)Nchrom1T)orderedchrom2`z` column is non-integer. z6If string, use `decode_chroms=True` to convert to enumrbin1_idbin2_idz!Found a non-integer anchor columnzFound an anchor position with negative value. Make sure your coordinates are 1-based or use the --zero-based option when loading. {} sepz8Found an anchor position exceeding chromosome length: {}rdropraisezFound lower triangle pairs: {}Unknown tril_action value: ''rightside)nparraypd Categoricalcontigscodesvaluesdtyper ranycopylenformatheadto_csv chromsizesloc ValueErrorchrom_binoffsetbinsize chrom_abspos start_absposzipappend searchsorted sort_values)&chunkgsr$r%r&r'r(r)r*r+r, chrom1_ids chrom2_idscoldtto_dropmaskanchor1anchor2is_negerr chromsizes1 chromsizes2 is_excessis_trilfieldrSrTrUrVbin1_idsbin2_idscid1pos1cid2pos2lohis& r_sanitize_recordsrtCsXX NN5?BJJ E K K XX NN5?BJJ E K K 8_++ 8_++ %z'7'788ZEUEU:VW R'+'C5 9:RS A~*q.1G vvgx% % d   " u:ii hhu\HQK789Ghhu\HQK789G1 1 =='--0 IB#B'#$GHH IA+'A+. 66&>-C%%+VCHHJ,=,=$,=,G%H  mm**:6 mm**:6 {*w/DE 66)  "CKRRHHJ%%$%/  * : %'G*; <  66'?i'w'w';G$j&96=W5EwwGW2 ''"2)E  '58A;+>">? '58A;+>">? '58A;+>">? '58A;+>">?&x'- '- !$-!$-d ((*'Gn#5<< ))d)3 !#? }A!NOO((OjjG  &)*gz7&S  "D$d &B *B OO// B'd);d)B  !&B *B OO// B'd);d)B   &$i#i*:6G9KKi*:6G9KKi !!9i"89 Lrc | t|}ni}|j}|jdi|t |}t |||d<ttfi|S#t$rtd|ddwxYw)a Builds a funtion to sanitize and assign bin IDs to a data frame of paired genomic positions based on a provided genomic bin segmentation. Parameters ---------- bins : DataFrame Bin table to compare records against. schema : str, optional Use pre-defined parameters for a particular format. Any options can be overriden via kwargs. If not provided, values for all the options below must be given. decode_chroms : bool Convert string chromosome names to integer IDs based on the order given in the bin table. Set to False if the chromosomes are already given as an enumeration, starting at 0. Records with either chrom ID < 0 are dropped. is_one_based : bool Whether the input anchor coordinates are one-based, rather than zero-based. They will be converted to zero-based. tril_action : 'reflect', 'drop', 'raise' or None How to handle lower triangle ("tril") records. If set to 'reflect', tril records will be flipped or "reflected" to their mirror image: "sided" column pairs will have their values swapped. If set to 'drop', tril records will be discarded. This is useful if your input data is symmetric, i.e. contains mirror duplicates of every record. If set to 'raise', an exception will be raised if any tril record is encountered. chrom_field : str Base name of the two chromosome/scaffold/contig columns. anchor_field : str Base name of the positional anchor columns. sided_fields : sequence of str Base names of column pairs to swap values between when mirror-reflecting records. suffixes : pair of str Suffixes used to identify pairs of sided columns. e.g.: ('1', '2'), ('_x', '_y'), etc. sort : bool Whether to sort the output dataframe by bin_id and bin2_id. validate : bool Whether to do type- and bounds-checking on the anchor position columns. Raises BadInputError. Returns ------- callable : Function of one argument that takes a raw dataframe and returns a sanitized dataframe with bin IDs assigned. NzUnknown schema: 'r>r\r) SANITIZE_PRESETSKeyErrorrRrKupdaterrrrt)binsschemakwargsoptionsrPs rsanitize_recordsr}sv F&v.GllnGGNNV%J&z48GDM $ 0 00 F0:; E Fs AA6c T|r||xxdzcc<||xxdzcc<|||||kD} tj| r|dk(r|j| |f|j| |f c|j| |f<|j| |f<|D]\} |j| | |dzf|j| | |dzf c|j| | |dzf<|j| | |dzf<^n+|dk(r|| }n|dk(r tdt d|d|r|j ||gS|S) Nr7rrr;r<"Found bin1_id greater than bin2_idr=r>)rBrJrQrrRrZ) r[r\r%r& bin1_field bin2_fieldr)r*r+rjrks r_sanitize_pixelsr-s jQ jQ  #eJ&77 66'?i' IIgz12IIgz12IIgz12IIgz12 *E  '58A;+>">? '58A;+>">? '58A;+>">? '58A;+>">?&wh'#$HII #? }A!NOO:>5  j*5 6IEIrc Vt|}t|||d<ttfi|S)ai Builds a function to sanitize an already-binned genomic data with genomic bin assignments. Parameters ---------- bins : DataFrame Bin table to compare pixel records against. is_one_based : bool, optional Whether the input bin IDs are one-based, rather than zero-based. They will be converted to zero-based. tril_action : 'reflect', 'drop', 'raise' or None How to handle lower triangle ("tril") pixels. If set to 'reflect' [default], tril pixels will be flipped or "reflected" to their mirror image: "sided" column pairs will have their values swapped. If set to 'drop', tril pixels will be discarded. This is useful if your input data is duplexed, i.e. contains mirror duplicates of every record. If set to 'raise', an exception will be raised if any tril record is encountered. bin1_field : str Name of the column representing ith (row) axis of the matrix. Default is 'bin1_id'. bin2_field : str Name of the column representing jth (col) axis of the matrix. Default is 'bin2_id'. sided_fields : sequence of str Base names of column pairs to swap values between when mirror-reflecting pixels. suffixes : pair of str Suffixes used to identify pairs of sided columns. e.g.: ('1', '2'), ('_x', '_y'), etc. sort : bool Whether to sort the output dataframe by bin_id and bin2_id. Returns ------- callable : Function of one argument that takes a raw dataframe and returns a sanitized dataframe. r\)rrrr)ryr{rPs rsanitize_pixelsr\s0^ %J%j$7F4L # .v ..rct|rb|ddk|ddkz}tj|r td|d|k\|d|k\z}tj|r td|r+|d|dkD}tj|r tdt|tj st j |}|r`|j ddg} | jr=|| } tdj| jjd |r|jddg}|S) Nr5rr6zFound bin ID < 0zaFound a bin ID that exceeds the declared number of bins. Check whether your bin table is correct.rzFound duplicate pixels: {}r8r9) rBrJr isinstancerD DataFrame duplicatedrMrNrOrZ) r[n_bins boundscheck triucheckdupcheck ensure_sortedrerirjis_duprfs r_validate_pixelsrs: "Q&5+;a+?@ 66&> 23 39%/E)4D4NO 66) ;   "U9%55 66'? DE E eR\\ * U#!!9i"89 ::<-C-44SXXZ5F5F45F5PQ !!9i"89 Lrc,tt|||||S)Nrrrrr)rrrs rvalidate_pixelsrs# #  rcJii|rdvr dd<dd<fd}|S)a= Generates a function that aggregates bin-assigned records by pixel. Parameters ---------- sort : bool, optional Sort group keys. Get better performance by turning this off. Note that this does not influence the order of observations within each group. count : bool, optional Output the number of records per pixel. Default is True. agg : dict, optional Dict of column names -> functions or names. rename : dict, optional Dict to rename columns after aggregating. Returns ------- Function that takes a dataframe of records with bin IDs assigned, groups them by pixel, counts them, and optionally aggregates other value columns. Notes ----- The GroupBy 'count' method ignores NaNs within groups, as opposed to 'size'. countsizer5c|jddgjjjS)Nr5r6)r+columns)groupby aggregaterename reset_index)r[aggrr+s r_aggregate_recordsz-aggregate_records.._aggregate_recordss9 MM9i0tM < Ys^ VFV # []  rr)r+rrrrs` `` raggregate_recordsrsF@ { ~ #I#y  rc eZdZdZddZddZy) ContactBinnerz3 Base class for iterable contact binners. c^|jj}|jdd|S)N_map)__dict__rKpop)selfds r __getstate__zContactBinner.__getstate__s' MM    fdrct)a Iterator over chunks of binned contacts (i.e., nonzero pixels) Chunks are expected to have the following format: * dict of 1D arrays * keys `bin1_id`, `bin2_id`, `count` * arrays lexically sorted by `bin_id` then `bin2_id` )NotImplementedErrorrs r__iter__zContactBinner.__iter__s "!rN)returndictrzIterator[dict[str, np.ndarray]])rrr__doc__rrrrrrrs  "rrcPeZdZdZ d dZd dZd dZd dZd dZddZ y)HDF5AggregatorzE Aggregate contacts from a hiclib-style HDF5 contacts file. c $||_|jdd|_|jdd|_|jdd|_|jdd|_t |||_||_|j|_ y) NC1chrms1P1cuts1C2chrms2P2cuts2) h5rrrrrrr\ chunksize _index_chromsr)rh5pairsrPryrr{s r__init__zHDF5Aggregator.__init__sx**T8,**T7+**T8,**T7+$Z6"++-rc t|j|j|j\}}}t t |t |k7r t dtt|t|||zS)Nz6Read pair coordinates are not sorted on the first axis) rrrrrLsetrRrrW)rstartslengthsrHs rrzHDF5Aggregator._index_chroms)se"*477477+;T^^"L s6{ s6{ *UV VCFFW,< =>??rc *td|j|j||fd|j|j||fd|j|j||fd|j|j ||fg}t j|S)N chrom_id1cut1 chrom_id2cut2)rrrrrrrDr)rrrrsdatas r _load_chunkzHDF5Aggregator._load_chunk0sdggdgg.r"56)"R01dggdgg.r"56)"R01   ||D!!rc#K|j}|j|j|j|jf\}}}}|j }|j j}|j j} |j j} |j j} |j j} |j j|} |jj| d\}}|}|}||kr|t||z|}}| | |||dz z}t!t#j$| |ddz }|d|}t'|||||}||k(r|}t(j+|d||j-||}| ||||||||z}| ||||||||z}t#j.||kDr t1d| =t#j$| |ddz |d<t#j$| |ddz |d <nt#j2|d | z j5t"j6}t#j2|d | z j5t"j6}| |d j8|z|d<| |d j8|z|d <|j;dd g}|d j=j?jAd di}|||kr yyw)N)rr7r?r@r  aFound a read pair that maps to the lower triangle of the contact map (side1 > side2). Check that the provided chromosome ordering and read pair file are consistent such that all pairs map to the upper triangle with respect to the given chromosome ordering.r5r6rrrrrr)!rrrrrrr\ryrTrSrUrVidmaprgetminintrBrYrloggerinforrJrRfloorastypeint64rHrrrr)rrrrrrrrryrTrSrUrVcidchrom_lochrom_hirrrsabsposbin_idbin_endtableabspos1abspos2rel_bin1rel_bin2gbyrs rrzHDF5Aggregator.aggregate;s''$''477DGG;BBNN ww||''//''11ww++ ww++ ggmmE"!^^//X>(  8mR)^X6B!#&R!V)< I QEFD P Q.%K! .$%II! . D DsME&F(E96E4E9 FE14E99FFFFF"c n|\}}}ddl}|j}|jj}|jj}|jj } |jj } |jj} t|j} |j} |j}tjd|d|d|d|jj|||f}g}|j|dd5}|j!}t#}|j%D]\}}|j||j&|j(| D]m}|| }t||| z } ||}|5| |}| |d z}|t-j.| ||| ||zd zd z }n | |||zz}||xxd z cc<o|s|j1t3j4|t7|j9t7|j;d gd j=d|j? dddtjd|d|d|dtA|rt3jB|dSdS#t*$rYGwxYw#1swY\xYw)NrBinning :-|*rrr)parserr7r?r@r5r6rrr6 Finished axis)"rrr\rTrrSrUrVrr%rrrrfetchrasTupleriterrowsrr rwrBrYrXrDrlistkeysrHrZclearrLconcat)rgranger1rr rrrTrrSrUrVdecrrr these_binsrowsrr accumulatorr5bin1liner3rqrprrrsr6s rrzTabixAggregator.aggregates$s==''// ''11ww++ ww++ 4$$% WW WW hvhawauB78WW]]FE3#78  __XsW_ =. $]]_F!)K!+!4!4!6* $ GGFDJJGP.D!"XFtBx=4/D!$V} ,T2,TAX6 oo ,R 3 ,T 2T 9%,    #2$"747?"K(A-(3.6# LL'.'+K,<,<,>'?%)+*<*<*>%? !@ "k), !!#U* $ . $`  ixqqR89*-d)ryyA&==O$! !. $. $s2A4J+:J?CJ+ J( $J+'J( (J++J4c#Kt|j|j|j}|j |j |D]5}||j Dcic]\}}||jc}}7ycc}}wwrrr\rrrrrrHrgrangesrrrs rrzTabixAggregator.__iter__k$TWWdmmT=N=NO))DNNG4 ;B~/1xxz:tq!q!(({:: ;:A B B #B:B N) rrrPrryrr rrrr%rr!ztuple[str, int, int]rzpd.DataFrame | Nonerrrrrr rrrrrrrrsp"4 4 4  4  4  4 4 lG>*G> G>R;rrcTeZdZdZedddf d dZ d dZd dZy) PairixAggregatorzm Aggregate contacts from a sorted, BGZIP-compressed and pairix-indexed tab-delimited text file. r7F|c  ddl} ddl} ddl} | j| j d<||_||_t||_ ||_ | j|d} | j|_ | j|_| j!|_| j%|_| j)} || dvrt+d|d| ddt-t.j0j3| Dcgc]}|j5|c}|_t9|j6s t;dt/j<|j6d D]G\}}| j?||s| j?||s,t;d d |||d |||d ztA| d r| jC}|dkr d}||z|z}tEd}|d z|z}|j}tG|j||_|j|kDr&tHjKd|d|jdtM|||_'||_(d|_)|jNjTD]-}||j6vstWjXdd|dz/y#t$r tddwxYwcc}w)Nrz1pypairix is required to read pairix-indexed filesrrz The contig separator character `z+` does not appear in the first block name `z8`. Please specify the correct character as `block_char`.rr z'Pairs are not triangular: found blocks r>z'' and ' get_linecountigחAzPairs file has z lines. Increasing max-split to .rrr)-pypairixrrrrrrrrr% block_charopen get_chr1_colr get_chr2_colrget_startpos1_colrget_startpos2_colrget_blocknamesrRr itertoolschain from_iterablesplitrrLr  combinationsexists2hasattrr4rmaxrrrr\rrrFr r )rrrPryr rr%r7r{r6rrr blocknames blocknamec1c2n_linesMAXINT32 max_chunkold_nrs rrzPairixAggregator.__init__s   $*$;$; j!    .$ MM(C (..".."%%'%%'%%' Z] *2:,?33=a=/BAA   OO ) )=G+09  ++   4$$%>? ?,,T->->B FByyR QYYr2%6"="j\"XbT*bTKL  1o &oo'G{%"X-7E I!|y0HMME x8DM}}u$ %gY///3}}oQ@ %Z6! WW__ ED--- *r%8O-PP I C  :+sJ2K 2Kc |\}}}ddl}|j}|jj}|jj}|jj } |jj } |jj} t|j} |j} |j}tjd|d|d|d|j|d}|jj|||f}|jj |d}t#}g}|j%D]v\}}|j'D]\}}||}||k7r?|j)||r-|j+|d|||j,|j.}| }n,|j+||j,|j.|d|}|}|D]b}t||| z }|5| |}| |dz}|t1j2| ||| ||zd zdz }n | |||zz}||xxdz cc<d|s|j5t7j8|t;|j=t;|j?d gd  jAd |jCytjd |d|d|dtE|rt7jF|dSdS)Nrrrrrrr7r?r@rrr6rr)$r6rr\rTrPrSrUrVrr%rrrrr8rrrrrrCquery2Drr rBrYrXrDrrrrHrZrrLr ) rr!r1rr r6rrTrPrSrUrVr"rrrr#remaining_chromsr%r$r5r&r3rp chrom2_sizeiteratorpos2_colr'rqrrrsr6s rrzPairixAggregator.aggregatems$s==''//WW'' ''11ww++ ww++ 4$$%WW WW hvhawauB78 MM(C (WW]]FE3#78 77==1i '0021 MGT 0 6 6 8 . (0 V# &&(A yy; DHH H "H yy DHHfa H "H$.DtH~.5D,T2,TAX6 oo ,R 3 ,T 2T 9%,    #2$"747?"K(A-(%. .D KK #*#' (8(8(:#;!%k&8&8&:!; < +i(     c1 f  ixqqR89*-d)ryyA&==rc#Kt|j|j|j}|j |j |D]5}||j Dcic]\}}||jc}}7ycc}}wwrr)r*s rrzPairixAggregator.__iter__r,r-N)rrrPrryrr rrrr%rr7rr.rr/rrrr1r1s"UUU U  U  UUUnP>*P> P>d;rr1c$eZdZdZdZdZddZy)SparseBlockLoaderrct||}||_t|j|_||_t |}t |}|djj}tj|dzt|_ d}tt|D]\} } } | |j|| dz| dz}||j|d||_yNrr7)rIrrryrindex chromosomesrrLcatrGrBzerosroffsetsrWrmapping rrPryr_rn_chromsr chrom_idscurr_valr_lengthvalues rrzSparseBlockLoader.__init__$ +  0 01"z?TM%%++ xx1 C8 %((9*=%> ! !E7E16DLLEAI .qyH !#) XY rc  |j||f}|S#t$rJ |j||fj}Y|S#t$rtjd|d|dwxYwwxYwNz Block for {z, z } not foundr_rwTr r rr1r3blocks r select_blockzSparseBlockLoader.select_block LL0E    VV^466   VHBvhlKL   A(='A$$A(c #Kddl}|j}t|D]\}}|j|}|j|dz|z }t d||j }|D]\}} g} ||dD]=} |j || } | j| j|| ddf?|jj| jj} | j| j| j}}}||z||zk}||||||}}}||z||z|dy#t$rYwxYww)Nrr7r) scipy.sparser[ enumerater^rrrmrwrXtocsrsparsehstacktocoorowr_r)rscipyr[rnr1offset chrom1_nbinsspansrrrschunksr3rlXijrrbtriu_itriu_jtriu_vs rrzSparseBlockLoader.__iter__sg && %k2 LD&\\$'F<<q1F:Lat~~>E B)$%0;F! $ 1 1&& AMM%++-2q"9: ; LL''/557==?%%a1  3)*4!D'1T7 &%#   $! !s+A0E3D8B3E8 E EE ENrrrrrrrmrrrrrVrVs $ rrVc,eZdZdZ ddZddZy) ArrayLoaderz Load a dense genome-wide numpy array contact matrix. Works with array-likes such as h5py.Dataset and memmapped arrays See Also -------- numpy.save, numpy.load (mmap_mode) clt||jdk7r td||_||_y)Nrz4Number of bins must equal the dimenion of the matrix)rLshaperRrCr)rryrCrs rrzArrayLoader.__init__s2 t9 A &ST T "rc#K|jjd}td||j}|D]T\}}|j||ddf}t j |\}}||z|k}||||} } || z| || | fdVyw)Nrr)rCrrrrBnonzero) rrr{rrrsr}r~rrbrrs rrzArrayLoader.__iter__s!!!$!VT^^4 FB 2b5!8$A::a=DAqFq=DtWagFF;!66>*  sB B N)ryrrCznp.ndarray | h5py.Datasetrrr)rrrrrrrrrrrs- # #) # #rrc"eZdZdZdZdZdZy)ArrayBlockLoaderrct||}||_t|j|_||_t |}t |}|djj}tj|dzt|_ d}tt|D]\} } } | |j|| dz| dz}||j|d||_yrXrYr`s rrzArrayBlockLoader.__init__3rfrc  |j||f}|S#t$rJ |j||fj}Y|S#t$rtjd|d|dwxYwwxYwrhrirks rrmzArrayBlockLoader.select_blockErnroc #K|j}t|D]\}}|j|}|j|dz|z }td||j}|D]\}}g} ||dD]/} |j || } | j| ||ddf1tj| d} tj| \} }|| z||zk}| |||}}||z||z| ||fdy#t $rYwxYww)Nr7rrr) r[rrr^rrrmrwrXrB concatenater)rr[rnr1ryrzr{rrrsr|r3rlr}r~rrbrrs rrzArrayBlockLoader.__iter__Ps>&& %k2 LD&\\$'F<<q1F:Lat~~>E B)$%03F! $ 1 1&& AMM%2q/2 3 NN62zz!}1  3!"4!D' &%vv~.   $! !s+A+D.C8A8D8 D DD DNrrrrrr0s $ rrr)ryrrzz str | Noner&Callable[[pd.DataFrame], pd.DataFrame])Frr5r6rr!T)ryrrr) rrrrrrrrrrrr)TTNN) r+rrrrzdict[str, Any] | Nonerzdict[str, str] | Nonerr)4r __future__rr>r bisectr collectionsrr functoolsrtypingrr r h5pynumpyrBpandasrDpandas.api.typesr _loggingr _typingrutilrrrrrrrrRrrvrtr}rrrrrrrrr1rVrrrrrrs#,** -!  O $ J  1  ( 4^FF1 F1 F1, F1X  ,J^1/ 1/,1/hD     , $!%$( 4 4 4 4 " 4 , 4n""0m;]m;`J;mJ;Zt;}t;n> >B&-&R;};r