DUf dZddlmZddlZddlZddlZddlm Z m Z m Z ddl m Z mZmZddlmZddlZdd Z dd Zd Zefd ZGddZGddZGddZedddddddefdZdS)a Collection of classes and functions used for snipping and creation of pileups (averaging of multiple small 2D regions) The main user-facing function of this module is `pileup`, it performs pileups using snippers and other functions defined in the module. The concept is the following: - First, the provided features are annotated with the regions from a view (or simply whole chromosomes, if no view is provided). They are assigned to the region that contains it, or the one with the largest overlap. - Then the features are expanded using the `flank` argument, and aligned to the bins of the cooler - Depending on the requested operation (whether the normalization to expected is required), the appropriate snipper object is created - A snipper can `select` a particular region of a genome-wide matrix, meaning it stores its sparse representation in memory. This could be whole chromosomes or chromosome arms, for example - A snipper can `snip` a small area of a selected region, meaning it will extract and return a dense representation of this area - For each region present, it is first `select`ed, and then all features within it are `snip`ped, creating a stack: a 3D array containing all snippets for this region - For features that are not assigned to any region, an empty snippet is returned - All per-region stacks are then combined into one, which then can be averaged to create a single pileup - The order of snippets in the stack matches the order of features, this way the stack can also be used for analysis of any subsets of original features This procedure achieves a good tradeoff between speed and RAM. Extracting each individual snippet directly from disk would be extremely slow due to slow IO. Extracting the whole chromosomes into dense matrices is not an option due to huge memory requirements. As a warning, deeply sequenced data can still require a substantial amount of RAM at high resolution even as a sparse matrix, but typically it's not a problem. )partialN)is_compatible_viewframeis_cooler_balancedis_valid_expected)assign_view_automake_cooler_viewpool_decorator) LazyToeplitzbedc|}|dkr|ddg|ddg<|d|dzdz |d<tj|d|z ||zz t|d<tj|d|z ||zzd zt|d <|d|z|d<|d |z|d<nz|d krs|gd |gd <|d|dzdz |d<|d|dzdz |d<tj|d|z ||zz t|d<tj|d|z ||zzd zt|d<|d|z|d<|d|z|d<tj|d|z ||zz t|d<tj|d|z ||zzd zt|d<|d|z|d<|d|z|d<|S)aShort summary. Parameters ---------- features_df : pd.DataFrame Dataframe with feature coordinates. flank : int Flank size to add to the central bin of each feature. resolution : int Size of the bins to use. format : str "bed" or "bedpe" format: has to have 'chrom', 'start', 'end' or 'chrom1', 'start1', 'end1', 'chrom2', 'start2', 'end1' columns, repectively. Returns ------- pd.DataFrame DataFrame with features with new columns "center", "orig_start" "orig_end" or "center1", "orig_start1", "orig_end1", "center2", "orig_start2", "orig_rank_end2", depending on format. r startend orig_startorig_endrcenterlohibedpe)start1end1start2end2) orig_start1 orig_end1 orig_start2 orig_end2rrcenter1rrcenter2lo1hi1lo2hi2)copynpfloorastypeint) features_dfflank resolutionformats S/var/www/html/software/conda/lib/python3.11/site-packages/cooltools/api/snipping.pyexpand_align_featuresr/3s0""$$K 2=w>N2O \:./!,W!5 E8J!Ja O H H[*Z7 8 85J;N N &++ D H[*Z7 8 85J;N NQR R &++ D +40:= G(.; E 7   <<< =  D D D #.h"7+f:M"MQR!R I"-h"7+f:M"MQR!R I H[+j8 9 9EZ start.)indexchromrrrr) ValueError isinstancepdSeriesr2r&asarraylenshapeanyr(r) DataFrame)binsizechroms centers_bpflank_bpregion_start_bp ignore_indexr2left_bpright_bpleftrightleft_bin right_bin flank_binrrwindowss r.make_bin_aligned_windowsrKrsD w ! # #JKKK&")$$\  Z  FJ''J : !!QQQT"aaad#''( vg !!=;<<< _ $D  &Ew&&s++H((--IG#I I B Y  "Bl'''GGGG|GG'\GENIIcNNGDMIIcNNGDM Nr0c |\}}t|dkrd|vr|dj}|dj}||z t}||ks Jdt jt||d|dft j}n|dj} |dj} |dj} |d j} | | z t} | | z t}| | ks Jd||ks Jdt jt|| d|dft j}||d jfSt|d kr|\}}n|x}}d|vr|dj} |d j}| |}}n4|d j} |dj}|dj}|dj}|||}ttt||||t| |||}t j |}||d jfS)Nrrrrz,Pileup accepts only windows of the same sizer!r"r#r$_rankrrrrrr)r9valuesr(r)maxminr&fullnanlistmaprzipstack) data_select data_snipargsupport feature_grouprrsrVr!r"r#r$s1s2region1region2e1e2datas r._extract_stackrds G] 7||q m # #t$+Bt$+Bb  %%A5577aeegg%%%'U%%%GS//1qtGGEmG,333 7||q"##'- 7 # * 5 ! (RB 8 $ + 6 " ) 8 $ + 6 " ) ;w ( (D WYgw??RRQSATATUU V VE HUOOE -(/ //r0c @|drtjd|}|dd|d<t t||d<t|tt||| dd\}}tj |d}tj |}tj|}||d d d d f}|S) a Creates a stackup of snippets (a 3D array) by selecting each region present in the `features` (using the `data_select` function) and then extracting all snippets from the region (using `data_snip`). Handles on-diagonal and off-diagonal cases. Internal, so assumes correctly formatted input created by `pileup`. Parameters ---------- features : DataFrame Table of features. Requires columns ['chrom', 'start', 'end']. Or ['chrom1', 'start1', 'end1', 'chrom1', 'start2', 'end2']. start, end are bp coordinates. lo, hi are bin coordinates. data_select : callable Callable that takes a region as argument and returns the data, mask and bin offset of a support region data_snip : callable Callable that takes data, mask and a 2D bin span (lo1, hi1, lo2, hi2) and returns a snippet from the selected support region map : callable Callable that works like builtin `map`. regionzJSome features do not have view regions assigned! Some snips will be empty.rMF)sortraxisN)isnullr;warningswarnr%fillnaranger9rUrrdgroupbyr& concatenateargsort)featuresrWrXrT cumul_stack orig_rankidxs r._pileuprws":  ""&&((  X   }}H!(+22 HXc(mm,,HW!  NK ; ;   XE  2 2  K.1555Ky))I *Y  Cc111aaai(K r0c"eZdZddZdZdZdS) CoolerSnipperNrc<|t|}n7 t||dd}n"#t$r}td|d}~wwxYw|d|_||_|jj|_i|_i|_ d|_ |in||_ |j ddd|j vrN|j ddurd|_ n>|j dd us |j dd|_ n|j d|_ nd|_ ||_dS) aClass for generating snips with "observed" data from a cooler Parameters ---------- clr : cooler.Cooler Cooler object with data to use cooler_opts : dict, optional Options to pass to the clr.matrix() method, by default None Can be used to choose the cooler weight name, e.g. cooler_opts={balance='non-standard-weight'}, or use unbalanced data with cooler_opts={balance=False} view_df : pd.DataFrame, optional Genomic view to constrain the analysis, by default None and uses all chromosomes present in the cooler min_diag : int, optional This number of short-distance diagonals is ignored, by default 2 NT check_sorting raise_errors0view_df is not a valid viewframe or incompatiblenamesparsebalanceweightF)r r Exceptionr4 set_indexview_dfclrr=offsetsdiag_indicatorspad cooler_opts setdefaultclr_weight_namemin_diag)selfrrrr_es r.__init__zCoolerSnipper.__init__ s`( ?&s++GG +"&!%      F  ((00 x'  !!,!422+ ##Hd333 ( ( ( *d22'/$$ +u44#I.6'+$$'+'7 'B$$#+D   s( AAAc|jj|}|jj|}|j||j|dz |j|<|j||j|dz |j|<|jjdi|j||}|jrtj |j |j|j |_ tj |j |j|j |_ntj|j d|j t"|_ tj|j d|j t"|_|jdr|}|jY|j|\}}tj||z tj}t/| ||j|<|S)#Select a portion of the cooler for snipping based on two regions in the view In addition to returning the selected portion of the data, stores necessary information about it in the snipper object for future snipping Parameters ---------- region1 : str Name of a region from the view region2 : str Name of another region from the view. Returns ------- CSR matrix Sparse matrix of the selected portion of the data from the cooler rrrNdtype)rlocroffsetrmatrixrfetchrr&isnanbinsrN_isnan1_isnan2 zeros_liker(booltocsrrextentarangeint32r r rr_r`region1_coordsregion2_coordsrrrdiagss r.selectzCoolerSnipper.select[sR$)'2)'2 $ ? ?$(// 1 C C !  W!% ? ?$(// 1 C C !  W!444#344:: N     8  45;;NKKRDL8  45;;NKKRDLL= (..~>>EfTll L= (..~>>EfTll L  H % $\\^^F = $X__^44FBIb2gRX666E,8%,G,GD  ) r0c||\}}}}|j|} |j|} |j} || z| z || z| z } } || z| z || z| z }}| dksJ|dksJ|j\}}| | z ||z }}d}dx}x}x}}| dkr| }d}|dkr| }d}| |kr || |z z }d}||kr |||z z }d}|rbt| d}t | |}t|d}t ||}t j||ft j}ns|| | ||f d}t j||j | | ddf<t j|dd|j ||f<|j 1|j || | ||f|j k}t j||<|S)aExtract a snippet from the matrix Returns a NaN-filled array for out-of-bounds regions. Fills in NaNs based on the cooler weight, if using balanced data. Fills NaNs in all diagonals below min_diag Parameters ---------- matrix : SCR matrix Output of the .select() method region1 : str Name of a region from the view corresponding to the matrix region2 : str Name of the other regions from the view corresponding to the matrix tup : tuple (start1, end1, start2, end2) coordinates of the requested snippet in bp Returns ------- np.array Requested snippet. rFNTfloat)rr=r:rOrPr&rQrRtoarrayr(rrrr)rrr_r`tupr]rar^rboffset1offset2r=r!r"r#r$mndmdn out_of_boundspad_left pad_right pad_bottompad_topi0i1j0j1snippetDs r.snipzCoolerSnipper.snips,BB,w',w','MW,rW}.GS'MW,rW}.GSaxxxxaxxxx|1sC#IB 6:::9:zG 77J M 77tH M 77C!GnG M 77cAgI M  7S!BS!BS!BS!Bgr2h//GGSWc#g-.6688??HHG02GDLS)111, -02GAAAt|CG,, - = $$W-c#gs3w.>?$-OAGAJr0)NNr__name__ __module__ __qualname__rrrrr0r.ryrysH9!9!9!9!v111f?????r0ryc*eZdZ ddZdZdZdS) ObsExpSnipperNr balanced.avgc||_||_||_|t|}n7 t ||dd}n"#t $r}t d|d}~wwxYw t|d|||jgd}n"#t $r}t d|d}~wwxYw|d|_ |jj |_ i|_ i|_ d|_ |in||_|jd dd |jvrN|jd durd |_n>|jd d us |jd d|_n|jd |_nd |_||_dS) aClass for generating expected-normalised snips from a cooler Parameters ---------- clr : cooler.Cooler Cooler object with data to use expected : pd.DataFrame Dataframe containing expected interactions in the cooler cooler_opts : dict, optional Options to pass to the clr.matrix() method, by default None Can be used to choose the cooler weight name, e.g. cooler_opts={balance='non-standard-weight'}, or use unbalanced data with cooler_opts={balance=False} view_df : pd.DataFrame, optional Genomic view to constrain the analysis, by default None and uses all chromosomes present in the cooler min_diag : int, optional This number of short-distance diagonals is ignored, by default 2 expected_value_col : str, optional Name of the column in the expected dataframe that contains the expected interaction values, by default "balanced.avg" NTr{r~cis verify_coolerexpected_value_colsr}provided expected is not validrrrrF)rexpectedexpected_value_colr rrr4rrrr=rrrrrrr) rrrrrrrrrs r.rzObsExpSnipper.__init__s>  "4 ?&s++GG +"&!%      F  F!!+%"   AA F F F=>>A E F((00 x'  !!,!422+ ##Hd333 ( ( ( *d22'/$$ +u44#I.6'+$$'+'7 'B$$#+D   -= AAA A<< BBBcv||kstd|jj|}|jj|}|j||j|dz |j|<|j||j|dz |j|<|jjd i|j||}|jdr| }|j rtj |j |j |j|_tj |j |j |j|_ntj|j d|jt&|_tj|j d|jt&|_t)|jddg||f|jj|_|jY|j|\}}tj||z tj}t)| ||j|<|S) rz3ObsExpSnipper is implemented for cis contacts only.rrrr_r`Nrr)r4rrrrrrrrrrr&rrrNrrrr(rr rrp get_groupr _expectedrrrrrrs r.rzObsExpSnipper.select's$'!!RSS S)'2)'2 $ ? ?$(// 1 C C !  W!% ? ?$(// 1 C C !  W!444#344:: N    H % $\\^^F   8  45;;NKKRDL8  45;;NKKRDLL= (..~>>EfTll L= (..~>>EfTll L& M ! !9i"8 9 9 Y) * *4+B D    = $X__^44FBIb2gRX666E,8%,G,GD  ) r0c|\}}}}|j|} |j|} |j} || z| z || z| z } } || z| z || z| z }}| dksJ|dksJ|j\}}| | z ||z }}d}dx}x}x}}| dkr| }d}|dkr| }d}| |kr || |z z }d}||kr |||z z }d}|rat| d}t | |}t|d}t ||}t j||ft jS|| | ||f d}t j||j | | ddf<t j|dd|j ||f<|j | | ||f}|j 1|j|| | ||f|j k}t j||<||z S)aExtract an expected-normalised snippet from the matrix Returns a NaN-filled array for out-of-bounds regions. Fills in NaNs based on the cooler weight, if using balanced data. Fills NaNs in all diagonals below min_diag Parameters ---------- matrix : SCR matrix Output of the .select() method region1 : str Name of a region from the view corresponding to the matrix region2 : str Name of the other regions from the view corresponding to the matrix tup : tuple (start1, end1, start2, end2) coordinates of the requested snippet in bp Returns ------- np.array Requested snippet. rFNTr)rr=r:rOrPr&rQrRrr(rrrrr) rrr_r`rr]rar^rbrrr=r!r"r#r$rrrrrrrrrrrrrrrrs r.rzObsExpSnipper.snipas8,BB,w',w','MW,rW}.GS'MW,rW}.GSaxxxxaxxxx|1sC#IB 6:::9:zG 77J M 77tH M 77C!GnG M 77cAgI M  7S!BS!BS!BS!B7B8RV,, ,SWc#g-.6688??HHG02GDLS)111, -02GAAAt|CG,, - N3s7CG+ , = $$W-c#gs3w.>?$-OAGAJ{r0)NNrrrrr0r.rrs_ )T!T!T!T!l888tAAAAAr0rc$eZdZ ddZdZdZdS)ExpectedSnipperNrrc||_||_||_|t|}n7 t ||dd}n"#t $r}t d|d}~wwxYw t|d|||jgd}n"#t $r}t d|d}~wwxYw|d|_ |jj |_ i|_ i|_ ||_ dS) aClass for generating expected snips Parameters ---------- clr : cooler.Cooler Cooler object to which the data corresponds expected : pd.DataFrame Dataframe containing expected interactions in the cooler view_df : pd.DataFrame, optional Genomic view to constrain the analysis, by default None and uses all chromosomes present in the cooler min_diag : int, optional This number of short-distance diagonals is ignored, by default 2 expected_value_col : str, optional Name of the column in the expected dataframe that contains the expected interaction values, by default "balanced.avg" NTr{r~rrrr)rrrr rrr4rrrr=rrr)rrrrrrrrs r.rzExpectedSnipper.__init__s9(  "4 ?&s++GG +"&!%      F  F!!+%"   AA F F F=>>A E F((00 x'  !  rc||kstd|jj|}|jj|}|j||j|dz |j|<|j||j|dz |j|<t j|j||_ t j|j||_ t|j ddg||f|jj|_|jY|j|\}}t j||z t j}t| ||j|<|jS)a4Select a portion of the expected matrix for snipping based on two regions in the view In addition to returning the selected portion of the data, stores necessary information about it in the snipper object for future snipping Parameters ---------- region1 : str Name of a region from the view region2 : str Name of another region from the view. Returns ------- CSR matrix Sparse matrix of the selected portion of the data from the cooler z5ExpectedSnipper is implemented for cis contacts only.rr_r`Nr)r4rrrrrr&diffrrrr rrprrrNrrrrr)rr_r`rrrrrs r.rzExpectedSnipper.selects&'!!TUU U)'2)'2 $ ? ?$(// 1 C C !  W!% ? ?$(// 1 C C !  W88998899% M ! !9i"8 9 9 Y) * *4+B D    = $X__^44FBIb2gRX666E,8%,G,GD  )~r0c|\}}}}|j|} |j|} |j} || z| z || z| z } } || z| z || z| z }}| dksJ|dksJ| | z ||z }}| dks|dks| |jks ||jkr!t j||ftjS|| | ||f}|j1|j|| | ||f|jk}tj||<|S)a}Extract an expected snippet Returns a NaN-filled array for out-of-bounds regions. Fills NaNs in all diagonals below min_diag Parameters ---------- exp : SCR matrix Output of the .select() method region1 : str Name of a region from the view corresponding to the matrix region2 : str Name of the other regions from the view corresponding to the matrix tup : tuple (start1, end1, start2, end2) coordinates of the requested snippet in bp Returns ------- np.array Requested snippet. r) rr=rrr&rQrRrr)rexpr_r`rr]rar^rbrrr=r!r"r#r$rrrrs r.rzExpectedSnipper.snip s#,BB,w',w','MW,rW}.GS'MW,rW}.GSaxxxxaxxxxsC#IB 77cAggtvtv7B8RV,, ,c#gs3w&' = $$W-c#gs3w.>?$-OAGAJr0)Nrrrrr0r.rrsKJX:!:!:!:!x(((T'''''r0rriautorrc hd|jrd} n.hd|jrd} ntd|t|}n7 t ||dd} n"#t $r} td | d} ~ wwxYwt ||}|t|||j| }n| }| dkrW|d |jz t|d <|d |jz t|d<n|d|jz t|d<|d|jz t|d<|d|jz t|d<|d|jz t|d<|dvr: t||d} n&#t $r} td|d| d} ~ wwxYw|dkrH|dvrDt|d|jdd}n|dvrd}|gd|jd !} tt'|d"j| }|d#||d$<| dkrV|d dg|d$d%d%! t|d dg<n|ddg|d$d%d%! t|ddg<|ddg|d$d%d%! t|ddg<|t1||d&|i|'}nt3|||d&|i||(}t5||j|j| )}| dkr)t;j|t;j|d*+}|S),a Pileup features over the cooler. Parameters ---------- clr : cooler.Cooler Cooler with Hi-C data features_df : pd.DataFrame Dataframe in bed or bedpe format: has to have 'chrom', 'start', 'end' or 'chrom1', 'start1', 'end1', 'chrom2', 'start2', 'end2' columns. view_df : pd.DataFrame Dataframe with the genomic view for this operation (has to match the expected_df, if provided) expected_df : pd.DataFrame Dataframe with the expected level of interactions at different genomic separations expected_value_col : str Name of the column in expected used for normalizing. flank : int How much to flank the center of the features by, in bp min_diag: str or int All diagonals of the matrix below this value are ignored. 'auto' tries to extract the value used during the matrix balancing, if it fails defaults to 2 clr_weight_name : str Value of the column that contains the balancing weights force : bool Allows start>end in the features (not implemented) nproc : int, optional How many processes to use for calculation. Ignored if map_functor is passed. map_functor : callable, optional Map function to dispatch the matrix chunks to workers. If left unspecified, pool_decorator applies the following defaults: if nproc>1 this defaults to multiprocess.Pool; If nproc=1 this defaults the builtin map. Returns ------- np.ndarray: a stackup of all snippets corresponding to the features, with shape (n, D, D), where n is the number of snippets and (D, D) is the shape of each snippet >rr3rr >rchrom1chrom2rrrzUnknown feature_df formatNTr{r~)r-rrrrrr!rr"rr#rr$)NF)r}z#provided cooler is not balanced or z is missingrzbins/ ignore_diagsr)r3rrrrirrf region_offsetrr)rrr)rrrr)rT)rrr)axes) issubsetcolumnsr4r rrrr/r=r%r(r)rdictopenattrsgetapplyrrUrNreplacesubtractrnryrrwrrr&fmax transpose)rr*r expected_dfrr+rrnproc map_functor feature_typerrregion_offsetsregion_offsets_dictsnipperrVs r.pileupr5sp!  ))+*=>>6 A A A J J  6 4555"3'' X'"! AA  X X XOPPVW W X#;88K +  L   "&&(( 5 !,W!5 !C K KC P PK !,U!3ck!A I I# N NK  "-h"7#+"E!M!Mc!R!RK "-f"5 "C!K!KC!P!PK "-h"7#+"E!M!Mc!R!RK "-f"5 "C!K!KC!P!PK m++ "3dKKKAA   RoRRR   6o]BB #JJKK#.x#8#@#@AT#U#UK u t % XO,33A66VC[[ T4L!!  ' XO,33A66VC[[ UEN#  ' XO,33A66VC[[ UEN# "O4      "O41     K; O O OEur|E BBBCC Ls0!A55 B?BB%G88 HHH)r )rrF)__doc__ functoolsrrlnumpyr&pandasr6 lib.checksrrr lib.commonrr r lib.numutilsr r/rKrdrTrwryrrrrr0r.rs  B LKKKKKKKKK''''''<<<<F CCCCL*0*0*0Z367777tnnnnnnnnbRRRRRRRRjNNNNNNNN` %   hhhhhhr0