DUfydZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z mZmZddlmZmZddlmZmZddlmZmZddlmZmZdd lmZmZdd lm Z m!Z!dd l"m#Z#e$e dZ%e$e d Z&e$e d Z'e$e dZ(dZ)ej*ej+ d"dZ,dZ-dZ.idd de/fdZ0 d"dZ1 d#dZ2 d$d!Z3dS)%a This module enables construction of observed over expected pixels tables and storing them inside a cooler. It includes 3 main functions. expected_full - is a convenience function that calculates cis and trans-expected and "stitches" them togeter. Such a stitched expected that "covers" entire Hi-C heatmap can be easily merged with the pixel table. expected_full_fast - generated the same output as `expected_full` but ~2x faster. Efficiency is achieved through calculating cis and trans expected in one pass of the pixel table. Post-processing is not fully implemented yet. obs_over_exp - is a function that merges pre-calculated full expected with the pixel table in pd.DataFrame or dask.DataFrame formats. obs_over_exp_generator - is a function/generator(lazy iterator) that wraps `obs_over_exp` function and yields chunks of observed/expected pixel table. Such a "stream" can be used in cooler.create as a "pixels" argument to write obs/exp cooler-file. It also includes 3 helper functions (used in `expected_full_fast`): make_pairwise_expected_table - a function that creates an empty table for the full expected with all the right sizes and number of valid pixels pre-filled. Combines functionality of `make_diag_tables` and `make_block_tables` from the API. sum_pairwise - a function that calculates the full pixel summary for all pairwise combinations of the regions in the view, and for each genomic separation for cis-combinations of regions. In a nutshell - it calls `make_pairwise_expected_table` to generate empty table for expected, and it gets filled by applying `_sum_` to the pixels table. _sum_ - a function that does the actual summing of pixel values grouped by regions and genomic separations - can work on a chunk of pixel table. N) partition)teechaincombinations_with_replacement)reducepartial) expected_cisexpected_trans)assign_supportsmake_cooler_view)is_compatible_viewframeis_cooler_balanced)make_block_tablemake_diag_tables)diag_expected_dtypesblock_expected_dtypes)expected_smoothing)levelF皙?expectedweight逖c Jtj} t||d|||||| |  } tj| z } tjd| ddtj} t |||| | }t |d<tj| z } tjd| dd|d d i  d }|rd nd}|r |d}|r|d}| | | |<|rd nd}|r|d}ddg}|r| d| |dj |d|dj |dg|d}|d|dz |d<|r|d|dz |d<|| ||<t!j| |gd}|d j |d|d<|d j |d|d<tjd|S)Z Generate a DataFrame with expected for *all* 2D regions tiling entire heatmap in clr. Such 2D regions are defined as all pairwise combinations of the regions in view_df. Average distance decay is calculated for every cis-region (e.g. inter- and intra-arms), and a "simple" average over each block is caculated for trans- regions. When sub-chromosomal view is provided, trans averages can be aggregated back to the level of full chromosomes. Parameters ---------- clr : cooler.Cooler Cooler object view_df : viewframe expected is calculated for all pairwise combinations of regions in view_df. Distance dependent expected is calculated for cis regions, and block-level average is calculated for trans regions. smooth_cis: bool Apply smoothing to cis-expected. Will be stored in an additional column aggregate_smoothed: bool When smoothing cis expected, average over all regions, ignored without smoothing. smooth_sigma: float Control smoothing with the standard deviation of the smoothing Gaussian kernel. Ignored without smoothing. aggregate_trans : bool Aggregate trans-expected at the inter-chromosomal level. expected_column_name : str Name of the column where to store combined expected ignore_diags : int, optional Number of intial diagonals to exclude for calculation of distance dependent expected. clr_weight_name : str or None Name of balancing weight column from the cooler to use. Use raw unbalanced data, when None. chunksize : int, optional Size of pixel table chunks to process nproc : int, optional How many processes to use for calculation Returns ------- expected_df: pd.DataFrame cis and trans expected combined together F) view_df intra_onlysmooth smooth_sigmaaggregate_smoothedclr_weight_name ignore_diags chunksizenprocz!Done calculating cis expected in .3f sec ...)r r%r'r(distz#Done calculating trans expected in indexrcolumnsname balanced.avg count.avgz .smoothed.aggn_valid count.sum balanced.sumchromregion1region2sumz count.avg.aggzbalanced.avg.aggT) ignore_indexr1r2z&Returning combined expected DataFrame.)time perf_counterr logginginfor TRANS_DIST_VALUE reset_indexrename set_indexcopyappendgroupbylocto_numpy transformpdconcat)clrr smooth_cisr$r#aggregate_transexpected_column_namer&r%r'r( time_startcvd time_elapsedcpb view_labelcis_expected_nametrans_expected_name additive_cols_cpb_agg expected_dfs b/var/www/html/software/conda/lib/python3.11/site-packages/cooltools/sandbox/obs_over_exp_cooler.py expected_fullr]Ssz"$$J  !-'!   C$&&3L LO\OOOOPPP"$$J  '    C#CK$&&3L LQ|QQQQRRR ..6"" +:J{;0;;;  ;#4 : : :  #$5 6 ; ; = =C-<L..S!4:::";/  1   0 0 0;;7#'I7@@BB7#'I7@@BB     #5))  ( 4Xi5HHO  S&.~&>x ?R&RC" # #$7 8 = = ? ?C)S#JT:::K#3+K ,BCLLNNK"3+K ,BCLLNNK L:;;; c ptt|d\}}d|D}d|D}t|\}}t j|dg}t j|dg}t||||}t|\}}t j|dg}t j|dg}t||||}g} |ddgD]W\} } } |ddgD]2\} }}| | kr$| |krz|| |f }| d d | | d d | | | d d tg| |krt j|| |fd g }| d tt| d d | | d d | | | d d tg4Yt j| S) z create a DataFrame for accumulating expected summaries (blocks and diagonal ones) it also contains "n_valid" column for dividing summaries by. rc3DK|]\}}|j|jk||fVdSNr7.0r<r=s r\ z/make_pairwise_expected_table..s8LLfb"bh"(6J6J"b6J6J6J6JLLr^c3DK|]\}}|j|jk||fVdSrarbrcs r\rez/make_pairwise_expected_table..s8PPB"(bh:N:NB8:N:N:N:NPPr^Indexr.r%r7r0rr=r<)r,)rr itertuplesziprL DataFramedroprrrCinsertrGrE _DIST_NAMErBrM)rNr r% cis_combs trans_combsregions1regions2dtablesbtables_tables_r1chrom1name1_r2chrom2name2dfs r\make_pairwise_expected_tabler}s!%g&8&8&:&:A>>I{ML LLLIPP+PPPKiHh|H%%**G9*==H|H%%**G9*==HsHhXXXGk*Hh|H%%**G9*==H|H%%**G9*==HsHhXXXGG%wv&67BBDDKKVU")76*:";"F"F"H"H K K Cs FNN %0<<>>BIIas+++IIas+++NN2<<tZ0H#I#IJJJf$$guen&=aSIIIBIIa-=>>>IIas+++IIas+++NN2<<tZ0H#I#IJJJ K 9g  r^c|\}}|dd}t|||d<|||} tj| |d} || ddg} n | dd|dz|d zg} | ttd } | d | d k} t| j ddtf<| j | d f| j | dfz | j | tf<| D]\} } | | | | <| ddtg} | | dS)a> calculates summaries for every pixel block defined by pairwise combinations of the specified regions: calculates per-diagonal sums for intra-chromosomal blocks and overall sums for inter- chromosomal blocks. pixels in the blocks are labeled with regions' serial numbers, and groupby-s are used grouping. Trans-values have a special value for their diagonal/dist -1 Return: dictionary of DataFrames with diagonal sums and overall sums for the "fields", indexed with the (i,j) combinations of serial numbers of the regions. Nr-Freplacer<r=subset12)r<r=rwrzbin2_idbin1_id.sum)binsr pixelscoolerannotatedropnaastypeintrBrIrnitemsrHr: add_suffix)rNfields transformsr%regionsspanlohirrcis_maskfieldt_blockss r\_sum_rs"FB 88::aaa=Dg..DI ZZ\\"R% F _VT5 9 9 9FtTl33$# 57LM   ]]3// 0 0Fh6(#33H!1FJqqq*}'-z(I2E'FT\^gTgIh'hFJx#$$$&&""q& u nndD*566G 6?   + +F 3 33r^c Btdt||}tt dg|}d|D} t ||dd} n"#t $r} td| d} ~ wwxYwt|||} | D]} d| | <tt||||| }|||}td || }|rJt|D]:}||td j}t"j|j|| f<;|td |dt,|j|jd df |dt0|j|jddf |d dgdd|S)aU Intra-chromosomal diagonal summary statistics for asymmetric blocks of contact matrix defined as pairwise combinations of regions in "view_df. Note ---- This is a special case of asymmetric diagonal summary statistic that is efficient and covers the most important practical case of inter-chromosomal arms "expected" calculation. Parameters ---------- clr : cooler.Cooler Cooler object view_df : viewframe view_df of regions for intra-chromosomal diagonal summation, has to be sorted according to the order of chromosomes in cooler. transforms : dict of str -> callable, optional Transformations to apply to pixels. The result will be assigned to a temporary column with the name given by the key. Callables take one argument: the current chunk of the (annotated) pixel dataframe. clr_weight_name : str name of the balancing weight vector used to count "bad" pixels per diagonal. Set to `None` not to mask "bad" pixels (raw data only). chunksize : int, optional Size of pixel table chunks to process map : callable, optional Map functor implementation. Returns ------- Dataframe of diagonal statistics for all intra-chromosomal blocks defined as pairwise combinations of regions in the view rcountcg|]}|dS)r)rdrs r\ z sum_pairwise..zs999nnn999r^T check_sorting raise_errorszprovided view_df is not validNrhc0||dS)Nr) fill_value)add)df1df2s r\zsum_pairwise..sswwsqw'A'Ar^F)r drop_level)rinplacer<r0rr=)rrlr)rlenrlistrr Exception ValueErrorr}rrrJrrangexsrnr,npnanrIrCrm _REGION1_NAMEget_level_values _REGION2_NAME)rNr rr%r&r'mapspansrsummary_fields_e exp_tableagg_namejobresults result_df_d_idxs r\ sum_pairwiserIsK\ aSZZ\\**I 6 6E % :.. / /F99&999NA #       AAA899q@A -S'?[[[I#   (  sFJ9I9I9K9K  Cc#uooGAA7IVVI9 %% 9 9B<<*<GGMD24&IM$. / / D999 Q w{9?3S3STX3Y3Y[a3a'b'k'k'm'mnnn Q w{9?3S3STX3Y3Y[a3a'b'k'k'm'mnnn t 4FFF sA00 B:B  Bc  |t|}n7 t||dd} n"#t$r} td| d} ~ wwxYwtt dd} | i}d| d<d | d <n@t ||r|d z|d zd fdi}d| d<d| d <ntd|d| dkrtj| }|j }nt}tj } t|||||| |}| dkr| n #| dkr| wwxYw|| d|| d|| d <|ddid}|j|ddf|d<|j|ddf|d<|t+|j|t.jd||fvrT|j|ddf|d <|j|ddf|d!<|r#|dkrd d!g}n&|d"krd}ntd#t2t4g}|tt6k}|tt6k}| d| dg}|r~t9j|j|||| $}| d | d%z}||t|gg|pgtd&'}|j||f|j||f<n|r|j|g|pgtd(| d)!d*}|| dd*|| dd*||<|||j||f<n|j|| d f|j||f<|r|dkrQ|j|d d!gd(| d)!d*}nX|d"krC|j|}|j||f d)!d*}ntd+|| dd*|| dd*||<|||j||f<n|j|| d f|j||f<tj |z }tEj#d,|d-d.|S)/rNTrz0view_df is not a valid viewframe or incompatiblez.smooth)r+n_pixels smooth_suffixr5 n_contactsr2 contact_freqrrbalancedc8|d|z|zS)Nrr)pweight1weight2s r\rz$expected_full_fast..sAgJ7,Caj,Pr^r6r1z+cooler is not balanced, orbalancing weight z is not available in the cooler.r)r rr%r&r'rrr,r-r.r0r8r<r9r=)rIcolumnvaluer7rwrzgenomez2aggregate_cis could be only chrom, genome or False)rH sigma_log10colsrleft)onhow)observedr:r3z4aggregate_trans could be only chrom, genome or FalsezDone calculating full expected r)r*)$r r rrrn_NUM_VALID_NAMErmpPoolrr>r?rclosedividerCrDrErIrJrmrr/rrrrrBragg_smooth_cvdmergerHrKrr@rA) rNr rO aggregate_cisr#rPrQr&r%r'r(rrrrpoolmap_rRresultrV grp_columns _cis_mask _trans_maskrY _smooth_df_smooth_col_name_agg_df _trans_agg_df _trans_dfrTrrs @@r\expected_full_fastrsx&s++GG X'"! AA X X XOPPVW W X #"  D  (\*^ C 1 1  !C'!C' "P"P"P"P"PQ +\-^ R / R R R    qyywu~~x"$$J  !+%    199 JJLLL 199 JJLLLL $*$|*<#=#D#DtJ $$F4   ..6"" >&"3S"89BBDDF4L>&"3S"89BBDDF4L MMc&.))2FbfMUUU?M222%>&*;W*DENNPPx%>&*;W*DENNPPx5 G # ##X.KK h & &KKQRR R$m4 z"&66I$(88K:&\(:>R6S 922337=jDQ_L`A`6a 9223f g % %"J{3(H-==mM5!!F## M ( ( ;/I"J{M'AB5!!F## MSTT T.;lAS<[<[<[.\.c.c tJ/777 8/ / *+9FFZ8[ ; 44559? ;PTUcPdCd8e ; 445$&&3L LM<MMMMNNN Ms * A AA (DD6r-oecZ|r d}|d}|d} nd}|d} |d} tj||d} |r8| d| |z| | z| |<| | | || g} n| | | g} | | t| ti} | d| d z | d <| d | d k} | d | t | d <| || | d |gd | | d g} | || |z | |<| S)a  A function that returns pixel table with observed over expected. It takes a DataFrame of pixels (complete or a chunk, in pandas or dask formats), and merges it with the `expected_full` DataFrame, in order to assign appropriate expected for each pixels. This assignment is done according to the pixels' location, specifically - which combination of regions in the view and genomic distance for cis-pixels. Parameters ---------- pixels: pd.DataFrame | dask.DataFrame DataFrame of pixels bins : pd.DataFrame A bin table with a view column annotation. expected_full : pd.DataFrame DataFrame expected for all regions in the view used for annotation of bins. view_column_name : str Name of the column with the view annotations in `bins` and `expected_full` expected_column_name : str Name of the column with the expected values in `expected_full` clr_weight_name : str or None Name of balancing weight column from the cooler to use. Use raw unbalanced data, when None. oe_column_name : str Name of the column to store observed over expected in. Returns ------- pixels_oe : pd.DataFrame | dask.DataFrame DataFrame of pixels with observed/expected rrrrFrrrrr+rwrzr)rr)rrrrrwhererBr)rrr]view_column_namerQr%oe_column_nameobserved_column_name weight_col1 weight_col2 view_col1 view_col2 pixels_oers r\ obs_over_exprsT')(+++ (+++ &$&&&I#&&&Ie<< ?)L`Ba aIn r^r@Bc #K|"t|}n|}d}|dd} t| || |<t dt ||} | D]D} | \} } t|| | | |||||}|dd|gVEdS)an Generator yielding chunks of pixels with pre-caluclated observed over expected. Parameters ---------- clr : cooler.Cooler Cooler object expected_full : pd.DataFrame DataFrame expected for all pairwise combinations of regions in view_df view_df : viewframe viewframe of regions that were used to calculate expected_full expected_column_name : str Name of the column with the combined expected oe_column_name : str Name of the column to store observed over expected clr_weight_name : str or None Name of balancing weight column from the cooler to use. Use raw unbalanced data, when None. chunksize : int, optional Size of pixel table chunks to process and output Yields ------ pixel_df: pd.DataFrame chunks of pixels with observed over expected Nr-r)rrQr%rrr)r rJrr rrrr)rNr]r rQr%rr' view_arrayr bins_viewrrrroe_chunks r\obs_over_exp_generatorrsJ%c**3355 %%''  111 I"1)Z"H"HI aSZZ\\**I 6 6E??B JJLLB   -!5+)    9n=>>>>>??r^) NFFrFrrrrr)r-rrr)Nrrrr)4__doc__r>r@numpyrpandasrL multiprocessrr cooler.utilr itertoolsrrr functoolsrr cooltoolsr r cooltools.lib.commonr r cooltools.libr rcooltools.api.expectedrrcooltools.lib.schemasrrcooltools.sandboxrrrrrnrrB basicConfigINFOr]r}rrrrrrrr^r\r s<  !!!!!! &%%%%%%%  FEEEEEEE 100000)**1- )**1- T& ' ' * $+,,Q/',''''  ' IIIIX1 1 1 h343434r ____H' ddddV#TTTTt #????????????r^