DUfdZddlmZmZddlZddlZddlZddlmZddl m Z ddl Z ddl ZddlmZddlZddlmZmZdd lmZmZdd lmZmZmZdd lmZejd ejj ej ej!dZ"dZ#dZ$dZ%dZ&dZ'gdZ(dZ)dZ*dZ+e"fdZ, d6dZ-d7dZ.dZ/dZ0d Z1e"fd!Z2d"Z3e"fd#Z4d$e"fd%Z5e"d&d'd(d)fd*Z6ee7fd+Z8edde7fd,Z9 d8d5Z:dS)9a Collection of functions related to dot-calling The main user-facing API function is: .. code-block:: python dots( clr, expected, expected_value_col="balanced.avg", clr_weight_name="weight", view_df=None, kernels=None, max_loci_separation=10_000_000, max_nans_tolerated=1, n_lambda_bins=40, lambda_bin_fdr=0.1, clustering_radius=20_000, cluster_filtering=None, tile_size=5_000_000, nproc=1, ) This function implements HiCCUPS-style dot calling, but enables user-specified modifications at multiple steps. The current implementation makes two passes over the input data, first to create a histogram of pixel enrichment values, and second to extract significantly enriched pixels. - The function starts with compatibility verifications - Recommendation or verification for `kernels` is done next. Custom kernels must satisfy properties including: square shape, equal sizes, odd sizes, zeros in the middle, etc. By default, HiCCUPS-style kernels are recommended based on the binsize. - Lambda bins are defined for multiple hypothesis testing separately for different value ranges of the locally adjusted expected. Currently, log-binned lambda-bins are hardcoded using a pre-defined BASE of 2^(1/3). `n_lambda_bins` controls the total number of bins. for the `clr`, `expected` and `view` of interest. - Genomic regions in the specified `view`(all chromosomes by default) are split into smaller tiles of size `tile_size`. - `scoring_and_histogramming_step()` is performed independently on the genomic tiles. In this step, locally adjusted expected is calculated using convolution kernels for each pixel in the tile. All surveyed pixels are histogrammed according to their adjusted expected and raw observed counts. Locally adjusted expected is not stored in memory. - Chunks of histograms are aggregated together and a modified BH-FDR procedure is applied to the result in `determine_thresholds()`. This returns thresholds for statistical significance in each lambda-bin (for observed counts), along with the adjusted p-values (q-values). - Calculated thresholds are used to extract statistically significant pixels in `scoring_and_extraction_step()`. Because locally adjusted expected is not stored in memory, it is re-caluclated during this step, which makes it computationally intensive. Locally adjusted expected values are required in order to apply different thresholds of significance depending on the lambda-bin. - Returned filtered pixels, or 'dots', are significantly enriched relative to their locally adjusted expecteds and thus have potential biological interest. Dots are further annotated with their genomic coordinates and q-values (adjusted p-values) for all applied kernels. - All further steps perform optional post-processing on called dots - enriched pixels that are within `clustering_radius` of each other are clustered together and the brightest one is selected as the representative position of a dot. - cluster-representatives along with "singletons" (enriched pixels that are not part of any cluster) can be subjected to further empirical enrichment filtering in `cluster_filtering_hiccups()`. This both requires clustered dots exceed prescribed enrichment thresholds relative to their local neighborhoods and that singletons pass an even more stringent q-value threshold. )partialreduceN)convolve)poisson)Birch) LazyToeplitz get_kernel)assign_regionsmake_cooler_view)is_cooler_balancedis_compatible_viewframeis_valid_expected)pool_decoratorignore)actioncategory)levelcountzexp.rawcd|dS)Nla_exp..value kernel_names T/var/www/html/software/conda/lib/python3.11/site-packages/cooltools/api/dotfinder.pyris(E+(E(E(Ecd|dS)Nr.nnansrrs rrrjs)F;)F)F)Frbin1_idbin2_id)chrom1start1end1chrom2start2end2c&t||z SN)int) basepairsbinsizes r bp_to_binsr.xs y7" # ##rc |dkrtd|d|dkrd\n+|dkrd\n|dkrd \ntd |d tjd d d|gd}fd|D}|S)a: Return a recommended set of convolution kernels for dot-calling based on the resolution, or binsize, of the input data. This function currently recommends the four kernels used in the HiCCUPS method: donut, horizontal, vertical, lowerleft. Kernels are recommended for resolutions near 5 kb, 10 kb, and 25 kb. Dots are not typically visible at lower resolutions (binsize >28kb) and the majority of datasets are too sparse for dot-calling at very high resolutions (<4kb). Given this, default kernels are not recommended for resolutions outside this range. Parameters ---------- binsize : integer binsize of the provided cooler Returns ------- kernels : {str:ndarray} dictionary of convolution kernels as ndarrays, with their names as keys. i`mz"Provided cooler has resolution of zc bases, which is too coarse for automated kernel recommendation. Provide custom kernels to proceed.iPF)i@)ri)zProvided cooler has resolution za bases, which is too fine for automated kernel recommendation. Provide custom kernels to proceed.z-Using recommended donut-based kernels with w=z, p=z for binsize=)donutvertical horizontallowleftc4i|]}|t|Sr)r ).0kpws r z%recommend_kernels..s'<<z)is_compatible_kernels..s"999ASVV999rz,all 'kernels' must have the same size, now: r1rrzASize of the convolution kernels has to be odd and > 3, currently z)Too many NaNs allowed max_nans_tolerated=T) isinstancedictr?itemsminmax is_integer)rCr-max_nans_tolerated kernel_widths kernel_widthkernel_half_widths ris_compatible_kernelsrUs gt $ $  R   :9999M =S////W WWXXX}%%L%)Q.Q'8'C'C'E'E ^P\ ^ ^     - - L8J L L   4rc |}t|j}|D]\}}t j|j}t j|d|d||d<||d |dg|d|dgd}d|d }| d |i }| ||j d d |fS) a Add columns with the qvalues to a DataFrame of scored pixels Parameters ---------- pixels_df : pandas.DataFrame a DataFrame with pixel coordinates that must have at least 2 columns named 'bin1_id' and 'bin2_id', where first is pixels's row and the second is pixel's column index. qvalues : dict of DataFrames A dictionary with keys being kernel names and values DataFrames storing q-values for each observed count values in each lambda- bin. Colunms are Intervals defined by 'ledges' boundaries. Rows corresponding to a range of observed count values. obs_raw_name : str Name of the column/field that carry number of counts per pixel, i.e. observed raw counts. Returns ------- pixels_qvalue_df : pandas.DataFrame DataFrame of pixels with additional columns la_exp.{k}.qval, storing q-values (adjusted p-values) corresponding to the count value of a pixel, its kernel, and a lambda-bin it belongs to. rr)binslbinsF ignore_index)_)left_onright_onsuffixes.qvalvaluecolumnsN) copylistrcrMpd IntervalIndexcutmergemelt reset_indexrenameappendloc) pixels_dfqvalues obs_raw_namepixels_qvalue_dfcolsr;qval_dfrX qval_col_names rannotate_pixels_with_qvaluesrvs86!~~''  ( ) )Dmmoo## 7 11$&F 0q000 1% % % !,11 LLeL , , 8 8 : :!7+"$7a$7$7$78 2  +!*** +22G];S2TT M""""  4 ((rc_labelc_sizec |||gjtj}|j}t d|d}|||j} |j} tj | dd\} } } | | }tj | | d}tj d| j d| d d | d d t!j||d |zd |zg }| tj||<|tj||<|S)a Group significant pixels by proximity using Birch clustering. We use "n_clusters=None", which implies no AgglomerativeClustering, and thus simply reporting "blobs" of pixels of radii <="threshold_cluster" along with corresponding blob-centroids as well. Parameters ---------- pixels_df : pandas.DataFrame a DataFrame with pixel coordinates that must have at least 2 columns named 'bin1_id' and 'bin2_id', where first is pixels's row and the second is pixel's column index. threshold_cluster : int clustering radius for Birch clustering derived from ~40kb radius of clustering and bin size. bin1_id_name : str Name of the 1st coordinate (row index) in 'pixel_df', by default 'bin1_id'. 'start1/end1' could be usefull as well. bin2_id_name : str Name of the 2nd coordinate (column index) in 'pixel_df', by default 'bin2_id'. 'start2/end2' could be usefull as well. clust_label_name : str Name of the cluster of pixels label. "c_label" by default. clust_size_name : str Name of the cluster of pixels size. "c_size" by default. Returns ------- peak_tmp : pandas.DataFrame DataFrame with the following columns: [c+bin1_id_name, c+bin2_id_name, clust_label_name, clust_size_name] row/col (bin1/bin2) are coordinates of centroids, label and sizes are unique pixel-cluster labels and their corresponding sizes. NT) n_clusters thresholdcompute_labels)return_inverse return_countsraxisz detected z clusters of z.2fz+/-z sizec)indexrc)valuesastypenpfloat64rrfitlabels_subcluster_centers_uniquetaker@rAsizemeanstdrf DataFrameint64)rothreshold_cluster bin1_id_name bin2_id_nameclust_label_nameclust_size_namepixels pixel_idxsbrcclustered_labelsclustered_centroids uniq_labels inverse_idx uniq_counts cluster_sizescentroids_per_pixelcentroids_n_labels_dfs rclust_2D_pixelsrsZ l3 4 ; B B2: N NFJ #    C GGFOOO{1,.IT---)Kk ,M'"57GaPPP LjK$jj;3C3C3E3EjjjkooN_N_jjjj L|#S<%78 /?.E.Ebh.O.O*+-:-A-A"(-K-K/*  rc #K||zr ||zdz}n||z}tjd|d|d||zd|d|d |rtjd|dt|D]}t|D]v}td ||z|z }td ||z|z }t |||dzz|z} t |||dzz|z} ||z| |zf||z| |zffVwd S) ay Generate a stream of coordinates of tiles that cover a matrix of a given size. Matrix has to be square, on-digaonal one: e.g. corresponding to a chromosome or a chromosomal arm. Parameters ---------- matrix_size : int Size of a squared matrix offset : int Offset coordinates of generated tiles by 'offset' tile_size : int Requested size of the tiles. Tiles near the right and botoom edges could be rectangular and smaller then 'tile_size' pad : int Small padding around each tile to be included in the yielded coordinates. Yields ------ Pairs of indices/coordinates of every tile: (start_i, end_i), (start_j, end_j) Notes ----- Generated tiles coordinates [start_i,end_i) , [start_i,end_i) can be used to fetch heatmap tiles from cooler: >>> clr.matrix()[start_i:end_i, start_j:end_j] 'offset' is useful when a given matrix is part of a larger matrix (a given chromosome or arm), and thus all coordinated needs to be offset to get absolute coordinates. Tiles are non-overlapping (pad=0), but tiles near the right and bottom edges could be rectangular: * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * ... * * * * * * * * * * * * * * * * * r1z matrix Xz to be split into z tiles of .z tiles are padded (width=z&) to enable convolution near the edgesrN)r@debugrangerOrN) matrix_sizeoffset tile_sizepad num_tilestitjstart_istart_jend_iend_js rtile_square_matrixrs`Y-9,q0 9,  My;yyyy I@Uyyajyymvyyy   S S S S   I Y Y "" Y YB!Y^c122G!Y^c122G Y"q&%9C%?@@E Y"q&%9C%?@@EV#UV^4w7GQW6XX X X X X Y Y Yrc#bK|dD]\}}}}||||f\} } | | z } t| | ||D]Z\} } | d| dz }| d| dz }d|}}t||t ||z d|zkr|| | fV[dS)a A generator yielding corrdinates of heatmap tiles that are needed to cover the requested band_to_cover around diagonal. Each tile is "padded" with the pad of size 'pad_size' to allow for convolution near the boundary of a tile. Parameters ---------- clr : cooler Cooler object to use to extract chromosome extents. view_df : viewframe Viewframe with genomic regions to process, chrom, start, end, name. pad_size : int Size of padding around each tile. Typically the outer size of the kernel. tile_size : int Size of the heatmap tile. band_to_cover : int Size of the diagonal band to be covered by the generated tiles. Typically correspond to the max_loci_separation for called dots. Returns ------- tile_coords : tuple Generator of tile coordinates, i.e. tuples of three: (region_name, tile_span_i, tile_span_j), where 'tile_span_i/j' each is a tuple of bin ids (bin_start, bin_end). F)r)rrrrrr1rN) itertuplesextentrrNrO)clrview_dfpad_sizer band_to_coverchromstartend region_name region_start region_end region_size tile_span_i tile_span_jtile_diag_start tile_diag_end band_startband_ends rgenerate_tiles_diag_bandrs :+2*<*<5*<*I*I<<&uc;#&::ueS.A#B#B j </ (:# ) ) )  < < $K*!n{1~=O'N[^;M#$mJHm,,s:/O/OOH "; ;;;;# <<   Xgw//N K~ . .E;=&E" ubA 6 6 67:<&E" ubA 6 6 67 Ie^ , ,E M"(5//28E?? ; ;EE%LE%LE%L :ek " "DAq| B17799r>\a\g\g\i\ijjkkH Hh 7 7 7-d-d#*==??, d, d K%js1MMMB%js1MMMB  RX&&1$$RX.. B')iB&7&7 #[(?@@F MV;VVBVVQSVVV    7=llnnH2{222 368hhjjH2{222 379{CZC`C`CbCb7c7cH3k33 4 4Y, d-d-d-d-d-d-d-d-d-d-d-d-d-d-d-d` Os ELL Lc|\}} } | d| df} t|j||f|} |dt | t | f} | t | t | f}|t | |}|t | |}t | | |||f|}|d|dk}|d|d|z k}tj|d|D|kd }|d |Dd }|||z|z|z d }|gdd|Dz d|DS)ag The main working function that given a tile of a heatmap, applies kernels to perform convolution to calculate locally-adjusted expected and then calculates a p-value for every meaningfull pixel against these locally-adjusted expected (la_exp) values. Parameters ---------- tile_cij : tuple Tuple of 3: region name, tile span row-wise, tile span column-wise: (region, tile_span_i, tile_span_j), where tile_span_i = (start_i, end_i), and tile_span_j = (start_j, end_j). clr : cooler Cooler object to use to extract Hi-C heatmap data. expected_indexed : pandas.DataFrame DataFrame with cis-expected, indexed with 'region1', 'region2', 'dist'. expected_value_col : str Name of a value column in expected DataFrame clr_weight_name : str Name of a value column with balancing weights in a cooler.bins() DataFrame. Typically 'weight'. kernels : dict A dictionary with keys being kernels names and values being ndarrays representing those kernels. max_nans_tolerated : int Number of NaNs tolerated in a footprint of every kernel. band_to_cover : int Results would be stored only for pixels connecting loci closer than 'band_to_cover'. Returns ------- res_df : pandas.DataFrame results: annotated pixels with calculated locally adjusted expected for every kernels, observed, precalculated pvalues, number of NaNs in footprint of every kernels, all of that in a form of an annotated pixels DataFrame for eligible pixels of a given tile. rF)balance)rrrrrCr!r"cg|]}d|d S)rr rr:r;s rrJzscore_tile..0s$555#!###555rr1rcg|]}d|S)rrrs rrJzscore_tile..4s! G G G!!5!!5!5 G G GrrcTdroprcg|]}d|d Srrrrs rrJzscore_tile..>s$*P*P*P1+>Q+>+>+>*P*P*Prci|] }d|dd S)rrrrrs rr>zscore_tile..?s&DDDq'a'''DDDr)dtype) r rnto_numpymatrixslicerWrrallrkr)tile_cijrexpected_indexedexpected_value_colclr_weight_namerCrQrrrr tile_start_ijlazy_exprr bal_weight_i bal_weight_jresult upper_bandis_inside_banddoes_comply_nansfinite_values_onlyres_dfs r score_tilers+d-5)Kk ^[^4M [+567IJSSUUH zz%z(( )Q)QRH{+UK-@@AH88::e[12?CLLNNL88::e[12?CLLNNL2!<0 F "VI%66JI&&*;m*KLNv55W55569KKRS  G Gw G G GHLLR[L\\J/2BBEWW X d d eF '''*P*P*P*P*PP  fDDGDDDfEEFrcHi}|D]}tj|d|d|}|||gddd|ddtj||<|S)a An attempt to implement HiCCUPS-like lambda-binning statistical procedure. This function aims at building up a histogram of locally adjusted expected scores for groups of characterized pixels. Such histograms are subsequently used to compute FDR thresholds for different "classes" of hypothesis (classified by their locally-adjusted expected (la_exp)). Parameters ---------- scored_df : pd.DataFrame A table with the scoring information for a group of pixels. kernels : dict A dictionary with keys being kernels names and values being ndarrays representing those kernels. ledges : ndarray An ndarray with bin lambda-edges for groupping locally adjusted expecteds, i.e., classifying statistical hypothesis into lambda-bins. Left-most bin (-inf, 1], and right-most one (value,+inf]. obs_raw_name : str Name of the column/field that carry number of counts per pixel, i.e. observed raw counts. Returns ------- hists : dict of pandas.DataFrame A dictionary of pandas.DataFrame with lambda/observed 2D histogram for every kernel-type. Notes ----- returning histograms corresponding to the chunks of scored pixels. rrF)dropnarr) rfrhgroupbyrunstackfillnarrr) scored_dfrCledgesrqhistsr;rXs rhistogram_scored_pixelsrBsN E   y!41!4!4!45v>>   |U3EE  R R#!### UWW WYY VAYY VBH   a Lrci}i}|D]\}}|jddddjddd}|jdddf}tj|}|jD]8} |j} tj | | j || <9||z}|j dz} ||z|z } | | dk| t$j||<||z ||<||||dkd||<tj||j||_||fS)aJ given a 'gw_hist' histogram of observed counts for each lambda-bin and for each kernel-type, and also given a FDR, calculate q-values for each observed count value in each lambda-bin for each kernel-type. Parameters ---------- gw_hist_kernels : dict dictionary {kernel_name : 2D_hist}, where '2D_hist' is a pd.DataFrame fdr : float False Discovery Rate level Returns ------- threshold_df : dict each threshold_df[k] is a Series indexed by la_exp intervals (IntervalIndex) and it is all we need to extract "good" pixels from each chunk ... qvalues : dict A dictionary with keys being kernel names and values pandas.DataFrames storing q-values: each column corresponds to a lambda-bin, while rows correspond to observed pixels values. Nrrr1g?)rMiloccumsumrfr reindex_likercrrrsfrightrOcummaxmaskidxminrrrrcumminrg) gw_histfdrrp threshold_dfr;_histrcs_histnorm unit_Poissonlbin _occurances _high_valuefdr_diffs rdetermine_thresholdsr4sfGLMMOO%H%H5:ddd#***227"=}QT"|~~228<< $ E ED".1133K!(K!D!DL  l* n((**Q. 8^|3;;== MM(Q, ' ' VXX VK VBH   Q #X-5577 QZ__WQZ#%5s;; " 0a1F G G Q   rcXg}|D]x\}}|d|d}tj||d}|j|} |||| ky|t j|dS)a Implementation of HiCCUPS-like lambda-binning statistical procedure. Use FDR thresholds for different "classes" of hypothesis (classified by their locally-adjusted expected (la_exp) scores), in order to extract "enriched" pixels. Parameters ---------- scored_df : pd.DataFrame A table with the scoring information for a group of pixels. thresholds : dict A dictionary {kernel_name : lambda_thresholds}, where 'lambda_thresholds' are pd.Series with FDR thresholds indexed by lambda-bin intervals ledges : ndarray An ndarray with bin lambda-edges for groupping locally adjusted expecteds, i.e., classifying statistical hypothesis into lambda-bins. Left-most bin (-inf, 1], and right-most one (value,+inf]. obs_raw_name : str Name of the column/field with number of counts per pixel, i.e. observed raw counts. Returns ------- scored_df_slice : pandas.DataFrame Filtered DataFrame of pixels that satisfy thresholds. rrF)labelsrr)rMrfrhr rmrrr) r thresholdsrrqcompliant_pixel_masksrr{lambda_of_pixelslbin_idxthreshold_of_pixelss rextract_scored_pixelsr<s8","2"2"4"4    Y$%B{%B%B%BC6*F5AAA'nX6$$ l # , , . .2E2N2N2P2P P     RV1::: ;;rregionc 2dddd|h|std|}||jvrMt jdt j|d|dk|dt j||<g}| |d}|D]D\}}t j d |t||dd }| |Et j d |sMt jd tjgt|j|d z|dzddddgz} | Stj|d} tj|| ddd} | |t&| |d z<| |t&| |dz<| |d z|dzdgd} | j| |} | S)a Group together adjacent significant pixels into clusters after the lambda-binning multiple hypothesis testing by iterating over assigned regions and calling `clust_2D_pixels`. Parameters ---------- scored_df : pandas.DataFrame DataFrame with enriched pixels that are ready to be clustered and are annotated with their genomic coordinates. dots_clustering_radius : int Birch-clustering threshold. assigned_regions_name : str | None Name of the column in scored_df to use for grouping pixels before clustering. When None, full chromosome clustering is done. obs_raw_name : str name of the column with raw observed pixel counts Returns ------- centroids : pandas.DataFrame Pixels from 'scored_df' annotated with clustering information. Notes ----- 'dots_clustering_radius' in Birch clustering algorithm corresponds to a double the clustering radius in the "greedy"-clustering used in HiCCUPS r#r&r$r'z7Scored pixels provided for clustering are not annotatedzMNo regions assigned to the scored pixels before clustering, using chromosomesT)rz&clustering enriched pixels in region: )rrrzClustering is completez7No clusters found for any regions! Output will be empty12rwrxcstart1cstart2rbFrYleft)how left_index right_index)issubsetr?rdrcr@warningrwhererrrArrmrfrreconcatrirstrrnidxmax)rdots_clustering_radiusassigned_regions_namerqpixel_clust_listscored_pixels_by_regionr=_df pixel_clust empty_outputpixel_clust_dfdfchrom_clust_group centroidss rclustering_steprX sF h(L A J J9 U UTRSSS  I !Y%6 6 6 \   ,.8 h 9X#6 6 (8KRV, , '( '//0EPT/UU. - -  FfFFGGG% 4!!     ,,,, L)***  QRRR| *++%+%+      5    >v$D   B'))>&?&F&Fs&K&KBs"#&()>&?&F&Fs&K&KBs"#  $&;c&A9M#,'..00I rg?g?g@g{Gz?cd|vrtddddd|h|stdhd}||std ||||d zk||||d zkz||||d zkz||||d zkz||||d zk||||d zkzz|ddk|d|dz|dz|dz|kzz}tjd|dt |d||dS)ag Centroids of enriched pixels can be filtered to further minimize the amount of false-positive dot-calls. First, centroids are filtered on enrichment relative to the locally-adjusted expected for the "donut", "lowleft", "vertical", and "horizontal" kernels. Additionally, singleton pixels (i.e. pixels that do not belong to a cluster) are filtered based on a combined q-values for all kernels. This empirical filtering approach was developed in Rao et al 2014 and results in a conservative dot-calls with the low rate of false-positive calls. Parameters ---------- centroids : pd.DataFrame DataFrame that stores enriched and clustered pixels. obs_raw_name : str name of the column with raw observed pixel counts enrichment_factor_vh : float minimal enrichment factor for pixels relative to both "vertical" and "horizontal" kernel. enrichment_factor_d_and_ll : float minimal enrichment factor for pixels relative to both "donut" and "lowleft" kernels. enrichment_factor_d_or_ll : float minimal enrichment factor for pixels relative to either "donut" or" "lowleft" kenels. FDR_orphan_threshold : float minimal combined q-value for singleton pixels. Returns ------- filtered_centroids : pd.DataFrame filtered dot-calls rxz7input dataframe of pixels does not seem to be clusteredr#r&r$r'zKinput dataframe of clustered pixels provided for filtering is not annotated>la_exp.donut.qvalla_exp.donut.valuela_exp.lowleft.qvalla_exp.lowleft.valuela_exp.vertical.qvalla_exp.vertical.valuela_exp.horizontal.qvalla_exp.horizontal.valuezNclustered pixels provided for filtering were not scored with 4 hiccups kernelsr]r[r_rar1r\rZr^r`z filtered z out of z2 centroids to reduce the number of false-positivesTr)r?rGr@rAsumrHrk)rWrqenrichment_factor_vhenrichment_factor_d_and_llenrichment_factor_d_or_llFDR_orphan_threshold_hiccups_kernel_cols_setenrichment_fdr_complys rcluster_filtering_hiccupsriss!X y STTT h(L A J J9 U U  Y      $ , ,Y 7 7  \    l #(95K+LL M l #(95I+JJ K   l #"Y/F%GG H  l #"Y/H%II J $,'+i8N.OOP,'+i8L.MMN  # 6x 1 $34 345 678 89: ( ( 7% R L{)--//{{Y{{{ * + 7 7T 7 B BBrc  tjdt|dtt||||||tt |fd} | dkr@t ttj t|| z } ni} | | |fi| } fd}t|| }D]}||j d d d f}| d krtd |d |jd||j|| d kd }||jd d d |f||<||jjstd|d|S)z This implements the 1st step of the lambda-binning scoring procedure - histogramming. In short, this pipes a scoring operation together with histogramming into a single pipeline of per-chunk operations/transforms. convolving z* tiles to build histograms for lambda-binsrrrr rCrQr)rCrc,|Sr*r)tileto_histto_scores rrz0scoring_and_histogramming_step..swwxx~~..rr1 chunksizeci}D]X}||||ddtj||<Y|S)Nr) fill_value)addrrrr)hxhyhxyr;rCs r _sum_histsz2scoring_and_histogramming_step.._sum_hists$s_ O OAUYYr!uY33::1==DDRXNNCFF rNrrzThere are la_exp.z .value in z, please check the histogramzHistogram for z-kernel is not sorted)r@rArHrrrrLr+rceilrr rbr?namercrnris_monotonic_increasing)rrrr tilesrCrrQloci_separation_binsnproc map_functorjob map_kwargshistogram_chunksry final_histr;last_lambda_binlast_non_empty_lbinrorps ` @@rscoring_and_histogramming_steprs( LUs5zzUUUVVV )-'-*   H-wvNNNG / . . . .C qyyCE U0B(C(C$D$DEEE  #{3<<<< $455J H H$Q-,QQQU3    A % %cAcc1Eccc )m3JqM4E4E4G4G!4KLRP"1 )!!!-A.A-A*AB 1 !}": HFaFFFGG G H rc tjdt|dtt||||||| tt ||fd}| dkrhtjd| dt|d t ttj t|| z  }ntjd i}| ||fi|}tj |d }| rtd|| | gd S)z This implements the 2nd step of the lambda-binning scoring procedure, extracting pixels that are FDR compliant. In short, this combines scoring with with extraction into a single pipeline of per-chunk operations/transforms. rkz! tiles to extract enriched pixelsrl)r7rc,|Sr*r)rn to_extractrps rrz-scoring_and_extraction_step..oszz((4..11rr1zcreating a Pool of z workers to tackle z tilesrqz"fallback to serial implementation.TrYzRSome pixels were scored more than one time, matrix tiling procedure is not correct)byr)r@rArHrrr<rLr+rrzrfrJ duplicatedanyr? sort_valuesrk)rrrr r}rCrr7rQr~rrrrrrfiltered_pix_chunkssignificant_pixelsrrps @@rscoring_and_extraction_stepr@s2 LLs5zzLLLMMM )-'-*   HJ 2 1 1 1 1C qyy W5WWSZZWWWXXXCE U0B(C(C$D$DEEE  9::: &+c5??J??#6TJJJ$$&&**,,  a     ) )lL-I ) J J V V  W  r balanced.avgweight逖r1(皙? N@KLc|t|}n7 t||dd}n"#t$r}td|d}~wwxYw|r: t ||d}n5#t$r}td|d|d}~wwxYwtd t |d |||gd }n"#t$r}td |d}~wwxYw|gd }|j}t||}t| |}|r&t|||rtj d nt|}td|D}t!|dz dz }d|cxkrdksntd|d}t#jt"j gt#jd|dz ||t"jt"jgf}t-t/|||||}t1j}t5||||||||||  }t1j|z }t7jd|ddt;|| \}}t7jdt1j}t=||||||||||| dd }t1j|z }t7jd|ddt7jd t?|d!t7jd"tA||}tCj"||#gd#d$}| s=g} | tHz } | tJgz } | d%|Dz } | d&|Dz } || StM||}tO|| (d'}!g} | tHz } | d(d)d*d+tJgz } | d,|Dz } | d-|Dz } | | rtS|!}"n| s|!}"|"S).a Call dots on a cooler {clr}, using {expected} defined in regions specified in {view_df}. All convolution kernels specified in {kernels} will be all applied to the {clr}, and statistical testing will be performed separately for each kernel. A convolutional kernel is a small squared matrix (e.g. 7x7) of zeros and ones that defines a "mask" to extract local expected around each pixel. Since the enrichment is calculated relative to the central pixel, kernel width should be an odd number >=3. Parameters ---------- clr : cooler.Cooler A cooler with balanced Hi-C data. expected : DataFrame in expected format Diagonal summary statistics for each chromosome, and name of the column with the values of expected to use. expected_value_col : str Name of the column in expected that holds the values of expected clr_weight_name : str Name of the column in the clr.bins to use as balancing weights. Using raw unbalanced data is not supported for dot-calling. view_df : viewframe Viewframe with genomic regions, at the moment the view has to match the view used for generating expected. If None, generate from the cooler. kernels : { str:np.ndarray } | None A dictionary of convolution kernels to be used for calculating locally adjusted expected. If None the default kernels from HiCCUPS are going to be recommended based on the resolution of the cooler. max_loci_separation : int Miaximum loci separation for dot-calling, i.e., do not call dots for loci that are further than max_loci_separation basepair apart. default 10Mb. max_nans_tolerated : int Maximum number of NaNs tolerated in a footprint of every used kernel Adjust with caution, as large max_nans_tolerated, might lead to artifacts in pixels scoring. n_lambda_bins : int Number of log-spaced bins, where FDR-testing will be performed independently. TODO: generate lambda-bins on the fly based on the dynamic range of the data (i.e. maximum pixel count) lambda_bin_fdr : float False discovery rate (FDR) for multiple hypothesis testing BH-FDR procedure, applied per lambda bin. clustering_radius : None | int Cluster enriched pixels with a given radius. "Brightest" pixels in each group will be reported as the final dot-calls. If None, no clustering is performed. cluster_filtering : bool whether to apply additional filtering to centroids after clustering, using cluster_filtering_hiccups() tile_size : int Tile size for the Hi-C heatmap tiling. Typically on order of several mega-bases, and <= max_loci_separation. Controls tradeoff between memory consumption and speed of execution. nproc : int Number of processes to use for multiprocessing. Returns ------- dots : pandas.DataFrame BEDPE-style dataFrame with genomic coordinates of called dots and additional annotations. Notes ----- 'clustering_radius' in Birch clustering algorithm corresponds to a double the clustering radius in the "greedy"-clustering used in HiCCUPS (to be tested). TODO describe sequence of processing steps NT) check_sorting raise_errorsz0view_df is not a valid viewframe or incompatible)rz#provided cooler is not balanced or z is missingz*calling dots on raw data is not supported.cis) verify_coolerexpected_value_colsrz#provided expected is not compatible)region1region2distzVCompatibility checks for 'kernels' are not fully implemented yet, use at your own riskc34K|]}t|VdSr*rGrs r zdots..s(88!s1vv888888rr1rr2zIncompatible n_lambda_bins=gr(?r)numbaser)rr r}rCrrQr~rzDone building histograms in z.3fz sec ...z.Determined thresholds for every lambda-bin ...r!r") rr r}rCrr7rQr~rrrz#Done extracting enriched pixels in zBegin post-processing of z filtered pixelsz(preparing to extract needed q-values ...)rrr)replacecg|]}d|d Srrrs rrJzdots..ms$===+!+++===rcg|]}d|d Srr`rrs rrJzdots..ns$<<.s$999A'a'''999rcg|]}d|d Srrrs rrJzdots..s$8881&a&&&888r)*r r Exceptionr?r r set_index sort_indexr-r.rUwarningswarnrDrOrr+r concatenateinflogspacerrertime perf_counterrr@rAr4rrHrvcoolerannotaterWbedpe_required_colsobserved_count_namer rXrkri)#rrrr rrCmax_loci_separationrQ n_lambda_binslambda_bin_fdrclustering_radiuscluster_filteringrrr\er-r~tile_size_binsrSrTBASErr} time_startr) elapsed_timer+rpfiltered_pixelsfiltered_pixels_qvalsfiltered_pixels_annotated output_colsrWpostprocessed_callss# rdotsrsj"3'' X'"! AA  X X XOPPVW W X G "3dKKKAA   RoRRR   EFFF G    "!  GGG>??QFG!!"@"@"@AALLNNHkG%&97CC 733N-(';MNN- d    $G,,88w~~'7'788888L\A-233  $ $ $ $" $ $ $ $F}FFGGG D ^fWI K!!j    VH   F  +^=Q    E"$$J, -'-1   G$&&3L LJ JJJJKKK1.IIL' LABBB"$$J1 -'-1O$&&3L LQ|QQQQRRR LSS-A-ASSSTTT L;<<<9'RR &sxxzz*C*C*CDd!!!  6 **     ==W==== <>BB-- C 7CC )rr!r"rwrx)r) rrNNrr1rrrNrr1);__doc__ functoolsrrr@rr scipy.ndimager scipy.statsrnumpyrpandasrfsklearn.clusterrr lib.numutilsr r lib.commonr r lib.checksr rrr simplefiltererrorsPerformanceWarning basicConfigINFOrexpected_count_nameadjusted_exp_namenans_inkernel_namerrrr.rDrUrvrrrrrrr4r<rXrimaprrrrrrrsJJX&%%%%%%% """"""!!!!!! 3333333399999999 ('''''X 0LMMMM',''''EEFF  $$$555p***ZCV0)0)0)0)j c!c!c!c!VGYGYGYGYT1<1<1