DUfg[ <dZddlmZddlZddlZddlZddlmZddlZ ddl Z dgZ e j e jjZe je je jeedZgdZdd d dd dd d d d ddd Zd ddd dd ddZiZdJdZeddKdZed dLdMdZed dLdMdZed dKd!Zed"dKd#Zed$dKd%Zed&dKd'Z ed(dKd)Z!ed*dKd+Z"ed,dKd-Z#ed.dKd/Z$ed0dKd1Z% dNdOd8Z&dKd9Z'dPd;Z(dQd=Z) dLd>d?d2dd2d?d@dAdRdIZ*dS)SaValidate a BED dataframe against the BED specification. The BED specification is defined here: https://github.com/samtools/hts-specs/blob/master/BEDv1.pdf Some facts ---------- * Intervals are 0-based, half-open * Fields use 7-bit printable ASCII, including spaces but excluding tabs, newlines and other control characters * Flavors: BED{3,4,5,6,7,8,9,12}+m: m corresponds to custom fields, you can also do BEDn+ for an unspecified number of custom fields * First 3 fields are mandatory, last 9 are optional * BED10 and BED11 are illegal * Order is "binding": if an optional field is filled then all previous ones must also be filled * Standard BED fields can never be empty - must use a special null or "uninformative" placeholder value * Custom BED fields can be empty when a single tab is used as delimiter Delimiters ---------- While the BED spec allows for the use of either spaces or tabs as delimiters, even permitting a mixture in the same file, we do not validate any of the whitespace constraints imposed on fields in the spec to deal with the possibility of either space or mixed whitespace delimiters in the file. We assume that the dataframe will be written using a single tab as the sole delimiter, as recommended. Information we are agnostic to: - The delimiter used in the file before it was parsed: we work with files that have already been parsed into a dataframe. - Comment lines and blank lines: we assume that the dataframe contains only data lines. - Custom field names, dtypes, and values. Information supplied out-of-band: - Assembly/chromsizes: a dictionary or pandas Series mapping chromosome names to lengths [optional]. - Custom fields in positions 4-12: which of the first 4 to 12 fields are provided as standard BED fields and which are custom fields. Note that the spec is overly strict. For example, many BED files in the wild will use "." as the uninformative score value for all features, but the spec requires that the score be an integer between 0-1000. We provide some lenience by allowing floats as well, which many tools use in practice. The spec defines the uninformative score value as 0. We also don't enforce limiting name fields to 7-bit printable ascii. ) annotationsN)Callableto_bed)IntegerUnsignedFloat CharacterString) chromstartendnamescorestrand thickStartthickEnditemRgb blockCount blockSizes blockStartsOUiuiufiOU_.0)r r r rrrrcolstrreturnrcdfd }|S)Nfuncrr c|t<|SN)BED_FIELD_VALIDATORS)r"rs L/var/www/html/software/conda/lib/python3.11/site-packages/bioframe/io/bed.py decoratorzvalidator..decoratorws$(S! )r"rr r)rr's` r& validatorr*vs) r(r df pd.DataFrame dict[bool]c|djd}|dj}|dk|dkz}||dS)z Validate the chromosome names of a BED dataframe. The chrom column is limited to non-whitespace word characters only (alphanumeric characters and underscores). Each name must be between 1 and 255 characters in length, inclusive. r z^[A-Za-z0-9_]+$)zchrom.is_alnumzchrom.is_len_ok)rmatchalllen)r+is_alnumlengths is_len_oks r& check_chromr7~sx'{$$%788<<>>Hko!!##GQ,7c>27799I#$  r(r chromsizesdict | pd.Series | Nonec|ddk}|dtk}||d}|Ctj|}|d||dk}||d<|S)at Validate the start coordinates of a BED dataframe. Start must be an integer greater than or equal to 0 and less than or equal to the total number of bases of the chromosome to which it belongs. If the size of the chromosome is unknown, then start must be less than or equal to 2**64 - 1, which is the maximum size of an unsigned 64-bit integer. r r)zstart.is_nonnegzstart.is_le_64Nr r zstart.is_lt_chromr2 UINT64_MAXpdSeries)r+r8 is_nonnegis_le_64out is_lt_chroms r& check_startrCsG!&&((I7 z)..00H%"  C Yz** %y:bk#::??AA #.  Jr(r cf|ddk}|dtk}|d|dk}|||d}|Ctj|}|d||dk}||d<|S)az Validate the end coordinates of a BED dataframe. End must be an integer greater than or equal to the value of start and less than or equal to the total number of bases in the chromosome to which it belongs. If the size of the chromosome is unknown, then end must be less than or equal to 2**64 - 1, the maximum size of an unsigned 64-bit integer. r rr )z end.is_nonnegz end.is_le_64zend.is_end_ge_startNr zend.is_le_chromr;)r+r8r?r@is_end_ge_startrA is_le_chroms r& check_endrGsEa$$&&I5 Z',,..H%yBwK/4466O# .  CYz** %yJr'{$;;@@BB !,  Jr(rc|dj}|dk|dkz}d|iS)z Validate the name column of a BED dataframe. Name must be 1 to 255 non-tab characters. Multiple data lines may share the same name. If all features have uninformative names, dot (.) may be used as a name on every data line. rr/r0zname.is_len_ok)rr3r2)r+r5r6s r& check_namerIsJjn  ""GQ,7c>27799I ) r(rc`|ddk|ddkz}d|iS)aC Validate the score column of a BED dataframe. Integer between 0 and 1000, inclusive. When all features have uninformative scores, 0 should be used as the score on every data line. Note: Using "." is illegal in the spec, but is used in practice. 0 is the the uninformative score used in the spec. rrizscore.is_in_ranger2)r+ is_in_ranges r& check_scorerMs=wK1$G)<=BBDDK [ r(rcn|djd}d|iS)z Validate the strand column of a BED dataframe. Strand must be one of +, -, . (no strand), or ? (unknown strand). When parsing files that are not BED6+, strand should be treated as ".". rz ^[+\-.?]$zstrand.is_pattern_ok)rr1r2)r+ is_pattern_oks r& check_strandrPs;xL$**<88<<>>M   r(rc|d|dk}|d|dk}||dS)z Validate the thickStart column of a BED dataframe. Must be an integer between start and end, inclusive. When all features have uninformative thickStarts, the value of start should be used. rr r )zthickStart.is_ge_startzthickStart.is_le_endrKr+ is_ge_start is_le_ends r&check_thickStartrUsYl#r'{27799KL!RY.3355I#. )  r(rc|d|dk}|d|dk}||dS)z Validate the thickEnd column of a BED dataframe. Must be an integer greater than or equal to start and less than or equal to end, inclusive. When all features have uninformative thickEnds, the value of end should be used. rr r )zthickEnd.is_ge_startzthickEnd.is_le_endrKrRs r&check_thickEndrW"sWj>R[05577KJ2e9,1133I!,'  r(rc|dtjd|dtdkz}|dtjdd}||dS)a9 Validate the itemRgb column of a BED dataframe. A triple of 3 integers separated by commas. Each integer is between 0 and 255, inclusive. To make a feature black, itemRgb may be a single 0, as a shorthand for 0,0,0. When all features have uninformative itemRgb values, 0 should be used. rz^(\d{1,3},){2}\d{1,3}$r,c4td|DS)NcZg|](}t|dkot|dk)S)rr0int.0is r& z3check_itemRgb....Ks1GGGc!ffk;c!ffmGGGr(rKxs r&zcheck_itemRgb..KsGGQGGGHHr()zitemRgb.is_pattern_okzitemRgb.is_in_range)astyperr1r2splitapply)r+rOrLs r& check_itemRgbrh6s 9 S!!%++,EFF i=   $ $ + - cee 9 S!! UU3ZZ HH I I cee "/*  r(rcF|ddk}d|iS)z Validate the blockCount column of a BED dataframe. Must be an integer greater than 0. Note: mandatory in BED12+ files. rrzblockCount.is_gt_0rK)r+is_gt_0s r&check_blockCountrkTs0,!#((**G g r(rc"|djd}|djdjddz}||dk}||dS)a/ Validate the blockSizes column of a BED dataframe. Comma-separated list of length blockCount containing the size of each block. There must be no spaces before or after commas. There may be a trailing comma after the last element of the list. Note: mandatory in BED12+ files. r^(\d+,)*\d+(,)?$rYr/r)zblockSizes.is_pattern_okzblockSizes.is_n_blocks_ok)rr1r2rstripcount)r+rOn_blocksis_n_blocks_oks r&check_blockSizesrres|$(../BCCGGIIM,#**3//399#>>BH"\"227799N%2%3  r(rcz|djd}|djdjdd}|djdjdd}t j|||d|dgd }|t|d k}|d d }|d d }|dd }|d} d} t|j |j D]I\} } tt| d z D]$} | | | | z| | d zkrd} n%J|||||| | dS)ap Validate the blockStarts column of a BED dataframe. Comma-separated list of length blockCount containing each block's start position, relative to start. There must not be spaces before or after the commas. There may be a trailing comma after the last element of the list. Each element in blockStarts is paired with the corresponding element in blockSizes. Each blockStarts element must be an integer between 0 and end - start, inclusive. Each block must be contained within the feature. That means that for each couple i of (blockStart, blockSize), the quantity start + blockStart + blockSize must be less or equal to end. The first block must start at start and the last block must end at end. The blockStarts must be sorted in ascending order. The blocks must not overlap. Note: mandatory in BED12+ files. rrmrYcd|DS)Nc,g|]}t|Sr)r\r^s r&raz7check_blockStarts....,,,Q#a&&,,,r(r)rbs r&rdz#check_blockStarts..,,!,,,r(rcd|DS)Nc,g|]}t|Sr)r\r^s r&raz7check_blockStarts....rvr(r)rbs r&rdz#check_blockStarts..rwr(r r r/)axisrc ztfdttdDS)Ncdg|],}d|dkod|dk-S)rrr r))r_r`rcs r&raz7check_blockStarts....sP   - #q(LQ}-=a-@AeH-L   r(r)r2ranger3rbs`r&rdz#check_blockStarts..sL#    s1]#34455     r(c&|dddkS)Nrrr)rbs r&rdz#check_blockStarts..s1]#A&!+r(cb|dd|ddz|d|dz kS)Nrrr r r)rbs r&rdz#check_blockStarts..s01]#B'!L/"*==5AgJAVVr(c(|t|kSr$)sortedrbs r&rdz#check_blockStarts..sQ&))^r(TF)zblockStarts.is_pattern_okzblockStarts.is_n_blocks_okzblockStarts.is_in_rangez blockStarts.is_first_block_startzblockStarts.is_last_block_endzblockStarts.is_sortedzblockStarts.is_no_overlap) rr1r2rnrfrgr=concatr3zipvaluesr})r+rO block_starts block_sizes bs_start_endrqrLis_first_block_startis_last_block_end is_sorted is_no_overlaprow_block_startsrow_block_sizesr`s r&check_blockStartsr~sm6}%)//0CDDHHJJM = VVC[[ UU3ZZ ,, - -  < VVC[[ UU3ZZ ,, - -  9 {BwKE;!L #((--L1AAFFHHN$$  % cee(-- + +1. cee %** V V + cee ""#;#;<<@@BBIM-0[/..)/s+,,q011  A"_Q%77:J1q5:QQQ % R &3&4#.,@):!*%2  r(Ffields list[str] strict_scorebool#tuple[set[str], set[str], set[str]]ct}|D]E}||jj}|r |dkrd}n t|}||vr||Fi}|D]>}||vr8|dkr|s||||<?tj|}t|j |j } i} |D]e}||vr_|dkr|s|dvr*| t|||=| t||ftj| } t| j | j } || | fS)a Validate the fields of a BED dataframe. Parameters ---------- df : pd.DataFrame BED dataframe to validate. fields : list of str List of fields to validate. chromsizes : dict or Series, optional [default: None] Assembly/chromsizes to validate against. strict_score : bool, optional [default: False] Whether to strictly enforce the score field. Returns ------- Sets containing: (1) names of fields having an invalid dtype, (2) names of fields containing at least one null value, (3) properties that failed validation. Notes ----- The BED spec is overly strict. For example, many BED files in the wild will use "." as the uninformative score value for all features, but the spec requires that the score be an integer between 0-1000. We provide some lenience by allowing floats as well, which many tools use in practice. The spec defines the uninformative score value as 0. rr)r r ) setdtypekindBED_FIELD_KINDSaddnotnullr2r=r>locindexupdater%) r+rr8r dtype_failedrr allowed_kindsrnotnull_failedprops prop_faileds r&validate_bed_fieldsrsD55L""#w}!  1C7NN MM+C0M } $ $   S ! ! !G33 l " "g~~l~c7??,,0022GCLi  GgX.455N E<< l " "g~~l~&&& 1#6r:FFGGGG 1#6r::;;; Ie  Eei'-..K  44r(ctjdtj|djdd|djddkdzf}|d|}t |t tj|k}d}|ddD]\}}|d }|d }tj||f} tj || |rtj || |sd}n||d S) a Validate that a BED dataframe is sorted. BED dataframes should be sorted by chrom, then by start, then by end. The scheme for sorting the chrom column doesn't matter. The only thing that matters is that all rows with the same chrom value occur consecutively. rr r/NrTF)sortr r )zsorted.is_chrom_consecutivezsorted.is_sorted_start_end) npr_ flatnonzerorto_numpyr3uniquegroupbylexsort array_equal) r+ run_starts run_valuesis_chrom_consecutiveis_sorted_start_endrgroupstartsendsindicess r&check_is_sortedr sL 2>"W+,QRR0BwK4Fss4KK L Lq PPJG%%'' 3Jz??c")J2G2G.H.HHJJwUJ33  5w((**U|$$&&*dF^,, N6'?F 3 3 tG}d33 #(  E  (<&9  r(tuple[int, bool]cdD]}t|dz |jvr|}ntdt|j|k}||fS)N) r/zCould not infer a BED schema.)BED_FIELD_NAMEScolumns ValueErrorr3)r+r`nextendeds r&infer_bed_schemarCsc &:: 1q5 !RZ / /A E 089992:"H h;r(schemac$d}tj||}|std||dr"t |dnd}|ddu}||fS)Nz#^bed(3|4|5|6|7|8|9|12)?(\+(\d+)?)?$zInvalid BED schema name: r/r)rer1lowerrrr])rpatternr1rrs r&parse_bed_schemarNs4G HWfllnn - -E ?=V==>>>${{1~~4EKKNN1A{{1~~T)H h;r(inferTnan)rvalidate_fieldsrequire_sortedr8r replace_nana_reppathstr | pathlib.Path | Nonerrrr str | Nonec|dkrt|\} } nt|\} } d|jvsd|jvs d|jvrtd| dkr*d|jvsd|jvs d |jvrtd td | } t t | t |jz } t t | t | z } | r1t t |jt | z ng}t }|rXt|| || \}}}|rtd |d|r|std|d|rtd|d|rVtj t|}|j | }|rtd|dtj|j }| D]}|| vr5|dkr |d||<|dkr |d||<*t|||<;||vrPt!jd|dt|d||t|||<||||<|jD]}||vr ||||<||d|ddS)aWrite a DataFrame to a BED file. Parameters ---------- df : pd.DataFrame DataFrame to write. path : str or Path, optional Path to write the BED file to. If ``None``, the serialized BED file is returned as a string. schema : str, optional [default: "infer"] BED schema to use. If ``"infer"``, the schema is inferred from the DataFrame's columns. validate_fields : bool, optional [default: True] Whether to validate the fields of the BED file. require_sorted : bool, optional [default: False] Whether to require the BED file to be sorted. chromsizes : dict or pd.Series, optional Chromosome sizes to validate against. strict_score : bool, optional [default: False] Whether to strictly enforce validation of the score field (0-1000). replace_na : bool, optional [default: True] Whether to replace null values of standard BED fields with compliant uninformative values. na_rep : str, optional [default: "nan"] String representation of null values if written. Returns ------- str or None: The serialized BED file as a string if ``path`` is ``None``, otherwise ``None``. rr r r z=BED dataframe must have at least 3 fields: chrom, start, end.rrrrzYInformative blockCount, blockSizes, and blockStarts fields are mandatory in BED12+ files.N)r8rzFields contain invalid dtypes: rzFields contain null values: z#Properties that failed validation: z!DataFrame isn't properly sorted: )rrrzStandard column zK contains null values. These will be replaced with the uninformative value  F)seprrheader)rrrrrlistrr TypeErrorr=r>rrtolist DataFrameBED_FIELD_FILLVALUESwarningswarnfillnato_csv)r+rrrrr8rrrrr standard_cols fill_cols data_cols custom_colsfields_with_nulls dtypes_failed props_failedrbedrs r&rrXstX&r** 88&v.. 8 rz!! "* $ $  " " K    BwwBJ&& rz ) )  * * )   $BQB'MS''#bj//9::IS''#i..899I@HP$s2:]););;<<rs00b#"""""  *RXbi $ x Z                 7* 7<@@ 5<@!!!!!H 6" 7$ 8     <$ :& 9: <      <0 =ZZZZ@+/ A5A5A5A5A5H    F'+rP  *.rPrPrPrPrPrPrPrPr(