DUfd<$ddlZddlZddlmZddlmZmZddlZddlZ ddl m Z m Z m Z mZmZmZmZmZ ddlZdZn #e$rdZYnwxYwer e d dejd ed ede jfd ed ejf d Ze d de jd ed ede jfd ed e jf dZe d deeje jfd ed ede jfd ed eeje jff dZeej d dejd ed ede jfd ed ejf dZ d!dedededed ed df dZn1e d de jd ed ede jfd ed e jf dZee j d de jd ed ede jfd ed e jf dZej dddde jde jde jd e jd e jf dZ!dS)"N)singledispatch)overloadUnion)TempFileHolderglue_csvglue_hdf glue_parquet parse_csv parse_hdf parse_parquet_parallel_argsortTFdataaxistargetncpusreturncdSNrrrrs U/var/www/html/software/conda/lib/python3.11/site-packages/qnorm/quantile_normalize.pyquantile_normalizers /2ccdSrrrs rrr&s -0Crc@tdt|)a Quantile normalize your array/dataframe. It does quantile normalization in the "correct" way in the sense that it takes the mean of duplicate values instead of ignoring them. Args: data: numpy.ndarray or pandas.DataFrame to be normalized axis: axis along to normalize. Axis=1 (default) normalizes each column/sample which gives them identical distributions. Axis=0 normalizes each row/feature giving them all identical distributions. target: distribution to normalize onto ncpus: number of cpus to use for normalization Returns: a quantile normalized copy of the input. ,quantile_normalize not implemented for type NotImplementedErrortypers rrr.'0" G4:: G G   rc |}|dkr5t|jt||||dd<n4t|jt||||dd<|S)Nr)copyquantile_normalize_npvaluesastypefloat)rrrrqn_datas rquantile_normalize_pdr)Js))++ 199.%%e,,dFEGAAAJJ/%%e,,dFEGAAAJr順infileoutfile rowchunksize colchunksizec !"#|drd}t|\!}nf|drd}t|\!}}n;|drd}t|\!}}} nt dt !} t |} g} g} g}t j| }t5}ttj | |z D]}||zt j |dz|zd | }}|dkrtj|5"t "dksJ"d #!"#fd t||D}tj|d d }d d d n #1swxYwYn{|dkrQtj||dd d gt't|dz|dzd }n$|dkrtj|!||}t+|j||jj\}}~t j||d }t j|d }|||z ||z |z zz }| |dd| |dd||ddt j| d|t j| d|t j|d|~~~g}g}t j|tj t ||z D][}||ddtj||dd \||~ttj | |z D]}t j | |d}t j ||d}t j | |d}tC||||}~~~g}tEt j|tj |j#d |z D]M\}}|d|d|dd} || t j| |N~~||tHj%&|rtIj'||dkrtQ|!|n2|dkrtS|!||n|dkrtU|!||| d d d d S#1swxYwYd S)a Memory-efficient quantile normalization implementation by splitting the task into sequential subtasks, and writing the intermediate results to disk instead of keeping them in memory. This makes the memory footprint independent of the input table, however also slower.. Args: infile: path to input table. The table can be either a csv-like file of which the delimiter is auto detected. Or the infile can be a hdf file, which requires to be stored with format=table. outfile: path to the output table. Has the same layout and delimiter as the input file. If the input is csv-like, the output is csv- like. If the input is hdf, then the output is hdf. rowchunksize: how many rows to read/write at the same time when combining intermediate results. More is faster, but also uses more memory. colchunksize: how many columns to use at the same time when calculating the mean and normalizing. More is faster, but also uses more memory. ncpus: The number of cpus to use. Scales diminishingly, and more than four is generally not useful. )z.hdfz.h5hdf)z.csvz.tsvz.txtcsvz.parquetparquetzhOnly HDF ('.hdf', '.h5'), text ('.csv', '.tsv', '.txt'), and parquet ('.parquet') formats are supported.rrcHg|]}|Sr) select_column).0icolumnsr1keys r z2incremental_quantile_normalize..s=    ! --c71:>>   rrfloat32N#)sepcomment index_colusecols)r8qnorm_z.npy)prefixsuffixz.p) compressionT) allow_pickle_)+endswithr r r rlennpzerosrrangemathceilclippdHDFStorekeysconcatr&read_csvlist read_parquetrr%dtypetake_along_axismeanappend get_filenamesave array_split DataFrame to_pickleload_numba_accel_qnorm enumerateshapeospathexistsremover rr )$r,r-r.r/r dataformatindex delimiter index_usedschemanr_colsnr_rowstmp_valstmp_sorted_valstmp_idxsrtfhr7 col_startcol_endcolsdfr sorted_idx sorted_vals rankmeans qnorm_tmpindex_tmpfileschunkqnormed col_tmpfilesjtmpfiler8r1r9s$ @@@rincremental_quantile_normalizer_s|: ??? + + J&v..NGUU __5 6 6 J(1&(9(9 %GUII __j * * "J1>v1F1F .GUJ%B g,,e**'""   u N49W|%;<<==: 2: 2 $GQUl2Aw??# &&V,,G"388::!3333!hhjjm      %*9g%>%>    Yt!444;;IFFGGGGGGGGGGGGGGG 5((% #"#!" MT% A w{*K*K%L%L M  fY'' B 9,, '0A(BB $5Iubio$$ j 0 GKa888 9v-y(W5 $$HV$DD &&$$HV$DD$$HV$DD d++++[999 j111*kkI NtyUl!:;;  %%$$HT$BB U##--"2&D.   ^ , , ,49W|%;<<== / /wx{>>>WXa[tDDD  goa&8tLLL -*k6*k " )N7=+;l+J!K!K!! , ,HAu "..000A000/G!''000GGU++++U  ....w~~g&& # '"""U""'95555u$$'9i@@@@y((Wgy*fMMMku Nu Nu Nu Nu Nu Nu Nu Nu Nu Nu Nu Nu Nu Nu Nu Nu Nu Ns9A#X=B G XG XG PXXXc@tdt|)a Quantile normalize your array. It does quantile normalization in the "correct" way in the sense that it takes the mean of duplicate values instead of ignoring them. Args: data: numpy.ndarray or pandas.DataFrame to be normalized axis: axis along to normalize. Axis=1 (default) normalizes each column/sample which gives them identical distributions. Axis=0 normalizes each row/feature giving them all identical distributions. target: distribution to normalize onto ncpus: number of cpus to use for normalization Returns: a quantile normalized copy of the input. rrrs rrrr!r_datactjjtjst djdt fdtjtjfDr tj}n tj}|dkrtj n|dkrnt d|d|dkr- |}tj |d }n*|dkrt||\}}nt d tj ||d }|tj|d }nt|tjs3 tj|}n#t$$rt d wxYw|jdkrt d |jd |jd|jdkr,t d|jdd|jddtj|jtjst d|jdtj| |}t-||||}|dkr|j}|S)NzThe type of your data (z[) is is not supported, and might lead to undefined behaviour. Please use numeric data only.c3LK|]}tjj|VdSr)rK issubdtyperX)r6rXrs r z(quantile_normalize_np..>sB  .3 ek5))      rrrz`qnorm only supports 2 dimensional data, so the axishas to be either 0 or 1, but you set axis to .)rXr;z2The number of cpus needs to be a positive integer.z5The target could not be converted to a numpy.ndarray.zWThe target array should be a 1-dimensionsal vector, however you supplied a vector with z dimensionsz=The target array does not contain the same amount of values (z) as the data contains rows ()zThe type of your target ()rKrrXnumber ValueErroranyint32r<float64 transposer&argsortrrYrZ isinstancendarrayarray ExceptionndimrdsortrbT) rrrrrXrrx sorted_val final_ress ` rr$r$/s =bi 0 0  -ek - - -       8:"*7M        qyy U##          zz||%|((Z1--- ,UE5AAjjMNNN#D*1===J ~!,,,&"*--  &))    N  ;!  G.4kGGG  <?djm + +%LO%%JqM%%%  }V\2955 1DJ111  U3344"4ZHHI qyyK s &E;;F)nopythonfastmathcacheqnormrxrc|jd}|jd}t|D]}d}||krd}d} ||z|krN|||f|||z|fkr5| |||zz } |dz }||z|kr|||f|||z|fk5|dkr+| |z} t|D]} ||| z|f} | || |f<||z }||k|S)z9 numba accelerated "actual" qnorm normalization. rrg)rdrM) rrxrrn_rowsn_colscol_ir7nvalridxs rrbrbs,[^F [^Fv &jjAC Aq%x(Jq1ue|,DDDva!e}$Q Aq%x(Jq1ue|,DDD 1uuqq,,A$QUE\2C(+E#u*%% FA)&jj, Lr)rNr)r*r+r)"rerN functoolsrtypingrrnumbanumpyrKutilrrr r r r r rpandasrQ pandas_importModuleNotFoundErrorr_intrrregisterr)strrr$jitrbrrrrs $$$$$$""""""""                     MMMMMP '(=A()222!$2#(rz)9#:2#&2!# 222X2 '(=A()000!$0#(rz)9#:0#&0!# 000X0*.   BL"*,-  dBJ&'   r|RZ' (    ^ 6  ..*. ldBJ&'   /..$ nNnNnNnNnN nN  nN  nNnNnNnNnNf*.   j  dBJ&'       ^ 8RZ((&* RR :R R $ " #R  R Z RRR)(Rj D4t444' :' ' ' J ' Z '''54'''s;AA