o Df0S @sdZgdZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl mZmZddl mZmZmZddlmZmZmZmZddlmZzddlZWn eygdZYnwz dd lmZmZWn eydZdZYnwzddl Z e!e d sej"d kre#e d d Wn eydZ Ynwe $d Z%e%&re'e%(Z)ndZ)zddlm*Z*Wn eyddZ*Ynwde'fddZ+de'ddfddZ,de-de.fddZ/GdddeZ0Gddde0Z1Gddde0Z2Gd d!d!e2Z3Gd"d#d#e1Z4Gd$d%d%e2Z5Gd&d'd'e1Z6d(e-defd)d*Z7d(e-defd+d,Z8d(e-defd-d.Z9d/d0Z:d(e-fd1d2Z;d3e-dee-fd4d5ZdS)>z& Open compressed files transparently. )xopenPipedGzipWriterPipedGzipReader __version__N)ABCabstractmethod)PopenPIPEDEVNULL)OptionalTextIOAnyStrIO)version)igzip isal_zlib F_SETPIPE_SZlinuxiz/proc/sys/fs/pipe-max-size)fspathcCsDt|dr |Stdurt|tjrt|St|ts td|S)N __fspath__zpath must be a string)hasattrrpathlib isinstancePathstr TypeError)pathrW/var/www/html/software/conda/envs/catlas/lib/python3.10/site-packages/xopen/__init__.pyr8s  rreturnc CszttdWStyYnwddl}z:td }|}Wdn1s*wY|d|}|rOtt | d ddd d }|dkrO|WSWn t yYYnwz ddl}|WSttfyoYdSw) z Number of available virtual or physical CPUs on this system Adapted from http://stackoverflow.com/a/1006301/715090 rNz/proc/self/statusz(?m)^Cpus_allowed:\s*(.*)$r,1)lenossched_getaffinityAttributeErrorreopenreadsearchbinintgroupreplacecountOSErrormultiprocessing cpu_count ImportErrorNotImplementedError)r)fstatusmresr3rrr_available_cpu_countDs2    "  r;fdcCs>ttdrts dSz t|tjtWdStyYdSw)z Set pipe size to maximum on platforms that support it. :param fd: The file descriptor to increase the pipe size for. rN)rfcntl_MAX_PIPE_SIZErr2)r<rrr_set_pipe_size_to_max_s r?programc Cstjddd\}}zWt|d}|tdtdWdn1s'wYztj|dd |gd ttd }|j d kWWt |t |Stj y`YWt |t |d Swt |t |w)zt Check if a concatenated gzip file can be read properly. Not all deflate programs handle this properly. .gzzxopen.)suffixprefixwbsABsCDNz-cz-dT)checkstderrstdoutsABCDF)tempfilemkstempr*writegzipcompress subprocessrunr rGr&closeremoveCalledProcessError)r@r<Z temp_path temp_fileresultrrr_can_read_concatenated_gzls&       rTc@s4eZdZdZddZddZddZedd Zd S) Closingzl Inherit from this class and implement a close() method to offer context manager functionality. cC|SNrselfrrr __enter__zClosing.__enter__cGs |dSrW)rO)rYexc_inforrr__exit__ zClosing.__exit__cCs$z|WdStyYdSwrW)rO ExceptionrXrrr__del__s  zClosing.__del__cCdSrWrrXrrrrOsz Closing.closeN) __name__ __module__ __qualname____doc__rZr]r`rrOrrrrrUsrUc @seZdZdZ    ddedeededeefdd Zd d Zd edeeded e de f ddZ de ddfddZ dddZddZddZdS)PipedCompressionWriterzS Write Compressed files by running an external process and piping into it. wtNr@ compresslevel threads_flagthreadscCs|dvr td|t|||_d|_||_||_||_||_|dur*t t d}||_ z | ||||j|_ Wn tyF|jw|j jdusOJt|j jd|vret|j j|_dS|j j|_dS)a) mode -- one of 'w', 'wt', 'wb', 'a', 'at', 'ab' compresslevel -- compression level threads_flag -- which flag is used to denote the number of threads in the program. If set to none, program will be called without threads flag. threads (int) -- number of threads. If this is set to None, a reasonable default is used. At the moment, this means that the number of available CPU cores is used, capped at four to avoid creating too many threads. Use 0 to use all available cores. )wrgrDaatabz?Mode is '{}', but it must be 'w', 'wt', 'wb', 'a', 'at' or 'ab'FNb) ValueErrorformatr*outfileclosedname_mode_program _threads_flagminr;_threads _open_processprocessr2rOstdinr?filenoio TextIOWrapper_file)rYrr@moderhrirjrrr__init__s4      zPipedCompressionWriter.__init__cCd|jj|j|j|j|jSNz-{}('{}', mode='{}', program='{}', threads={})rr __class__rbrurvrwrzrXrrr__repr__zPipedCompressionWriter.__repr__rrsr c Cs|jg}|dkr|jdur||jt|g7}g}d|vr)|dur)|dt|g7}tt|td}tjdkr9d|d<t||fi|}|S)Nrrk-)r}rGrFwin32T close_fds) rwrxrdictr r sysplatformr) rYrrhrjrs program_args extra_argskwargsr|rrrr{s z$PipedCompressionWriter._open_processargcCs|j|dSrW)rrJ)rYrrrrrJszPipedCompressionWriter.writecCsL|jrdSd|_|j|j}|j|dkr$td|j|dS)NTrz.Output {} process terminated with exit code {}) rtrrOr|waitrsr2rrrw)rYretcoderrrrOs    zPipedCompressionWriter.closecCrVrWrrXrrr__iter__szPipedCompressionWriter.__iter__cCs td)Nz not readable)rUnsupportedOperationrXrrr__next__ zPipedCompressionWriter.__next__)rgNNNr N)rbrcrdrerr r.rrr rr{r rJrOrrrrrrrfs: *    rfc @seZdZdZ   d(dededeedeefdd Zd d Zd)d dZ ddZ d e fddZ d*de d dfddZd e fddZddZd e fddZd e fddZd+defd d!Zd e fd"d#Zd e fd$d%Zd)d&d'ZdS),PipedCompressionReaderzA Open a pipe to a process for reading a compressed file. rNr@rrirjcCs|dvr td|||_|d|g}|dur%|durd}||t|g7}||_t|ttd|_||_|jj dus;Jt |jj ||_ d|vrSt |jj |_n|jj |_|jjdus`Jt |jj|_d|_td |dS) z@ Raise an OSError when pigz could not be found. )rrtrbz.Mode is '{}', but it must be 'r', 'rt' or 'rb'z-cdNr)rGrFrpFg{Gz?)rqrrrwrrzrr r|rurGr?r~rvrrrrF_stderrrttimesleep_raise_if_error)rYrr@rrirjrrrrrs,     zPipedCompressionReader.__init__cCrrrrXrrrr1rzPipedCompressionReader.__repr__r cCsd|jrdSd|_|j}|dur|jd}nd}|j|j|j|d|jdS)NTF) allow_sigterm) rtr|poll terminaterrrOrr)rYrrrrrrO:s     zPipedCompressionReader.closecCrVrWrrXrrrrJr[zPipedCompressionReader.__iter__cC |jSrW)rrrXrrrrMrzPipedCompressionReader.__next__FrcCsh|j}|dur.|dkr0|r|tj ks2|j}|j|jt d ||dSdSdS)z Raise IOError if process is not running anymore and the exit code is nonzero. If allow_sigterm is set and a SIGTERM exit code is encountered, no error is raised. Nrz{} (exit code {})) r|rsignalSIGTERMrr+striprrOr2rr)rYrrmessagerrrrPs    z&PipedCompressionReader._raise_if_errorcG |jj|SrW)rr+rYargsrrrr+`r^zPipedCompressionReader.readcGrrW)rreadintorrrrrcr^zPipedCompressionReader.readintocGrrW)rreadlinerrrrrfr^zPipedCompressionReader.readlinecCrrW)rseekablerXrrrrirzPipedCompressionReader.seekablencCs |j|SrW)rpeek)rYrrrrrlr^zPipedCompressionReader.peekcCrrW)rreadablerXrrrrorzPipedCompressionReader.readablecCrrW)rwritablerXrrrrrrzPipedCompressionReader.writablecCrarWrrXrrrflushur[zPipedCompressionReader.flushrNNr)FrW)rbrcrdrerr r.rrrOrr rboolrr+rrrrrrrrrrrrs6 . rc0eZdZdZddedeeffdd ZZS) rz Open a pipe to pigz for reading a gzipped file. Even though pigz is mostly used to speed up writing by using many compression threads, it is also faster when reading, even when forced to use a single thread (ca. 2x speedup). rNrrjc sDz t|d|d|WdSty!t|d|d|YdSw)Npigz-prK)superrr2)rYrrrjrrrrs  zPipedGzipReader.__init__)rN rbrcrdrerr r.r __classcell__rrrrrys$rcs>eZdZdZ   d dedeedeeffdd ZZS) ra Write gzip-compressed files by running an external gzip or pigz process and piping into it. pigz is tried first. It is fast because it can compress using multiple cores. Also it is more efficient on one core. If pigz is not available, a gzip subprocess is used. On Python 3, gzip.GzipFile is on par with gzip itself, but running an external gzip can still reduce wall-clock time because the compression happens in a separate process. rgNrrhrjc sf|dur|tddvrtdzt|d||d|WdSty2t|d||d|YdSw)a mode -- one of 'w', 'wt', 'wb', 'a', 'at', 'ab' compresslevel -- compression level threads (int) -- number of pigz threads. If this is set to None, a reasonable default is used. At the moment, this means that the number of available CPU cores is used, capped at four to avoid creating too many threads. Use 0 to let pigz use all available cores. Nr z%compresslevel must be between 1 and 9rrrK)rangerqrrr2)rYrrrhrjrrrrs zPipedGzipWriter.__init__)rgNNrrrrrrs rcs(eZdZdZddeffdd ZZS)PipedIGzipReadera Uses igzip for reading of a gzipped file. This is much faster than either gzip or pigz which were written to run on a wide array of systems. igzip can only run on x86 and ARM architectures, but is able to use more architecture-specific optimizations as a result. rrcs$tdstdt|d|dS)NrzThis version of igzip does not support reading concatenated gzip files and is therefore not safe to use. See: https://github.com/intel/isa-l/issues/143)rTrqrr)rYrrrrrrs zPipedIGzipReader.__init__)r)rbrcrdrerrrrrrrrsrcr) PipedIGzipWritera Uses igzip for writing a gzipped file. This is much faster than either gzip or pigz which were written to run on a wide array of systems. igzip can only run on x86 and ARM architectures, but is able to use more architecture-specific optimizations as a result. Threads are supported by a flag, but do not add any speed. Also on some distro version (isal package in debian buster) the thread flag is not present. For these reason threads are omitted from the interface. Only compresslevel 0-3 are supported and these output slightly different filesizes from their pigz/gzip counterparts. See: https://gist.github.com/rhpvorderman/4f1201c3f39518ff28dde45409eb696b rgNrrhcs4|dur|tddvrtdt|d||dS)Nrroz%compresslevel must be between 0 and 3r)rrqrr)rYrrrhrrrrszPipedIGzipWriter.__init__)rgNrrrrrrs$ rrcCs*ttjtjd|d}t||ddS)N)rrkrF)rclosefd)rrr}rGr*r~)rZstdrrr_open_stdin_or_outsrcCs t||SrW)bz2r*filenamerrrr _open_bz2r^rcCstdurtdt||S)NzPCannot open xz files: The lzma module is not available (use Python 3.3 or newer))lzmar5r*rrrr_open_xzs  rc Cspd|vrzt||WSttfyt|||dYSwzt|||WSttfy7t||||dYSw)Nr)rj)rr2rqrrrrrrhrjrrr_open_gz_externals  rcCs|dkrzt||||WStyYnwd|vr*tdur$t||St||StdurKztj|||dur=tjdWS|dWStyJYnwtj|||durWddS|dS)Nrr)rh)rr2rr*rKrZISAL_DEFAULT_COMPRESSIONrqrrrr_open_gzs4     rrcCszJtt|jrEt|d }|d}Wdn1swY|dddkr/WdS|dddkr:Wd S|ddd krHWd SWdSWdStyTYdSw) z Attempts to detect file format from the content by reading the first 6 bytes. Returns None if no format could be detected. rrNsgzsBZhrs7zXZxz)statS_ISREGr&st_moder*r+r2)rfhbsrrr_detect_format_from_content s&   rcCs.|drdS|drdS|drdSdS)zv Attempts to detect file format from the filename extension. Returns None if no format could be detected. z.bz2rz.xzrrArN)endswith)rrrr_detect_format_from_extension"s   rrrhrjcCs|dvr|d7}|dvrtd|t|}|dkrt|St|}|dur/d|vr/t|}|dkr:t||||S|d krCt||S|d krLt||St ||S) a A replacement for the "open" function that can also read and write compressed files transparently. The supported compression formats are gzip, bzip2 and xz. If the filename is '-', standard output (mode 'w') or standard input (mode 'r') is returned. The file type is determined based on the filename: .gz is gzip, .bz2 is bzip2, .xz is xz/lzma and no compression assumed otherwise. mode can be: 'rt', 'rb', 'at', 'ab', 'wt', or 'wb'. Also, the 't' can be omitted, so instead of 'rt', 'wt' and 'at', the abbreviations 'r', 'w' and 'a' can be used. In Python 2, the 't' and 'b' characters are ignored. Append mode ('a', 'at', 'ab') is not available with BZ2 compression and will raise an error. compresslevel is the compression level for writing to gzip files. This parameter is ignored for the other compression formats. If set to None (default), level 6 is used. threads only has a meaning when reading or writing gzip files. When threads is None (the default), reading or writing a gzip file is done with a pigz (parallel gzip) subprocess if possible. See PipedGzipWriter and PipedGzipReader. When threads = 0, no subprocess is used. )rrkrlt)rrrgrDrmrnzMode '{}' not supportedrNrkrrr) rqrrrrrrrrrr*)rrrhrjZdetected_formatrrrr1s""   rr)?re__all__rKrrr&rrrrrrMrHabcrrrr r typingr r r r_versionrrrr5Zisalrrr=rrsetattrrZ_MAX_PIPE_SIZE_PATHexistsr. read_textr>rr;r?rrrTrUrfrrrrrrrrrrrrrrrrrs           d{