cdZddlmZddlZddlmZmZmZm Z m Z m Z mZdZeZdZdZd ZGd d eZGd d eZefdZedfdZ ddefdZ dS)aAThe module implements compression/decompression with snappy using Hadoop snappy format: https://github.com/kubo/snzip#hadoop-snappy-format Expected usage like: import snappy src = 'uncompressed' dst = 'compressed' dst2 = 'decompressed' with open(src, 'rb') as fin, open(dst, 'wb') as fout: snappy.hadoop_stream_compress(src, dst) with open(dst, 'rb') as fin, open(dst2, 'wb') as fout: snappy.hadoop_stream_decompress(fin, fout) with open(src, 'rb') as fin1, open(dst2, 'rb') as fin2: assert fin1.read() == fin2.read() )absolute_importN) _compress _uncompressstream_compressstream_decompress check_formatUncompressError _CHUNK_MAXic0tjd|}|S)N>I)structpack)numbig_endian_uints 4lib/python3.11/site-packages/snappy/hadoop_snappy.pypack_intr)sk$,,O c8tjd|dS)Nrr)runpack)datas r unpack_intr.s =t $ $Q ''rc2eZdZdZdZdZdZddZdZdS) StreamCompressoraiThis class implements the compressor-side of the hadoop snappy format, taken from https://github.com/kubo/snzip#hadoop-snappy-format Keep in mind that this compressor object does no buffering for you to appropriately size chunks. Every call to StreamCompressor.compress results in a unique call to the underlying snappy compression method. cdSNselfs r__init__zStreamCompressor.__init__<s rc<g}t|}|t|t|}t|}|t|||d|S)aMAdd a chunk containing 'data', returning a string that is compressed. This data should be concatenated to the tail end of an existing Snappy stream. In the absence of any internal buffering, no data is left in any internal buffers, and so unlike zlib.compress, this method returns everything. r)lenappendrrjoin)r routuncompressed_lengthcompressed_chunkcompressed_lengths r add_chunkzStreamCompressor.add_chunk?s!$ii 8/00111$T?? 011 8-../// #$$$xx}}rc,||S)zjThis method is simply an alias for compatibility with zlib compressobj's compress method. )r*)r rs rcompresszStreamCompressor.compressOs~~d###rNcdS)zeThis method does nothing and only exists for compatibility with the zlib compressobj Nr)r modes rflushzStreamCompressor.flushUs  rctS)zHThis method exists for compatibility with the zlib compressobj. )rrs rcopyzStreamCompressor.copy[s !!!rr) __name__ __module__ __qualname____doc__r!r*r,r/r1rrrrr2sn    $$$     """""rrcHeZdZdZgdZdZedZdZdZ dZ dS) StreamDecompressorawThis class implements the decompressor-side of the hadoop snappy format. This class matches a subset of the interface found for the zlib module's decompression objects (see zlib.decompressobj). Specifically, it currently implements the decompress method without the max_length option, the flush method without the length option, and the copy method. _buf _block_length_uncompressed_lengthc0d|_d|_d|_dS)Nrrr8rs rr!zStreamDecompressor.__init__ns $%!!!rc^t}t||dzkrtddS)aJust checks that first two integers (big endian four-bytes int) in the given data block comply to: first int >= second int. This is a simple assumption that we have in the data a start of a block for hadoop snappy format. It should contain uncompressed block length as the first integer, and compressed subblock length as the second integer. Raises UncompressError if the condition is not fulfilled. :return: None zToo short data lengthN) _INT_SIZEr#r )rint_sizes rr zStreamDecompressor.check_formatus6 t99x!| # ;!"9:: : rc&t}|xj|z c_g} t|j|krd|Sd}|jsbt |jd||_|j|d|_t|j|krd|St |j|||z}||z }t|j||zkrd|S|j|||z}|j||zd|_t |}|xjt|z c_|||j|jkrd|_d|_wx)abDecompress 'data', returning a string containing the uncompressed data corresponding to at least part of the data in string. This data should be concatenated to the output produced by any preceding calls to the decompress() method. Some of the input data may be preserved in internal buffers for later processing. TrrN) r?r9r#r%r:rrr;r$)r rr@ uncompressed next_startr)chunkuncompressed_chunks r decompresszStreamDecompressor.decompresss T   49~~( .xx ---J% 2%/ )8)0D%E%E" Ihii0 ty>>H,288L111 * *Z(%::;!!  ( "J49~~ 1J >> .xx ---I:(999E */@"@"A"ABDI!,U!3!3   % %-?)@)@ @ % %    2 3 3 3(D,>> ,-)%&"5 rc:|jdkrtddS)aAll pending input is processed, and a string containing the remaining uncompressed output is returned. After calling flush(), the decompress() method cannot be called again; the only realistic action is to delete the object. rzchunk truncated)r9r rs rr/zStreamDecompressor.flushs' 9  5!"344 4srcjt}|j|_|j|_|j|_|S)zReturns a copy of the decompression object. This can be used to save the state of the decompressor midway through the data stream in order to speed up random seeks into the stream at a future point. )r7r9r:r;)r r1s rr1zStreamDecompressor.copys3 "##I !/$($=! rN) r2r3r4r5 __slots__r! staticmethodr rFr/r1rrrr7r7asBAAI&&&\&$$$L     rr7c2t|||tS)N) blocksizecompressor_cls)_stream_compressr)srcdstrLs rrrs#  SI6F   rc4t|||t|S)N)rLdecompressor_cls start_chunk)_stream_decompressr7)rOrPrLrSs rrrs&  SI+   rc2t|||tS)N)finrDrLrR) _check_formatr7)rVrDrLs rr r s#  u +   r)r5 __future__rrsnappyrrrrNrrTr rWr r SNAPPY_BUFFER_SIZE_DEFAULT_STREAM_TO_STREAM_BLOCK_SIZEr?rrobjectrr7rrrr]st*'&&&&& ()  (((,",",",","v,",","^aaaaaaaaH)C +G"&1Mr