o Nrfp{ @sUddlmZddlZddlZddlZddlZddlZddlZddlZddl m Z m Z ddl m Z mZmZmZmZmZmZmZmZe rRddlZddlmZedZnedZedZedZeeegefZGd d d Z Gd d d e Z!Gd dde Z"Gddde Z#Gddde Z$Gddde Z%Gddde Z&Gddde Z'GdddeeefZ(Gddde Z)de iZ*de+d<d(d)d&d'Z,e e!e%e"e$e#e&e'e)f D]Z-e,e-qdS)*) annotationsN)FutureThreadPoolExecutor) TYPE_CHECKINGAnyCallableClassVarGeneric NamedTupleOptional OrderedDictTypeVar) ParamSpecPTZfsspecc@sPeZdZUdZdZded<dd d ZdddZdddZdddZ dddZ dS) BaseCacheagPass-though cache: doesn't keep anything, calls every time Acts as base class for other cachers Parameters ---------- blocksize: int How far to read ahead in numbers of bytes fetcher: func Function of the form f(start, end) which gets bytes from remote as specified size: int How big this file is none ClassVar[str]name blocksizeintfetcherFetchersizereturnNonecCs.||_d|_||_||_d|_d|_d|_dSNr)rnblocksrr hit_count miss_counttotal_requested_bytesselfrrrr#W/var/www/html/software/conda/envs/catlas/lib/python3.10/site-packages/fsspec/caching.py__init__:s zBaseCache.__init__start int | NonestopbytescCs<|durd}|dur |j}||jks||krdS|||S)Nr)rrr"r&r(r#r#r$_fetchDs zBaseCache._fetchcCsd|_d|_d|_dS)zAReset hit and miss counts for a more ganular report e.g. by file.rN)rrr r"r#r#r$ _reset_statsMs zBaseCache._reset_statsstrcCs0|jdkr |jdkr dSd|j|j|j|jfS)z2Return a formatted string of the cache statistics.rz3 , %s: %d hits, %d misses, %d total requested bytes)rrrr r-r#r#r$ _log_statsSszBaseCache._log_statscCs@d|jjd|jd|jd|jd|jd|jd|jdS) Nz ) __class____name__rrrrrr r-r#r#r$__repr___szBaseCache.__repr__Nrrrrrrrrr&r'r(r'rr)rr)rr/) r3 __module__ __qualname____doc__r__annotations__r%r,r.r1r4r#r#r#r$r(s     rcsTeZdZdZdZ  d d!fdd Zd"ddZd#ddZd$ddZd%ddZ Z S)& MMapCachezmemory-mapped sparse file cache Opens temporary file, which is filled blocks-wise when data is requested. Ensure there is enough disc space in the temporary location. This cache method might only work on posix mmapNrrrrrlocation str | Noneblocksset[int] | Nonerrcs8t||||durtn||_||_||_dSN)superr%setr@r> _makefilecache)r"rrrr>r@r2r#r$r%wszMMapCache.__init__mmap.mmap | bytearraycCsddl}ddl}|jdkrtS|jdustj|jsB|jdur*|}t |_ nt |jd}| |jd| d|nt |jd}|||jS)Nrzwb+1zr+b)r=tempfiler bytearrayr>ospathexists TemporaryFilerDr@openseekwriteflushfileno)r"r=rKfdr#r#r$rEs       zMMapCache._makefiler&r'endr)c sDtd|d||durd}|durj}|jks!||kr#dS|j}|j}fddt||dD}fddt||dD}jt|7_jt|7_|r|d}|j}t |jj} j | |7_ td |d |d| d  || j || <j ||s]j ||S) NzMMap cache fetching -rr*csg|] }|jvr|qSr#r@.0ir-r#r$ z$MMapCache._fetch..rIcsg|] }|jvr|qSr#rYrZr-r#r$r]r^zMMap get block #z ())loggerdebugrrrangerlenrpopminr rrFr@add) r"r&rWZ start_blockZ end_blockZneedhitsr\Zsstartsendr#r-r$r,s.      zMMapCache._fetchdict[str, Any]cCs|j}|d=|S)NrF)__dict__copyr"stater#r#r$ __getstate__s zMMapCache.__getstate__rmcCs|j|||_dSrB)rjupdaterErFrlr#r#r$ __setstate__s zMMapCache.__setstate__)NN) rrrrrrr>r?r@rArr)rrHr&r'rWr'rr)rrirmrirr) r3r8r9r:rr%rEr,rnrp __classcell__r#r#rGr$r<ls  r<c0eZdZdZdZdfd d ZdddZZS)ReadAheadCachea!Cache which reads only when we get beyond a block of data This is a much simpler version of BytesCache, and does not attempt to fill holes in the cache or keep fragments alive. It is best suited to many small reads in a sequential order (e.g., reading lines from a file). Z readaheadrrrrrrrcs&t|||d|_d|_d|_dS)Nr*r)rCr%rFr&rWr!rGr#r$r%s zReadAheadCache.__init__r&r'rWr)cCs>|durd}|dus||jkr|j}||jks||krdS||}||jkr?||jkr?|jd7_|j||j||jS|j|krK|jkrhnn|jd7_|j||jd}|t|8}|j}n |jd7_d}t|j||j}|j ||7_ | |||_||_|jt|j|_||jd|SNrr*rI) rr&rWrrFrrcrerr r)r"r&rWlpartr#r#r$r,s. zReadAheadCache._fetchr5rqr3r8r9r:rr%r,rtr#r#rGr$rvs rvcru)FirstChunkCachezCaches the first block of a file only This may be useful for file types where the metadata is stored in the header, but is randomly accessed. firstrrrrrrrcs&||kr|}t|||d|_dSrB)rCr%rFr!rGr#r$r%s zFirstChunkCache.__init__r&r'rWr)cCs*|pd}||jkrtddSt||j}||jkr|jdurW|jd7_||jkrG|j|7_|d|}|d|j|_||dS|d|j|_|j|j7_|j||}||jkrv|j||j7_|||j|7}|j d7_ |S|jd7_|j||7_|||S)Nrz,FirstChunkCache: requested start > file sizer*rI) rr`rarerrFrr rr)r"r&rWdataryr#r#r$r,s0          zFirstChunkCache._fetchr5rqrzr#r#rGr$r{s r{csheZdZdZdZ d#d$fd d ZddZd%ddZd&ddZd'ddZ d(fdd Z d)d!d"Z Z S)* BlockCachea Cache holding memory as a set of blocks. Requests are only ever made ``blocksize`` at a time, and are stored in an LRU cache. The least recently accessed block is discarded when more than ``maxblocks`` are stored. Parameters ---------- blocksize : int The number of bytes to store in each block. Requests are only ever made for ``blocksize``, so this should balance the overhead of making a request against the granularity of the blocks. fetcher : Callable size : int The total size of the file being cached. maxblocks : int The maximum number of blocks to cache for. The maximum memory use for this cache is then ``blocksize * maxblocks``. Z blockcache rrrrr maxblocksrrcs<t|||t|||_||_t||j|_ dSrB) rCr%mathceilrr functools lru_cache _fetch_block_fetch_block_cachedr"rrrrrGr#r$r%7szBlockCache.__init__cC |jSz The statistics on the block cache. Returns ------- NamedTuple Returned directly from the LRU Cache used internally. r cache_infor-r#r#r$r? zBlockCache.cache_inforicCs|j}|d=|S)Nrrjrlr#r#r$rnJszBlockCache.__getstate__rmcCs&|j|t|d|j|_dS)Nr)rjrorrrrrlr#r#r$rpOs   zBlockCache.__setstate__r&r'rWr)cCst|durd}|dur |j}||jks||krdS||j}||j}t||dD]}||q)|j||||dS)Nrr*rIstart_block_numberend_block_number)rrrbr _read_cache)r"r&rWrr block_numberr#r#r$r,Us    zBlockCache._fetchrcst||jkrtd|d|jd||j}||j}|j||7_|jd7_td|t||}|S)= Fetch the block of data for `block_number`. 'block_number=(' is greater than the number of blocks (r_rIzBlockCache fetching block %d) r ValueErrorrr rr`inforCr,)r"rr&rWblock_contentsrGr#r$rls    zBlockCache._fetch_blockrrc C||j}||j}|jd7_||kr ||}|||S|||dg}|t|jt|d||||d|d|Sz Read from our block cache. Parameters ---------- start, end : int The start and end byte positions. start_block_number, end_block_number : int The start and end block numbers. rINr*rrrextendmaprbappendjoin r"r&rWrr start_posend_posblockoutr#r#r$r~s     zBlockCache._read_cacher rrrrrrrrrrrrrsrq)rrrr) r&rrWrrrrrrr)) r3r8r9r:rr%rrnrpr,rrrtr#r#rGr$r~s  r~csHeZdZUdZdZded< ddfdd ZdddZdddZZ S) BytesCacheaKCache which holds data in a in-memory bytes object Implements read-ahead by the block size, for semi-random reads progressing through the file. Parameters ---------- trim: bool As we read more data, whether to discard the start of the buffer when we are more than a blocksize ahead of it. r)rrTrrrrrtrimboolrrcs,t|||d|_d|_d|_||_dS)Nr*)rCr%rFr&rWr)r"rrrrrGr#r$r%s  zBytesCache.__init__r&r'rWcCs|durd}|dur |j}||jks||krdS|jdurC||jkrC|jdurC||jkrC||j}|jd7_|j||||S|jrPt|j||j}n|}||ks[||jkr]dS|jdusg||jkr|jdusq||jkr|j||7_|jd7_| |||_||_n|jdusJ|jdusJ|jd7_||jkr|jdus|j||jkr|j||7_| |||_||_nb|j|j|7_| ||j}||_||j|_nG|jdur)||jkr)|j|jkrn4||j|jkr|j||7_| |||_||_n|j||j7_| |j|}|j||_|jt |j|_||j}|j||||}|j rk|j|j|jd}|dkrk|j|j|7_|j|j|d|_|Srw) rr&rWrrFrrer rrrcr)r"r&rWoffsetZbendnewrnumr#r#r$r,sn          zBytesCache._fetchcCs t|jSrB)rcrFr-r#r#r$__len__s zBytesCache.__len__)T) rrrrrrrrrrrq)rr) r3r8r9r:rr;r%r,rrtr#r#rGr$rs   IrcsDeZdZUdZdZded<    ddfdd ZdddZZS)AllBytesz!Cache entire contents of the fileallrrNrr'rFetcher | Nonerr} bytes | NonerrcsNt||||dur"|jd7_|j|j7_|d|j}||_dS)NrIr)rCr%rr rrr})r"rrrr}rGr#r$r%s  zAllBytes.__init__r&r(r)cCs|jd7_|j||S)NrI)rr}r+r#r#r$r,szAllBytes._fetch)NNNN) rr'rrrr'r}rrrr6 r3r8r9r:rr;r%r,rtr#r#rGr$r s  rcsDeZdZUdZdZded<  ddfdd Zdfdd ZZS)KnownPartsOfAFilea Cache holding known file parts. Parameters ---------- blocksize: int How far to read ahead in numbers of bytes fetcher: func Function of the form f(start, end) which gets bytes from remote as specified size: int How big this file is data: dict A dictionary mapping explicit `(start, stop)` file-offset tuples with known bytes. strict: bool, default True Whether to fetch reads that go beyond a known byte-range boundary. If `False`, any read that ends outside a known part will be zero padded. Note that zero padding will not be used for reads that begin outside a known byte-range. partsrrNTrrrrrr}&Optional[dict[tuple[int, int], bytes]]strictr_rc st|||||_|rdt|}|dg}||dg} |ddD]3\} } |d\} } | | krH| | f|d<| d|| | f7<q&|| | f| || | fq&tt|| |_ dSi|_ dS)NrrI) rCr%rsortedkeysrdrdictzipr})r"rrrr}rrZ old_offsetsoffsetsr@r&r(Zstart0Zstop0rGr#r$r%=s      zKnownPartsOfAFile.__init__r&r'r(rr)cs:|durd}|dur |j}d}|jD]J\\}}}||kr$|kr^nq||}|||||}|jrC||krA|krZnn|d||t|7}|jd7_|S|}nq|jdurntd||fdt d||fdt d|d ||j ||7_ |j d7_ |t||S) Nrr*rIz&Read is outside the known file parts: z. z%. IO/caching performance may be poor!z!KnownPartsOfAFile cache fetching rX)rr}itemsrrcrrrwarningswarnr`rar rrCr,)r"r&r(rZloc0loc1r}offrGr#r$r,[s2 zKnownPartsOfAFile._fetch)NT) rrrrrrr}rrrrrr6rr#r#rGr$r$s  rc@sTeZdZdZGdddeZdd d d Zd!ddZd"ddZd#ddZ d$ddZ dS)% UpdatableLRUzg Custom implementation of LRU cache that allows updating keys Used by BackgroudBlockCache c@s.eZdZUded<ded<ded<ded<dS)UpdatableLRU.CacheInforrgmissesmaxsizecurrsizeN)r3r8r9r;r#r#r#r$ CacheInfos  rfuncCallable[P, T]max_sizerrrcCs0t|_||_||_d|_d|_t|_ dSr) collectionsr _cache_func _max_size_hits_misses threadingLock_lock)r"rrr#r#r$r%s zUpdatableLRU.__init__argsP.argskwargsP.kwargsrcOs|r td||j&||jvr-|j||jd7_|j|WdSWdn1s7wY|j|i|}|j,||j|<|jd7_t|j|j krk|jj ddWd|SWd|S1svwY|S)Nz Got unexpected keyword argument rIFlast) TypeErrorrrr move_to_endrrrrcrpopitem)r"rrresultr#r#r$__call__s.     zUpdatableLRU.__call__rrcGs4|j ||jvWdS1swYdSrB)rr)r"rr#r#r$ is_key_cacheds$zUpdatableLRU.is_key_cachedrcGsd|j%||j|<t|j|jkr |jjddWddSWddS1s+wYdS)NFr)rrrcrr)r"rrr#r#r$add_keys "zUpdatableLRU.add_keyrcCsH|j|j|jt|j|j|jdWdS1swYdS)N)rrrgr)rrrrcrrrr-r#r#r$rs$zUpdatableLRU.cache_infoN)r)rrrrrr)rrrrrr)rrrr)rrrrrrrr) r3r8r9r:r rr%rrrrr#r#r#r$rs    rcsveZdZUdZdZded< d(d)fdd Zd*ddZd+ddZd,ddZ d-ddZ d.d/fd"d# Z d0d&d'Z Z S)1BackgroundBlockCachea Cache holding memory as a set of blocks with pre-loading of the next block in the background. Requests are only ever made ``blocksize`` at a time, and are stored in an LRU cache. The least recently accessed block is discarded when more than ``maxblocks`` are stored. If the next block is not in cache, it is loaded in a separate thread in non-blocking way. Parameters ---------- blocksize : int The number of bytes to store in each block. Requests are only ever made for ``blocksize``, so this should balance the overhead of making a request against the granularity of the blocks. fetcher : Callable size : int The total size of the file being cached. maxblocks : int The maximum number of blocks to cache for. The maximum memory use for this cache is then ``blocksize * maxblocks``. backgroundrrrrrrrrrrrcsZt|||t|||_||_t|j||_t dd|_ d|_ d|_ t |_dS)NrI max_workers)rCr%rrrrrrrr_thread_executor_fetch_future_block_number _fetch_futurerr_fetch_future_lockrrGr#r$r%s zBackgroundBlockCache.__init__rcCrrrr-r#r#r$rrzBackgroundBlockCache.cache_inforicCs(|j}|d=|d=|d=|d=|d=|S)Nrrrrrrrlr#r#r$rnsz!BackgroundBlockCache.__getstate__cCsD|j|t|j|d|_tdd|_d|_d|_t |_ dS)NrrIr) rjrorrrrrrrrrrrlr#r#r$rps  z!BackgroundBlockCache.__setstate__r&r'rWr)c Cs|durd}|dur |j}||jks||krdS||j}||j}d}d}|jM|jduro|jdus6J|jrRtd|j |j |jd|_d|_nt ||jko]|kn}|ro|j}|j}d|_d|_Wdn1sywY|durtd|j | |t ||dD]}||q|d} |j%|jdur| |j kr|j| s| |_|j|j| d|_Wdn1swY|j||||dS)Nrr*z3BlockCache joined background fetch without waiting.z(BlockCache waiting for background fetch.rIasyncr)rrrrrdoner`rrrrrrbrrrsubmitrr) r"r&rWrrZfetch_future_block_numberZ fetch_futureZ must_joinrZend_block_plus_1r#r#r$r, sv            zBackgroundBlockCache._fetchsyncrlog_infor/csv||jkrtd|d|jd||j}||j}td|||j||7_|jd7_t||}|S)rrrr_z!BlockCache fetching block (%s) %drI) rrrr`rr rrCr,)r"rrr&rWrrGr#r$rVs   z!BackgroundBlockCache._fetch_blockrrc Crrrrr#r#r$rhs     z BackgroundBlockCache._read_cacherrrrrr7rq)r)rrrr/rr)r)r3r8r9r:rr;r%rrnrpr,rrrtr#r#rGr$rs   Lrz!dict[str | None, type[BaseCache]]cachesFclstype[BaseCache]clobberrrrcCs6|j}|s|tvrtd|dt||t|<dS)z'Register' cache implementation. Parameters ---------- clobber: bool, optional If set to True (default is False) - allow to overwrite existing entry. Raises ------ ValueError zCache with name z is already known: N)rrr)rrrr#r#r$register_caches  r)F)rrrrrr). __future__rrrloggingrrMrrconcurrent.futuresrrtypingrrrrr r r r r r=Ztyping_extensionsrrr getLoggerr`rr)rrr<rvr{r~rrrrrrr;rcr#r#r#r$sZ,    DV.. ee<Q