+dCddlZddlZddlZddlZddlmZmZmZddlm Z ej dZ GddeZ dS) N) DataSourceDataSourceBaseNoEntry)reload_on_changeintakeceZdZdZdZdZdZ d(fd ZedZ e d Z d Z d Z d Zd Ze dZed)dZdZed*dZdZdZdZd+dZedZfdZexZZedZdZdZdZ dZ!dZ"dZ#d Z$d!Z%d"Z&d#Z'd$Z(d%Z)d&Z*e d'Z+xZ,S),CatalogaManages a hierarchy of data sources as a collective unit. A catalog is a set of available data sources for an individual entity (remote server, local file, or a local directory of files). This can be expanded to include a collection of subcatalogs, which are then managed as a single unit. A catalog is created with a single URI or a group of URIs. A URI can either be a URL or a file path. Each catalog in the hierarchy is responsible for caching the most recent refresh time to prevent overeager queries. Attributes ---------- metadata : dict Arbitrary information to carry along with the data source specs. catalogN<Tdefaultc  tt|||_||_|pi|_||_||_||_| |_ t| tr.| r,ddl m  fd| D|_n5t| t t"frd| D|_ni|_|dvrt%d|z||_|r$t|t(rt%dt+j|_||n||_|dS) a Parameters ---------- entries : dict, optional Mapping of {name: entry} name : str, optional Unique identifier for catalog. This takes precedence over whatever is stated in the cat file itself. Defaults to None. description : str, optional Description of the catalog. This takes precedence over whatever is stated in the cat file itself. Defaults to None. metadata: dict Additional information about this data ttl : float, optional Lifespan (time to live) of cached modification time. Units are in seconds. Defaults to 1. getenv: bool Can parameter default fields take values from the environment getshell: bool Can parameter default fields run shell commands persist_mode: ['always', 'default', 'never'] Defines the use of persisted sources: if 'always', will use a persisted version of a data source, if it exists, if 'never' will always use the original source. If 'default', persisted sources will be used if they have not expired, and re-persisted and used if they have. storage_options : dict If using a URL beginning with 'intake://' (remote Intake server), parameters to pass to requests when issuing http commands; otherwise parameters to pass to remote backend file-system. Ignored for normal local files. r UserParametercZi|]'\}}|t|tr dd|i|n|(S)name) isinstancedict).0ruprs 3lib/python3.11/site-packages/intake/catalog/base.py z$Catalog.__init__..ds`$T$T$TowosuwDZXZ\`MaMa+i==+I+Id+Ib+I+I+Igi$T$T$Tc i|] }|d| Srrrrs rrz$Catalog.__init__..fs#M#M#MrBvJ#M#M#Mr)alwaysneverrz Persist mode (%s) not understoodzThe class intake.Catalog does not accept a string for `entries` Did you mean to use `intake.open_catalog`? Note that in versions of intake <=0.5.4 `intake.Catalog` was an alias for `intake.open_catalog`. It is now the intake base Catalog class.N)superr __init__r descriptionmetadatattlgetenvgetshellstorage_optionsrrlocalritemsuser_parameterslisttuple ValueErrorpmodestrtimeupdated_make_entries_container_entries force_reload) selfentriesrr#r$r%r&r' persist_moder(r+r __class__s @rr"zCatalog.__init__,sZ gt%%''' & B    . ot , , & & , , , , , ,$T$T$T$T|K|Q|Q|S|S$T$T$TD $ 7 7 &#M#M_#M#M#MD #%D = = P?,NOO O!  z'3// ! y{{ #*ZD an iterator of entry names * ``__getitem__(key)`` -> an Entry * ``items()`` -> an iterator of ``(key, Entry)`` pairs For best performance the object should also implement: * ``__len__()`` -> int * ``__contains__(key)`` -> boolean In ``__len__`` or ``__contains__`` are not implemented, intake will fall back on iterating through the entire catalog to compute its length or check for containment, which may be expensive on large catalogs. rrAs rr3zCatalog._make_entries_containers , rcdS)z#Override this: load catalog entriesNrrAs r_loadz Catalog._load rc^tj|_|dS)zImperative reload data nowN)r1r2rDrAs rr5zCatalog.force_reloads y{{  rc|j:tj|jz |jkr|dSdSdS)z,Reload catalog if sufficient time has passedN)r%r1r2r5rAs rreloadzCatalog.reloadsS H ty{{T\'ADH'L            rc8|jddS)Nversionr)r$getrAs rrJzCatalog.versions}  A...rrc ddl|fd||D}t ||jdz|j|j |j |j pi|j |j }||jd|j d<||_|D] }||_ |S)Nrczi|]6\}tfdD |7S)c3K|]K}|tvVLdSN)r0describevalueslower)rwordvs r z,Catalog.search...s|USUSEIUY]`abakakamamatatavav]w]w]}]}]]UUSUSUSUSUSUSr)anycopy)rkrTrWwordss @rrz"Catalog.search..sTTTtq!QTUSUSUSUSMRUSUSUSRSRST1diillTTTr)depth_search)rr%r&r'r$r(r+)textupstreamsearch)rWrRsplitwalkr*r r?rr%r&r'r$r(r+r>rQ_catalog)r6r\rZr7r>erWrYs @@rr^zCatalog.searchs  ""$$TTTTTtyyuy/E/E/K/K/M/MTTT Y&;]m)r//11 0 05577   +/DI!F!F X!!  AAJJ rc~tfd|jDS)a[ Create a Catalog of a subset of entries based on a condition .. warning :: This function operates on CatalogEntry objects not DataSource objects. .. note :: Note that, whatever specific class this is performed on, the return instance is a Catalog. The entries are passed unmodified, so they will still reference the original catalog instance and include its details such as directory,. Parameters ---------- func : function This should take a CatalogEntry and return True or False. Those items returning True will be included in the new Catalog, with the same entry names Returns ------- Catalog New catalog with Entries that still refer to their parents c2i|]\}}|||Srr)rkeyentryfuncs rrz"Catalog.filter..s1!d!d!deX\X\]bXcXc!d#u!d!d!dr)r r?r4r*)r6rgs `rfilterzCatalog.filters<8  !d!d!d!dt}?R?R?T?T!d!d!deeercV||ni}|gn|}|jD]\}}|jdkrT|dkrN ||||gz|dz n&#t$r}t |Yd}~nd}~wwxYwd||gz}|||<|S)aGet all entries in this catalog and sub-catalogs Parameters ---------- sofar: dict or None Within recursion, use this dict for output prefix: list of str or None Names of levels already visited depth: int Number of levels to descend; needed to truncate circular references and for cleaner output Returns ------- Dict where the keys are the entry names in dotted syntax, and the values are entry instances. Nr r.)r4r* _containerr` Exceptionprintjoin) r6sofarprefixrZoutritemrbns rr`z Catalog.walks&0eeb16---//  JD$)+   DFFKKVtf_eai@@@@ !HHHDDDD4&))ACFF s&A$$ B.BBc#~K|D]\}}||fVdS)zBGet an iterator over (key, source) tuples for the catalog entries.N) _get_entriesr*)r6rrfs rr*z Catalog.itemssT,,..4466  KD%-      rc#tK|D]}|VdS)z5Get an iterator over the sources for catalog entries.N)rurQ)r6rfs rrQzCatalog.valuessH&&((//11  E%''MMMM  rcddl}|ji|j|jd}|jD]\}}|j}|ddd| dgD|d< t|dtr0d |dj |djg|d<n#t$rYnwxYw||d |<||S) z Produce YAML version of this catalog. Note that this is not the same as ``.yaml()``, which produces a YAML block referring to this catalog. rN)r$sourcesrr#r cNi|]"}|j|d#S)r=)r __getstate__)rrXs rrz%Catalog.serialize..+s+eeeq(8(8(Beeer parametersdriverrjrx)yamlr$rr#r4r*_captured_init_kwargsrWpoprK issubclassrrn __module____name__ TypeErrordump)r6r}outputrerfkws r serializezCatalog.serializes   "mDI^b^noo---// ( (JC,1133B FF9d # # #eeBFFS_acLdLdeeeB|  blN;;^#&88R\-DblF[,\#]#]BxL    %'F9 c " "yy   s A C C%$C%cddlm}||gfi|piddid5}||ddddS#1swxYwYdS)z Output this catalog to a file as YAML Parameters ---------- url : str Location to save to, perhaps remote storage_options : dict Extra arguments for the file-system r) open_filesmodewtN)fsspecrwriter)r6urlr(rfs rsavez Catalog.save4s &%%%%% Z D D/"7R D Dt D D DQ G &1 GGDNN$$ % % % & & & & & & & & & & & & & & & & & &s(AAAc|j|}||_|j|_t d|jDfd|jD}||jpgz|_|S)Nc3^K|](}t|tr|dn|jV)dS)rN)rrrrs rrUz%Catalog._get_entry..Js;jjRjT&:&:G6 jjjjjjrc"g|] \}}|v | Srr)rrrup_namess r z&Catalog._get_entry..Ks'WWWhdB$hBVWrWWWr)r4rar/_pmodeset_user_parametersr+r*)r6rrfupsrs @r _get_entryzCatalog._get_entryDs d#z jjSXSijjjjjWWWW$"6"<"<">">WWW!$(>(D"!Euwwrc `ddlm}i}|D]\}}|jD]}t |trF||dkr9|}||d<|di|||<||]||j krC|j }||d<||d<|di|||<||tj di|}|j ||S)Nrrrrr)r)rrWr*r+rQrrrrr~r! configure_newupdate) r6r=rrrXrTrrnewr9s rrzCatalog.configure_newOs\((((((KKMM'')) & &DAq*1133 & &b$'' &BvJ&WWYY()9 !.!4!4!4!4A 1 BG|&5::<<()9 %&6 !.!4!4!4!4A 1  &$egg#--f-- ""3''' rc|jSrOr;rAs rruzCatalog._get_entriesis }rcDt|S)z,Return an iterator over catalog entry names.)iterrurAs r__iter__zCatalog.__iter__msD%%''(((rc t|S)z?Entry names in this catalog as an iterator (alias for __iter__))rrAs rkeysz Catalog.keysqsDzzrcDt|SrO)lenrurAs r__len__zCatalog.__len__us4$$&&'''rc.||vSrO)rur6res r __contains__zCatalog.__contains__xsd''))))rchd|Dt|jzS)Ncdg|]-}tjd|tj|+|.S)z[_A-Za-z][_a-zA-Z0-9]*$)rematchkeyword iskeyword)rrfs rrz#Catalog.__dir__..~sN   rx0I5'Q'Q ZaZklqZrZr    r)r,__dict__rrAs r__dir__zCatalog.__dir__|sF  #    M       rc t|SrO)r,rAs r_ipython_key_completions_z!Catalog._ipython_key_completions_sDzzrcd|jzS)NzrrAs r__repr__zCatalog.__repr__s% 11rc|dkrtd|ds+ ||S#t$r}t||d}~wwxYwt|)Nrfz(Source was not made from a catalog entry_)r startswithKeyErrorAttributeError)r6rrrbs r __getattr__zCatalog.__getattr__s 7? FDEE Es## 2 2Dz! 2 2 2$T**1 2T"""s4 AAAc||j|<dS)aAdd entry to catalog This relies on the `_entries` attribute being mutable, which it normally is. Note that if a catalog automatically reloads, any entry added here may be very transient Parameters ---------- key : str Key to give the entry in the cat entry : CatalogEntry The entry to include (could be local, remote) Nr;)r6rerfs r __setitem__zCatalog.__setitem__s# crc6|j|S)aSRemove entry from catalog and return it This relies on the `_entries` attribute being mutable, which it normally is. Note that if a catalog automatically reloads, any entry removed here may soon reappear Parameters ---------- key : str Key to give the entry in the cat )r4rrs rrz Catalog.pops}  %%%rct|ts`||vr\||}|jdkr:||_|j|j|S|St|trd|vr| d}t|trt|dd}d}|ro| d}||rdndz|z}||j vrCd |} |j ||}|S#t$rYkwxYw|on.t|tr|}|D] }||} |St|)aReturn a catalog entry by name. Can also use attribute syntax, like ``cat.entry_name``, or item lookup cat['non-python name']. This enables walking through nested directories with cat.name1.name2, cat['name1.name2'] *or* cat['name1', 'name2'] r rjNr)rr,r containerrr+rrWr0r_rr4rnrr-) r6respartsrpbitrestrqparts r __getitem__zCatalog.__getitem__s#t$$  $$A{i' !(()=)B)B)D)DEEEH c3   !C3J !))C..C c4  IIaaaLEF !iilĺb9C?T]*!88E??D!"mF3D9"suu #!!!!! !U # # C  $i355LsmmsD11 D>=D>cddd|jdS)Nr )rshapedtyper$)r$rAs rdiscoverzCatalog.discovers&RVR_```rcdSrOrrAs r_closezCatalog._closerErct|ds'ddlm}ddlm}||||_n d|j_|jS)N_guir)output_notebook)GUIT)hasattr interfacer interface.guirrvisible)r6rrs rguiz Catalog.guisitV$$ % 3 3 3 3 3 3 + + + + + + O   D DII $DI yr) NNNNr TTrNN)r)NNrrO)-rr __qualname____doc__rrauthr" classmethodr?propertyr=r3rDr5rHrJrr^rhr`r*rQrrrr__call__rKrurrrrrrrrrrrrrr __classcell__)r9s@rr r s(I D D  MMMMMM^[(22X20       //X/*fff<@    !!!,&&&& 0#"Hs)))(((***   222 # # #### & & &&&&Paaa     X     rr ) rloggingrr1 source.baserrrutilsr getLoggerloggerr rrrrs ==========######  8 $ $[[[[[j[[[[[r